Command Line Options

Dry run mode

Using the option –dry-run you get a tool that help you to analyse your pipeline with following rules when the option is set:

  • parallelism (matrix and tasks) is disabled
  • custom logging is disabled
  • the default logging is adjusted to have no timestamps
  • using docker(image) tasks the Dockerfile is printed as Bash comment; the Dockerfile is not written as a file.
  • The cleanup hooks are also not executed but logged.

As an example a docker(image) task might look similar to following output:

$ spline --definition=examples/docker-image.yaml --dry-run
spline.application - Running with Python 2.7.13 (default, Jan 19 2017, 14:48:08) [GCC 6.3.0 20170118]
spline.application - Running on platform Linux-4.9.0-3-amd64-x86_64-with-debian-9.1
spline.application - Processing pipeline definition 'examples/docker-image.yaml'
spline.application - Schema validation for 'examples/docker-image.yaml' succeeded
spline.components.stage - Processing pipeline stage 'Example'
spline.components.tasks - Processing group of tasks (parallel=disabled)
spline.components.tasks - Processing Bash code: start
spline.components.bash - Dry run mode for script /tmp/pipeline-script-TRd8fF.sh
spline.components.tasks -  | #!/bin/bash
spline.components.tasks -  | # Dockerfile:
spline.components.tasks -  | # >>
spline.components.tasks -  | # FROM centos:7
spline.components.tasks -  | # RUN yum -y install yum-utils git
spline.components.tasks -  | # RUN yum -y install https://centos7.iuscommunity.org/ius-release.rpm
spline.components.tasks -  | # RUN yum -y install python36u python36u-pip
spline.components.tasks -  | # RUN pip3.6 install tox
spline.components.tasks -  | #
spline.components.tasks -  | # <<
spline.components.tasks -  | # for visibility in logging
spline.components.tasks -  | echo "docker build -t python:3.6 < dockerfile.dry.run.see.comment"
spline.components.tasks -  | # trying to build docker image
spline.components.tasks -  | docker build -t python:3.6 - < dockerfile.dry.run.see.comment
spline.components.tasks -  | docker_error=$?
spline.components.tasks -  | # cleanup
spline.components.tasks -  | rm -f
spline.components.tasks -  | # give back result
spline.components.tasks -  | exit ${docker_error}
spline.components.tasks - Processing Bash code: finished

Debug

The option –debug adjust the Bash option set -x which activates the debug mode in Bash. The primes example in the spline repository gives you a good example. I’m just printing the first 20 lines:

$ spline --definition=examples/primes.yaml --debug 2>&1 | head -20
2018-01-05 19:31:12,023 - spline.application - Running with Python 2.7.13 (default, Jan 19 2017, 14:48:08) [GCC 6.3.0 20170118]
2018-01-05 19:31:12,028 - spline.application - Running on platform Linux-4.9.0-3-amd64-x86_64-with-debian-9.1
2018-01-05 19:31:12,028 - spline.application - Current cpu count is 4
2018-01-05 19:31:12,029 - spline.application - Processing pipeline definition 'examples/primes.yaml'
2018-01-05 19:31:12,032 - spline.application - Schema validation for 'examples/primes.yaml' succeeded
2018-01-05 19:31:12,032 - spline.components.stage - Processing pipeline stage 'Calculate Primes'
2018-01-05 19:31:12,033 - spline.components.tasks - Processing group of tasks (parallel=no)
2018-01-05 19:31:12,033 - spline.components.tasks - Processing Bash code: start
2018-01-05 19:31:12,043 - spline.components.bash - Running script /tmp/pipeline-script-5iRCsz.sh
2018-01-05 19:31:12,050 - spline.components.tasks -  | ++ seq 0 100
2018-01-05 19:31:12,051 - spline.components.tasks -  | + for n in $(seq 0 100)
2018-01-05 19:31:12,052 - spline.components.tasks -  | ++ is_prime 0
2018-01-05 19:31:12,052 - spline.components.tasks -  | ++ n=0
2018-01-05 19:31:12,052 - spline.components.tasks -  | ++ '[' 0 -lt 2 ']'
2018-01-05 19:31:12,052 - spline.components.tasks -  | ++ return
2018-01-05 19:31:12,053 - spline.components.tasks -  | + '[' '' == yes ']'
2018-01-05 19:31:12,053 - spline.components.tasks -  | + for n in $(seq 0 100)
2018-01-05 19:31:12,053 - spline.components.tasks -  | ++ is_prime 1
2018-01-05 19:31:12,053 - spline.components.tasks -  | ++ n=1
2018-01-05 19:31:12,053 - spline.components.tasks -  | ++ '[' 1 -lt 2 ']'

Temporary Scripts Path

The Python library functionality related to temporary folders is explained here: https://docs.python.org/2/library/tempfile.html#tempfile.mkstemp

However you can specify another path that is used to store splines temporary scripts by specifying the path with –temporary-scripts-path. When the path doesn’t exist the tool tries to create it for you. Here’s an example:

$ spline --definition=examples/colors.yaml --temporary-scripts-path=$PWD/temp
2018-02-06 05:52:57,547 - spline.application - Running with Python 2.7.13 (default, Nov 24 2017, 17:33:09) [GCC 6.3.0 20170516]
2018-02-06 05:52:57,553 - spline.application - Running on platform Linux-4.9.0-5-amd64-x86_64-with-debian-9.3
2018-02-06 05:52:57,553 - spline.application - Current cpu count is 4
2018-02-06 05:52:57,553 - spline.application - Processing pipeline definition 'examples/colors.yaml'
2018-02-06 05:52:57,557 - spline.application - Schema validation for 'examples/colors.yaml' succeeded
2018-02-06 05:52:57,558 - spline.components.stage - Processing pipeline stage 'Example'
2018-02-06 05:52:57,559 - spline.components.tasks - Processing group of tasks (parallel=no)
2018-02-06 05:52:57,566 - spline.components.tasks - Processing Bash code: start
2018-02-06 05:52:57,568 - spline.components.bash - Running script /work/pipeline/temp/pipeline-script-FxvNFG.sh