Backends¶

gwf supports multiple backends for running workflows. If you don’t find a backend that suits your needs here, it’s easy to write your own backend <writing_backends>`. The source code for the built-in backends is a great source of inspiration.

By default, gwf comes with the local, slurm, sge, and lsf backends.

Local¶

Backend that runs targets on a local cluster.

Changed in version 3.0.0: The local backend is now a resource-aware scheduler supporting target dependencies, cancellation, failure and timeout reporting, executors, and cores, memory, and walltime limits. It now communicates over a per-workflow Unix domain socket instead of a configurable TCP host/port.

To use this backend you must activate the local backend and start a local scheduler that the backend can submit targets to. To make two cores available to the scheduler, run the command:

gwf -b local workers -n 2

in the working directory of your project. The workflow file must be accessible to gwf. Thus, if your workflow file is not called workflow.py or the workflow object is not called gwf, you must specify this so that gwf can locate the workflow:

gwf -f myworkflow.py:wf1 -b local workers -n 2

If the local backend is your default backend you can of course omit the -b local option.

If the -n option is omitted, gwf will detect the number of cores available and use all of them. The scheduler uses the system’s physical memory by default; use --max-memory to set another capacity.

To run your workflow, open another terminal and then type:

gwf -b local run

To stop the scheduler press Control-c.

The scheduler communicates over a per-workflow Unix domain socket under $XDG_RUNTIME_DIR/gwf. If XDG_RUNTIME_DIR is unavailable, gwf uses a private directory under the system temporary directory.

Target options:

cores (int):
Number of cores allocated to this target (default: 1).
memory (str):
Memory allocated to this target (default: 1g).
walltime (str):
Wall-clock time limit, HH:MM:SS (default: 01:00:00).

Slurm¶

Backend for the Slurm workload manager.

Changed in version 3.0.0: slurm_args was added for passing extra arguments directly to sbatch.

To use this backend you must activate the slurm backend.

Backend options:

backend.slurm.log_mode (str): Must be either full, merged or
none. If full, two log files will be stored for each target, one for standard output and one for standard error. If merged, only one log file will be written containing the combined streams. If none, no logs will be stored. (default: full).
backend.slurm.accounting_enabled (str): If enabled, will use sacct to fetch job status from Slurm. This enables gwf to report when targets have failed (default: true).

Target options:

cores (int):
Number of cores allocated to this target (default: 1).
memory (str):
Memory allocated to this target (default: 1).
walltime (str):
Time limit for this target (default: 01:00:00).
queue (str):
Queue to submit the target to. To specify multiple queues, specify a comma-separated list of queue names. A queue is equivalent to a Slurm partition.
account (str):
Account to be used when running the target.
constraint (str):
Constraint string. Equivalent to setting the –constraint flag on sbatch.
qos (str):
Quality-of-service string. Equivalent to setting the –qos flog on sbatch.
mail_type (str):
Equivalent to the –mail-type flag on sbatch.
mail_user (str):
Equivalent to the –mail-user flag on sbatch.
gres (str):
Equivalent to the –gres flog on sbatch. Usually used to request access to GPUs.
slurm_args (list[str]):
Extra arguments passed directly to sbatch. Use this for Slurm options that are not otherwise supported by gwf.

Sun Grid Engine (SGE)¶

Backend for Sun Grid Engine (SGE).

Changed in version 3.0.0: sge_args was added for passing extra arguments directly to qsub, and executors are now supported.

To use this backend you must activate the sge backend. The backend currently assumes that a SGE parallel environment called “smp” is available. You can check which parallel environments are available on your system by running qconf -spl.

Backend options:

None.

Target options:

cores (int):
Number of cores allocated to this target (default: 1).
memory (str):
Memory allocated to this target (default: 1).
walltime (str):
Time limit for this target (default: 01:00:00).
queue (str):
Queue to submit the target to. To specify multiple queues, specify a comma-separated list of queue names.
account (str):
Account to be used when running the target. Corresponds to the SGE project.
sge_args (list[str]):
Extra arguments passed directly to qsub. Use this for SGE options that are not otherwise supported by gwf.

IBM Spectrum Load Sharing Facility (LSF)¶

Backend for IBM Load Sharing Facility (LSF).

Changed in version 3.0.0: lsf_args was added for passing extra arguments directly to bsub, and executors are now supported.

To use this backend, you must activate the lsf backend. This backend requires the commands bsub and bjobs.

Backend options:

None.

Target options:

cores (int):
Number of cores allocated to this target (default: 1).
memory (str):
Memory allocated to this target (default: 4GB).
queue (str):
Queue to submit the target to (default: normal). for different purposes or priorities.
lsf_args (list[str]):
Extra arguments passed directly to bsub. Use this for LSF options that are not otherwise supported by gwf.

Portable Batch System (PBS)¶

Backend for Portable Batch System (PBS).

Changed in version 3.0.0: walltime, account, and pbs_args target options were added, and executors are now supported.

To use this backend, you must activate the pbs backend. This backend requires the commands qsub and qstat.

Backend options:

None.

Target options:

cores (int):
Number of cores allocated to this target (default: 1).
memory (str):
Memory allocated to this target (default: 4GB).
walltime (str):
Wall-clock time limit, HH:MM:SS (default: 01:00:00). Emitted as #PBS -l walltime=....
queue (str):
Queue to submit the target to (default: normal).
account (str):
Accounting/group string for the job (default: unset). When set, emitted as #PBS -A <account>; when empty the directive is omitted entirely.
pbs_args (list[str]):
Extra arguments passed directly to qsub. Use this for PBS options that are not otherwise supported by gwf.