Backends¶

gwf supports multiple backends for running workflows. If you don’t find a backend that suits your needs here, it’s easy to write your own backend <writing_backends>`. The source code for the built-in backends is a great source of inspiration.

By default, gwf comes with the local, slurm, sge, and lsf backends.

Local¶

Backend that runs targets on a local cluster.

To use this backend you must activate the local backend and start a local cluster (with one or more workers) that the backend can submit targets to. To start a cluster with two workers run the command:

gwf -b local workers -n 2

in the working directory of your project. The workflow file must be accessible to gwf. Thus, if your workflow file is not called workflow.py or the workflow object is not called gwf, you must specify this so that gwf can locate the workflow:

gwf -f myworkflow.py:wf1 -b local workers -n 2

If the local backend is your default backend you can of course omit the -b local option.

If the -n option is omitted, gwf will detect the number of cores available and use all of them.

To run your workflow, open another terminal and then type:

gwf -b local run

To stop the pool of workers press Control-c.

Backend options:

local.host (str): Set the host that the workers are running on
(default: localhost).
local.port (int): Set the port used to connect to the workers
(default: 12345).

Target options:

None available.

Slurm¶

Backend for the Slurm workload manager.

To use this backend you must activate the slurm backend.

Backend options:

backend.slurm.log_mode (str): Must be either full, merged or
none. If full, two log files will be stored for each target, one for standard output and one for standard error. If merged, only one log file will be written containing the combined streams. If none, no logs will be stored. (default: full).
backend.slurm.accounting_enabled (str): If enabled, will use sacct to fetch job status from Slurm. This enables gwf to report when targets have failed (default: true).

Target options:

cores (int):
Number of cores allocated to this target (default: 1).
memory (str):
Memory allocated to this target (default: 1).
walltime (str):
Time limit for this target (default: 01:00:00).
queue (str):
Queue to submit the target to. To specify multiple queues, specify a comma-separated list of queue names. A queue is equivalent to a Slurm partition.
account (str):
Account to be used when running the target.
constraint (str):
Constraint string. Equivalent to setting the –constraint flag on sbatch.
qos (str):
Quality-of-service string. Equivalent to setting the –qos flog on sbatch.
mail_type (str):
Equivalent to the –mail-type flag on sbatch.
mail_user (str):
Equivalent to the –mail-user flag on sbatch.
gres (str):
Equivalent to the –gres flog on sbatch. Usually used to request access to GPUs.

Sun Grid Engine (SGE)¶

Backend for Sun Grid Engine (SGE).

To use this backend you must activate the sge backend. The backend currently assumes that a SGE parallel environment called “smp” is available. You can check which parallel environments are available on your system by running qconf -spl.

Backend options:

None.

Target options:

cores (int):
Number of cores allocated to this target (default: 1).
memory (str):
Memory allocated to this target (default: 1).
walltime (str):
Time limit for this target (default: 01:00:00).
queue (str):
Queue to submit the target to. To specify multiple queues, specify a comma-separated list of queue names.
account (str):
Account to be used when running the target. Corresponds to the SGE project.

IBM Spectrum Load Sharing Facility (LSF)¶

Backend for IBM Load Sharing Facility (LSF).

To use this backend, you must activate the lsf backend. This backend requires the commands bsub and bjobs.

Backend options:

None.

Target options:

cores (int):
Number of cores allocated to this target (default: 1).
memory (str):
Memory allocated to this target (default: 4GB).
queue (str):
Queue to submit the target to (default: normal). for different purposes or priorities.