A few tips and tricks for working with slurm (i.e. submitting jobs using sbatch) that I frequently forget!

Using slurm + conda

In order to use conda activate in an sbatch script you need to actually set up conda. You should have an environment variable called $CONDA_PREFIX in your settings. You can check that with:

envs | grep conda

Before you can do conda activate you need to source the definition file to get conda up and running. Add this to your sbatch script:

. "$CONDA_PREFIX/etc/profile.d/conda.sh"
conda activate myenv

Change myenv to whatever environment you want to activate! Note if you don’t have $CONDA_PREFIX, see if there is a variable that points to the base of your conda installation.

Using slurm and multiple tasks

This one is quite simple, but I often forget it!

You can specify multiple tasks and mutliple processors per task in your sbatch script:

#SBATCH --ntasks=5
#SBATCH --cpus-per-task=30

With this you can submit 5 different tasks each using 30 CPUs (essentially threads). But how do you submit those? One way is to use the & to background a task:

task 1 &
task 2 &
task 3 &

Or, more likely you will be doing something in a loop:

for ID in id1 id2 id3 id4 id5; do task $ID & echo $ID; done

The problem is that this will submit to the queue, start the jobs, but not wait for them to finish. The key is to add the command wait at the end of the file (or after the loop). This will wait for the child processes to finish, and so the jobs run on the cluster.

for ID in id1 id2 id3 id4 id5; do task $ID & echo $ID; done
wait