This is an old revision of the document!
$$$$$$$$$$$$ ====== Array jobs for clusters running SGE ======$$$$$$$$$$$$ $$$$$$$$$$$$ == The problem ==$$$$$$$$$$$$ $$$$$$$$$$$$ $$$$$$$$$$$$ $$$$$$$$$$$$ $$$$$$$$$$$$ A common problem is that you have a large number of jobs to run, and they are largely identical in terms of the command to run. For example, you may have 1000 data sets, and you want to run a single program on them, using the cluster. The naive solution is to somehow generate 1000 shell scripts, and submit them to the queue. This is not efficient, neither for you nor for the head node.$$$$$$$$$$$$ $$$$$$$$$$$$ $$$$$$$$$$$$ $$$$$$$$$$$$ $$$$$$$$$$$$ == Array jobs are the solution ==$$$$$$$$$$$$ $$$$$$$$$$$$ $$$$$$$$$$$$ $$$$$$$$$$$$ $$$$$$$$$$$$ There is an alternative on SGE systems – array jobs. The advantages are:$$$$$$$$$$$$ $$$$$$$$$$$$ $$$$$$$$$$$$
- You only have to write one shell script$$$$$$$$$$$$
- You don’t have to worry about deleting thousands of shell scripts, etc.$$$$$$$$$$$$
- If you submit an array job, and realize you’ve made a mistake, you only have one job id to qdel, instead of figuring out how to remove 100s of them.$$$$$$$$$$$$
- You put less of a burden on the head node.$$$$$$$$$$$$
$$$$$$$$$$$$
$$$$$$$$$$$$
In fact, there are no disadvantages that I’m aware of. Submitting an array job to do 1000 computations is entirely equivalent to submitting 1000 separate scripts, but much less work for you.$$$$$$$$$$$$
$$$$$$$$$$$$
$$$$$$$$$$$$
$$$$$$$$$$$$
$$$$$$$$$$$$
$$$$$$$$$$$$
$$$$$$$$$$$$
$$$$$$$$$$$$
$$$$$$$$$$$$
=== Pulling data from the i-th line of a file ===$$$$$$$$$$$$
$$$$$$$$$$$$
$$$$$$$$$$$$
$$$$$$$$$$$$
$$$$$$$$$$$$
Let’s say you have a list of numbers in a file, one number per line. For example, the numbers could be random number seeds for a simulation. For each task in an array job, you want to get the i th line from the file, where i equals SGE_TASK_ID, and use that value as the seed. This is accomplished by using the Unix head and tail commands. (Read the man pages for those commands – don’t ask me.)$$$$$$$$$$$$
$$$$$$$$$$$$
$$$$$$$$$$$$
$$$$$$$$$$$$
$$$$$$$$$$$$ $$$$$$$$$$$$ #!/bin/sh$$$$$$$$$$$$ #$ -t 1-10000$$$$$$$$$$$$ SEEDFILE=~/data/seeds$$$$$$$$$$$$ SEED=$(cat $SEEDFILE | head -n $SGE_TASK_ID | tail -n 1)$$$$$$$$$$$$ ~/programs/simulation -s $SEED -o ~/results/output.$SGE_TASK_ID$$$$$$$$$$$$ $$$$$$$$$$$$
$$$$$$$$$$$$ $$$$$$$$$$$$ $$$$$$$$$$$$ === What if you number files from 0 instead of 1? ===$$$$$$$$$$$$ $$$$$$$$$$$$ $$$$$$$$$$$$ $$$$$$$$$$$$ $$$$$$$$$$$$ The '-t' option will not accept 0 as part of the range, i.e. #$ -t 0-99 is invalid, and will generate an error. However, I often label my input files from 0 to n-1. That’s easy to deal with:$$$$$$$$$$$$ $$$$$$$$$$$$ $$$$$$$$$$$$
$$$$$$$$$$$$ $$$$$$$$$$$$ #!/bin/sh$$$$$$$$$$$$ # Tell the SGE that this is an array job, with "tasks" to be numbered 1 to 10000$$$$$$$$$$$$ #$ -t 1-10000$$$$$$$$$$$$ i=$(expr $SGE_TASK_ID - 1)$$$$$$$$$$$$ if [ ! -e ~/results/output.$i ]$$$$$$$$$$$$ then$$$$$$$$$$$$ ~/programs/program -i ~/data/input.$i -o ~/results/output.$i$$$$$$$$$$$$ fi$$$$$$$$$$$$ $$$$$$$$$$$$ $$$$$$$$$$$$
$$$$$$$$$$$$ $$$$$$$$$$$$ $$$$$$$$$$$$ $$$$$$$$$$$$ $$$$$$$$$$$$ $$$$$$$$$$$$ $$$$$$$$$$$$ $$$$$$$$$$$$ $$$$$$$$$$$$ $$$$$$$$$$$$ $$$$$$$$$$$$
