Using Greasy metascheduler in Puhti
In many cases, a computational analysis job contains a number of similar independent sub-tasks. A user may have several datasets that are analyzed in the same way, or the same simulation code is executed with a number of different parameters. These kind of tasks are often called as farming or embarrassingly parallel jobs as the work can in principle be distributed to as many processors as there are subtasks to run.
In Puhti GREASY can be used as an alternative for array jobs for running embarrasingly parallel tasks. Greasy is especiallally useful in cases where you need to define dependencies between the tasks to be executed.
GREASY was originally developed at BSC. At CSC we use the GREASY version that includes the extensions developed at CSCS. For detailed documentation please check:
Strengths of Greasy
- Easy to use
- OpenMP threading support
Disadvantages of Greasy
- Only simple dependencies can be handled
- No error recovery strategies
- Creates job steps
- Integrates with Slurm, but subtasks must use identical resources
You should not use GREASY to run MPI parallel tasks. GREASY is not able to manage MPI jobs effectively.
GREASY Task lists
Job scheduling with GREASY is based on task lists that have one task (command) in one row. In the simplest approach, the task list is just a file containing the commands to be executed. For example, analyzing 200 input files with program my_prog could be described with task list containing 200 rows:
my_prog < input1.txt > output1.txt my_prog < input2.txt > output2.txt my_prog < input3.txt > output3.txt ... my_prog < input200.txt > output200.txt
If needed, you can define dependencies between jobs with the
syntax. For example, if you would like to merge the output files of the previous
example into a one file, you could add one more row to the task list:
my_prog < input1.txt > output1.txt my_prog < input2.txt > output2.txt [#1#] my_prog < input3.txt > output3.txt ... my_prog < input200.txt > output200.txt [#1-200#] cat output* > all_output
By default, all tasks are executed in the directory where the task list processing is launched, but you can add task specific execution directories to the task list with:
[@ /path/to/folder_task1/ @] my_prog < input1.txt > output1.txt [@ /path/to/folder_task2/ @] my_prog < input2.txt > output2.txt [@ /path/to/folder_task3/ @] my_prog < input3.txt > output3.txt ... [@ /path/to/folder_task200/ @] my_prog < input200.txt > output200.txt
You should not include srun to your tasks as GREASY will add it when it executes the tasks. Please check the GREASY user guide for more detailed description of the task list syntax.
Executing a task list
To use GREASY in Puhti, load the GREASY module:
module load greasy
-c) 2. estimated average duration for one task (
-t) 3. number of nodes used to execute the tasks (
-N) 4. accounting project (
-A). 5. estimated memory usage for one task (
Alternatively you can define part or all of these parameters in command line:
sbatch-greasy tasklist -c 1 -t 15:00 -N 1 -A project_2012345
With the option
-f filename you can make sbatch-greasy to save the GREASY batch
file but not to send it to be executed. This batch job file can then be further
edited according to your needs, e.g. if you need to set up additional SLURM parameters.
Submit it normally with:
Performance in threaded (OpenMP) jobs can be sensitive to the thread binding. If your job is parallelized via OpenMP, make sure the performance of individual subjobs has not suffered. A single subjob must fit in one node, but such a job could also be run as an array job. GREASY is thus better suited for jobs (much) smaller than one node.
While Greasy only creates one batch job, it will create a job step for each task it runs. A huge number of job tasks in a batch job will be problematic. If you need to run hundreds or thousands of job steps, please contact firstname.lastname@example.org to look for alternatives.