meta data for this page
  •  

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
general:computerenvironment:slurm [2021/10/12 07:49] – [Cancle jobs] felixgeneral:computerenvironment:slurm [2023/10/16 12:04] (current) – [sview] ingo
Line 16: Line 16:
 ===== Creating a script file ===== ===== Creating a script file =====
  
-The script file tells SLURM what should be done. The example below serves as a general guideline. Please do not simply copy it, but instead try to understand what information is provided to SLURM, and then modify the information to match your requirements. This applies, in particular, to the options –cpu-per-task and –mem-per-cpu. If you set the values for these options too high, your job will either remain pending until the requested resources are free on the cluster, or alternatively, your job will run and will then block the requested resources although they will not be used.+The script file tells SLURM what should be done. The example below serves as a general guideline. Please do not simply copy it, but instead try to understand what information is provided to SLURM, and then modify the information to match your requirements. This applies, in particular, to the options ''–cpu-per-task'' and ''–mem-per-cpu''. If you set the values for these options too high, your job will either remain pending until the requested resources are free on the cluster, or alternatively, your job will run and will then block the requested resources although they will not be used.
  
  
Line 22: Line 22:
 ==== Example ==== ==== Example ====
  
-This example uses a simple seed file that provides input for the program calls, e.g. file names, parameter values and the like.+This example uses a simple seed file that provides input for the program calls, e.g. file names, parameter values and the like 
 + 
 +You can use any text editor like ''nano'' to create a file, you can then paste the contents of the example file below into that file. Exchange the example command with the actual command you want to use and don't forget to modify the CPU and memory requests.
  
 <code> <code>
 #!/bin/bash #!/bin/bash
-#SBATCH --partition=pool          # specifies the queue you are submitting to +#SBATCH --partition=pool              # specifies the queue you are submitting to 
-#SBATCH --account=praktikant      # specifies your role in the system +#SBATCH --account=intern              # specifies your role in the system 
-#SBATCH --cpus-per-task=1         # how many cpus will you require per task (be reasonable here) +#SBATCH --cpus-per-task=1             # how many cpus will you require per task (be reasonable here) 
-#SBATCH --mem-per-cpu=1mb         # how much RAM will you need (be reasonable here) +#SBATCH --mem-per-cpu=1mb             # how much RAM will you need (be reasonable here) 
-#SBATCH --job-name="Test"         # self-explanatory +#SBATCH --job-name="Test"             # self-explanatory
-#SBATCH --array=1-8%4             # specifies how many jobs you are submitting (here 8). +
-                                  # The %4 specifies that never more than 4 jobs run in parallel +
- +
-echo This is task $SLURM_ARRAY_TASK_ID # executes the command "echo", which prints some information +
-                                       # about your job +
- +
-SEED=$(awk "FNR==$SLURM_ARRAY_TASK_ID" seedfile.txt) # reads in the seed file and will place each line +
-                                                     # into the variable $SEED+
  
-echo "Hello $SEED"                    # executes the command "echo", which will now print each line of +echo "Hello World"                    # your command goes here
-                                      # the seed file. Place here any command that you want to run on the +
-                                      # remote computer+
 </code> </code>
  
-You can now start using slurm by simply changing the program call to the task you want to perform. Find below a number of settings that are specific to the Applied Bioinformatics system. 
  
-===== Create a Seedfile =====+===== Job submission =====
  
-A seedfile is no more than a textfile which enlists the paths to the files you'd like to process with your job array separated by a new line. This is easy to accomplish if all your files are located in a single directory called let's say 'my_dir'+Use the command //[[https://slurm.schedmd.com/sbatch.html|sbatch]]// to submit the job to SLURM. You simply use the path to the script you just created for this
  
 <code> <code>
-cd my_dir; +sbatch /path/to/<slurm_script>.sh
- +
-ls -d $PWD/*> seedfile.txt+
 </code> </code>
  
-You can generate an example seedfile by copying the information below and pasting it into a file calles //seedfile.txt//. +After your script was send to the cluster, SLURM will automatically create a file in the directory from which you issued the ''sbatch'' command. This file will contain all information that would be printed to the terminal when calling your command in the local terminal. This directory will also be used as the current directory for all relative paths in your slurm script.
-<code> +
-ANOGA@7165@1.fa +
-AQUAE@224324@1.fa +
-ARATH@3702@1.fa +
-ASPFU@330879@1.fa +
-BACSU@224308@1.fa +
-BACTN@226186@1.fa +
-BATDJ@684364@1.fa +
-BOVIN@9913@1.fa +
-BRADU@224911@1.fa +
-BRAFL@7739@1.fa +
-CAEEL@6239@1.fa +
-CANAL@237561@1.fa +
-CANLF@9615@1.fa +
-CHICK@9031@1.fa +
-CHLAA@324602@1.fa +
-</code> +
- +
-===== Job submission ===== +
- +
-Use the command //[[https://slurm.schedmd.com/sbatch.html|sbatch]]// to submit the job to SLURM +
- +
-<code> +
-sbatch /path/to/script.slurm +
-</code>+
  
 +Your script will run using your current (Anaconda) environment meaning that you can use all programs which are in the environment from which you issue the ''sbatch'' command.
 ===== Job status ===== ===== Job status =====
  
Line 102: Line 67:
 </code> </code>
  
 +<wrap important>Make sure that you double check that you cancel the correct job id</wrap>
 ===== Information about slurm settings on our system ===== ===== Information about slurm settings on our system =====
 Use the command [[https://slurm.schedmd.com/sinfo.html|sinfo]] to see the available queues/partitions and the assigned computers Use the command [[https://slurm.schedmd.com/sinfo.html|sinfo]] to see the available queues/partitions and the assigned computers
Line 108: Line 74:
 sinfo sinfo
 </code> </code>
 +
 +===== More info of what is going on ===== 
 +To open a window with a whole array of informations about the cluster like current running jobs/partitions/reservations/available resources and more run
 +<code> sview</code>
 ===== Common mistakes ===== ===== Common mistakes =====
 This is a non-exhaustive list of issues and mistakes in the context of SLURM This is a non-exhaustive list of issues and mistakes in the context of SLURM
Line 116: Line 86:
   * your job is in pending state forever -> it might be that non of the computers in the queue has the capacity((e.g. number of cpus, amount of memory)) that you specified    * your job is in pending state forever -> it might be that non of the computers in the queue has the capacity((e.g. number of cpus, amount of memory)) that you specified 
  
 +<WRAP tabs>
 +   * [[general:computerenvironment]]
 +</WRAP>