Time-stamp: <2001-02-28 21:16:53 gokieli>

Hints & Tips for the use of LSF in DELPHI

LSF in now the only batch scheduler to be used for DELPHI analysis jobs on the DELSHIFT cluster. In this page we describe the recommended way of submitting your jobs to this scheduler, and the commands to inspect your jobs.

Job submission:

To submit an LSF batch job we recommend you to use the DELPHI product delsub rather than the LSF command bsub. delsub is a wrapper around this bsub command, and hides some of its nasty features. A man-page exists. More on delsub can be found in the section Why delsub?.
The options you want to give to your batch job (like a job name, or a batch priority queue) can be embedded in the script itself. These are the recommended options for an imaginary script myjob.csh:
#!/bin/csh
#
# Load Sharing Facility (LSF) options
#
#BSUB -L /bin/csh               # job shell
#BSUB -J myjob.csh              # job title
#BSUB -o myjob.out              # output file (stdout and stderr merged)
#BSUB -q xx_8nm                 # default job queue
#BSUB -r                        # re-runnable (if the machine crashes)
#
and possibly also:
#BSUB -N                        # mail me when the job finishes
#BSUB -B                        # mail me when the job starts
There are some options we urge you NOT to use! E.g. the options that limit file size and CPU usage are for most jobs not needed and in fact quite often they merely create confusion.
All these options can also be specified on the command line. Note that command line options take preference over embedded options.
One of the important advantages of delsub is that it allows you to submit a script with parameters of its own to LSF.

Available batch queues on DELSHIFT:

 LSF queue     description
 ==========   =============

 xx_8nm       Short queue      (8 mins), intended for tests
 xx_1nh       Medium queue     (1 hr)                
 xx_8nh       Long queue       (8 hrs)
 xx_1nd       Very Long queue (24 hrs), with lower priority
 xx_deltape   Special queue for copying data to tape
These queues are available both from DXPLUS and HPPLUS clusters, but not from LXPLUS.

Other available batch queues:

DELPHI users may be interested also in the following queues:

np
Queue on the interactive clusters (DXPLUS, HPPLUS) open only during the non-prime time (nights, weekends). Implied time limit is 8 hrs. Jobs which did not finish before morning will be suspended and then resumed in the evening. Jobs only run when the interactive load is low, but they can take 100% of the CPU. Please note that on DXPLUS the connection to our disk server is as good as the connection from DELSHIFT, so its 5*2 CPU can add substantial power.

8nm, 1nh, 1nd, 1nw
Queues on LXBATCH available from LXPLUS interactive cluster. The machines are running under the Linux system. Implied time limits are respectively 8 mins, 1 hour, 24 hrs, 168 hrs.
Please note: these machines are very powerful and at the moment rather empty. Their use is very recommended and normal analysis jobs run there without problems, provided they do not use any language extensions and the script is machine-independent (using the environmental variables $XLF77, $ADDCOMP, $ADDLIB, ...).

lxcsf_8nm, lxcsf_1nd, lxcsf_1nw
Queues on the LinuX Central Simulation Facility, available from LXPLUS. The machines are running under the Linux system. Implied time limits are respectively 8 mins, 24 hrs, 168 hrs.

Please note that at the moment the old CSF cluster is not available to DELPHI users anymore. Those who need to use HP machines (because of old executables), should use the np queue on HPPLUS - see above.

Job control:

The most common commands to control your submitted jobs are listed in this table:
LSF       Function
===       ========
          
bjobs     to see the status of your batch jobs
bpeek     to look at the output of a running job
bkill     to delete a batch job
Please note that these commands must be used always from the same cluster, from which the job has been submitted.

Advanced features: Job arrays

LSF offers the possibility to submit multiple jobs in one go, by using the so-called "job array". This is useful when you want to submit many batch jobs that differ only slightly. For example: Assume you want to read the first 20 files on cassette EK1402, one file per job, so 20 batch jobs in total, with your script "myjob.csh". You can submit these 20 jobs by a single line:
     delsub -J "myjob[1-20]" -o myjob.out.%I myjob.csh
The important points here:
  • the -J "myjob[1-20]" option will lead to 20 jobs, with indices from 1 to 20
  • the double quotes " " around the job name will avoid the Unix shell interpretation of the special character [ !!
  • the %I in the output file option -o myjob.out.%I will be substituted by this index when the output file is returned, so 20 output files myjob.out.1, myjob.out.2 , .... will be created.
  • these options can of course also be embedded in your script.

    In your script myjob.csh you can construct the PDLINPUT file like this, using the environment variable LSB_JOBINDEX (which is defined only in a job array context!):

         if ( $?LSB_JOBINDEX ) then # Is this a job in a job array?
             echo "VID=EK1402,NUMBER=${LSB_JOBINDEX}" > PDLINPUT
         else # not a job array? then just the first file
             echo "VID=EK1402,NUMBER=1" > PDLINPUT
         endif
    
    A more realistic, and slightly more complicated example: if you want to read the first 200 files of FATMEN nickname XSHORT97_E2 in 20 jobs of 10 files, construct your PDLINPUT like:
         if ( $?LSB_JOBINDEX ) then # Is this a job in a job array?
             @ last  = ${LSB_JOBINDEX} * 10
             @ first = ${last} - 9
             echo "FAT=XSHORT97_E2/C${first}-${last}" > PDLINPUT
         else # not a job array? then just the first file
             echo "FAT=XSHORT97_E2/C1" > PDLINPUT
         endif
    
    Of course you can use this variable as well to make sure your output files are uniquely named:
         rfcp ww.ntp shd01:${DELPHI_PAW}/XSHORT97_E2_job${LSB_JOBINDEX}.ntp
    
         rfcp myfile.rz ${CORE_SCRATCH}/myfile.${LSB_JOBINDEX}.rz
    
    So: you can the use the %I in the (embedded) LSF options, and the environment variable LSB_JOBINDEX in your script

    Other commands: to bpeek or bkill the individual jobs, do like this:

         bkill -J "myjob[16]"
         Job <34416[16]> is being terminated
    
    Note again the use of double quotes to escape the special character "["

    bkill 34416 will kill ALL jobs in this job array!


    Advanced features: delkill

    The LSF command bkill can be used to send signals to running jobs. In fact, this signal is sent to all processes belonging to the LSF job. When you just want to kill a running job, this is fine. In special cases however, this may not be good enough.

    It may happen that you want to send a special user signal to the executable only, and not to the script that has created it. For example, your SKELANA job is producing an Ntuple, and for some reason you want to interrupt this, but still get whatever has been produced until now. So you want to stop just the executable, and let the rest of your script take care of the distribution of the output (rfcp, ftp, ...). If you send a signal called USR1 to the SKELANA executable, it will be trapped and will finish immediately. To send this signal to just the executable only, you need delkill.


    Why delsub?

    delsub is more versatile (and robust) than the native LSF command bsub. It you want to submit a script with embedded options, and with private parameters to LSF, it cannot be done!

    For example: you are analysing LEP-2 data, the FATMEN nickname of the data set is a parameter for your script myjob.csh, and you would like to submit

    myjob.csh XSDST97_E2/C1-5
    with embedded options as stated before.

    Now embedded LSF options are only considered by bsub if you redirect the standard input:

    bsub < myjob.csh 
    But this means that you cannot specify your parameter XSDST97_E2/C1-5!

    delsub takes care of this problem, and allows you to submit your job as

    delsub myjob.csh XSDST97_E2/C1-5

    More LSF information:

    For more information on LSF, and some extra examples, have a look at the CERN LSF documentation.
    As always, questions, complaints, remarks:
    delphi-core@cern.ch