Our batch servers, or batch cluster, comprises roughly 400 CPU cores, each with 2 gigabytes of RAM allocated per core.
Use our batch servers to process longer running jobs that have parallel components.
The Automated Condor Submission script makes the task of running jobs using the batch servers easier and more intuitive. The process it automates is described in Getting Started with Batch Processing. This script negotiates all job scheduling; it constructs the appropriate submit file for your job, and calls the condor_submit function. To use this utility you need a program to run. The format for using this script is:
In addition, the script can notify you when your job is done via email so you do not have to check the queue constantly using
condor_q. In future releases, the script also will be able to keep usage data so administrators can track overall performance.
The script can be run in two ways, interactively or from the command line. When running interactively, the script prompts you for the values required to run the batch job. If you supply arguments on the command line, these arguments are used in addition to default values for any values you do not supply.
-s, --submitfile FILE
-x, --executable FILE
-a, --arguments ARGS
-i, --input [FILE|PATT]
-o, --output [PATT]
-e, --error [PATT]
-l, --log [PATT]
-n, --iterations NUM
You have a compiled executable (named foo) that takes a data set and does some analysis. You have five different data sets to run against (named data.0, data.1 ... data.4). You want to save the submit file and be notified when the job is done.
condor_submit_util -x foo -i "data" -k -N
You have an R program that has some random output. You want to run it 10 times to see the results.
condor_submit_util -i random.R -n 10
You have an R program that will take a long time to complete. You only need to run it once, but you want to be notified when it is done.
condor_submit_util -i long.R -n 1 -N
-l, these options are considered base names for the implied files. The actual file names are created with a numerical extension tied to its condor process number (0 indexed). This means that if you execute
condor_submit_util -o "out" -n 3, three output files named
out.2 are created.
-i, the script first checks to see if the name supplied is an actual file on disk, if not it uses the argument as a base name, similar to
This guide helps you to set up and execute batch processing. It will walk you through an example batch submission as well as offer details to each step in the Batch Submission Workflow.
The default environment for batch processing supports the R language, but you can submit any executable file or script compatible with the Linux environment.
Note: All batch processing is performed in the RCE. You must have an RCE account to use the batch servers. See Research Computing Environment for more information.
We provide the Automated Condor Submission script, condor_submit_util, to automate the process of submitting and executing jobs to the batch servers' Condor system. You also can submit your jobs manually.
Use this guide to perform the following: