Interactive job; the shell opens on a compute node
qsub -W depend=afterok:<job-id> <script>.pbs
Submit, held until that job ends with 0
qstat -u $USER
Your queued and running jobs
qstat -xu $USER
Yours including finished ones
qstat -f <job-id>
Every attribute of one job
qstat -xf <job-id>
The same after it finished; kept about four days
qstat -f <job-id> \| grep comment
Why a queued job has not started
qstat -u $USER -Esw
Comments for all your queued jobs at once
qstat -T
Estimated start; top-ranked jobs only, your own only
qstat -t '<job-id>[]'
Subjobs of an array; add x for finished ones
qstat -fw '<job-id>[]' \| grep array_state_count
An array's tally by state
qstat -Q
Queues and their totals
qjobs
eResearch's own summary of your jobs
qdel <job-id>
Cancel; resets the queue wait the job had earned
qdel '<job-id>[3]'
Cancel one subjob of an array
qalter -l walltime=04:00:00 <job-id>
Change a queued job's walltime; users can only lower it
qhold <job-id> / qrls <job-id>
Hold and release your own job
pbsnodeinfo
Every node's current use
qmgr -c "print server" \| grep job_sort_formula
The formula that ranks the queue
Software
Command
What it does
module spider <name>
Search for software and its versions
module load <name>/<version>
Load an exact version
module list
What is loaded now
module purge
Unload everything
uv init --bare --pin-python --python 3.13
New project: pyproject.toml and .python-version
uv add <pkg>
Add a dependency; updates uv.lock and installs
uv sync --frozen
Rebuild .venv exactly as uv.lock says
uv run <command>
Run inside the project environment, no activation
conda init
Once, after installing Miniforge, then open a new shell
apptainer pull docker://<image>
Fetch a container image
apptainer exec <image>.sif <command>
Run one command inside it
apptainer shell <image>.sif
Open a shell inside it
Directives
Directive
Effect
#PBS -N <name>
Job name, and the prefix of the log files
#PBS -l select=1:ncpus=4:mem=8GB
Resources, per chunk
#PBS -l walltime=01:00:00
Hard limit; PBS kills the job at it
#PBS -l place=pack
Chunks on one node; scatter spreads them, group=cpu_id keeps one CPU kind
#PBS -P ABCDEF1234
Project code
#PBS -m abe
Mail on begin, end and abort
#PBS -M <address>
Send that mail somewhere other than the default
#PBS -j oe
One log file instead of .o and .e
#PBS -J 1-8
Array of 8 subjobs; 1-100:2 steps, 1-100%10 caps how many run
#PBS -c w=30
Requeue at the walltime; USR1 to the shell about every 30 minutes
-W depend=afterok:<job-id>
As a #PBS line with a literal id; on the command line when the id is in a variable
The select line
Part
Values on Aqua
select=<n>
Chunks; ncpus and mem are per chunk, not totals
ncpus=<n>
Cores
mem=<n>GB
Memory, always with a unit
ngpus=<n>
GPUs, or MIG slices on an interactive queue
gpu_id=
H100, A100, or omit for either
cpu_id=
any, AMD-25-17, AMD-25-1, AMD-23-49, Intel-6-143
mpiprocs=<n>
MPI ranks per chunk
+
Not supported on Aqua
Queues
Queue
Walltime
Memory
Cores
GPUs
cpu_batch_exec
10 min to 48 h
1 GB to 16384 GB
1 to 2048
—
cpu_inter_exec
10 min to 12 h
1 GB to 34 GB
1 to 8
—
gpu_batch_exec
10 min to 48 h
1 GB to 1920 GB
1 to 256
1 to 8
gpu_inter_exec
10 min to 12 h
1 GB to 68 GB
1 to 12
1 to 2
cpu_batch_exlm
10 min to 48 h
1479 GB to 6015 GB
1 to 180
—
You ask for
PBS routes it to
-I
An interactive queue
mem above 1.5 TB
cpu_batch_exlm
ngpus of 1 or more
A GPU queue
Anything else
cpu_batch_exec
Interactive limits apply across all of your interactive jobs at once, not per job.
Nodes
Tier
Hostnames
Cores
RAM
GPUs
CPU batch
cpu1n002 to cpu1n050
188
1478 GB
—
CPU interactive
cpu1n001
376 logical
1478 GB
—
Large memory
mem1n001
180
6014 GB
—
GPU H100 batch
gpu1n002 to gpu1n014
168
974 GB
4 × H100 80 GB
GPU A100 batch
5 nodes
120 to 124
974 GB
8 × A100 40 GB
GPU interactive
2 nodes
168 / 124
974 GB
44 MIG slices
One GPU's fair share of its host is about 42 cores and 240 GB on an H100, about 15 cores and 120 GB on an A100.
Filesystems
Path
Backed up
Speed
For
/home/$USER
Yes
Good
Scripts, configs, small personal data
/scratch/$USER/
No; swept after 30 days idle
Ultra-fast
Active analysis, heavy reads and writes
/work/<project>/
Yes
Good
Shared project data; requested by ticket
/work/datasets/
Yes
Good
Datasets shared across the cluster
$TMPDIR
No; deleted with the job
Ultra-fast
Per-job intermediates, copied back before the end
Millions of small files in /home slow Lustre for everyone; unpack and build on /scratch.
Environment variables
Variable
Holds
$PBS_O_WORKDIR
The directory qsub ran in
$PBS_JOBID
This job's id
$PBS_JOBNAME
The -N name
$NCPUS
Cores PBS gave the job
$TMPDIR
This job's scratch directory
$CUDA_VISIBLE_DEVICES
The GPUs PBS gave the job
$PBS_ARRAY_INDEX
This subjob's index
$PBS_ARRAY_ID
The parent array's id
$PBS_NODEFILE
File listing the nodes the job is running on
$PBS_QUEUE
The queue PBS routed the job to
$OMP_NUM_THREADS
Threads per process, if you set it
Job states
State
Meaning
Q
Queued, waiting to run
R
Running
H
Held: by qhold, by a dependency, or by PBS after repeated failures
B
An array with at least one subjob started
E
Exiting after running
F
Finished, whether it succeeded, failed or was deleted
S
Suspended by PBS
M
Moved to another server
T
In transition
W
Waiting for a -a start time
Exit statuses
Status
Meaning
0
The last command succeeded; read .e anyway
1, 2
Your program failed; the end of .e says why
-18
Aqua's checkpoint hook requeued it at the walltime; expected with -c
-20
The node failed; PBS queued the job again
-29
PBS stopped it at the walltime
137
PBS stopped it at the memory limit
143
Someone ran qdel
Queue comments
Comment
Meaning
Insufficient amount of resource: ncpus / mem / ngpus
Nothing that size is free yet; wait, or ask for less
Insufficient amount of resource: qlist
No node tagged for your queue had room this cycle
Queue gpu per-user limit reached on resource ngpus
Your own running jobs hold the per-user GPUs
Job would cross dedicated time boundary
It would still be running at the next maintenance window
Queue long job limit has been reached
Too many of your jobs in the long queue
Scheduler user rate limit. Deferring until next scheduler run
You submitted faster than the scheduler accepts
Can Never Run: Insufficient amount of server resource
A licence the software needs is not free
Job held, too many failed attempts to run
PBS failed to start it repeatedly; you cannot release this one
Rejected at submission
Message
Cause
Illegal attribute or resource value select.mem
Memory missing a unit, in the wrong unit, or negative
Illegal attribute or resource value Resource_List.ncpus
Something wrong with ncpus or its select line
You cannot explicitly request 0 gpus when specifying gpu_id
Naming a gpu_id means asking for at least one GPU
cpu_id=... must be one of {...}
Not a cpu_id Aqua has; the message lists the set
ROUTER: Unable to initialise node
+ used in the select line
Failed while running
Message
Cause
bad interpreter: No such file or directory
Windows line endings; run dos2unix
module: command not found
Shebang missing, misspelled, or with a blank line above it; use #!/bin/bash -l
conda: command not found
Inspect the shell's conda initialization and merge the required setup into the existing configuration. Preserve customizations; back up .bashrc and obtain explicit approval before any overwrite.
Lmod has detected the following error
No such module name or version; check module spider
Bus error (core dumped)
Touched memory it was not given; raise mem
Illegal instruction
Self-compiled software whose vector instructions this node lacks
installed in '/home/<username>/.local/bin' which is not on PATH
Add that directory to PATH in .bashrc
Templates
Batch job
One program, run unattended, with the job's own record kept beside the output.
#!/bin/bash#PBS -N my_job#PBS -l select=1:ncpus=4:mem=8GB#PBS -l walltime=01:00:00#PBS -P ABCDEF1234#PBS -m abeset-e
cd"$PBS_O_WORKDIR"status=0uvrunpythonmy_program.py||status=$?qstat-xf"$PBS_JOBID">"resource_usage_$PBS_JOBID"||echo"Warning: job accounting could not be saved">&2exit"$status"
For a GPU: #PBS -l select=1:ncpus=12:ngpus=1:mem=64GB:gpu_id=H100
Array
One script, many runs. The index picks each run's input and output. A rerun skips a member only when its final output and completion marker both exist.
This assumes the program exits successfully only after completing and validating its work; the nonempty-file check alone does not validate scientific results. It writes to a temporary path on the same filesystem, publishes the final file only after success, and then creates the marker. Failed runs retain their partial file for diagnosis. Reuse markers only for retries with unchanged inputs and configuration, use a new results directory for changed work, and avoid concurrent executions of the same member.
Turning the index into work
f=$(sed -n "${PBS_ARRAY_INDEX}p" files.txt)
A file from a list
args=$(sed -n "${PBS_ARRAY_INDEX}p" settings.txt)
A row of settings
from=$(( (PBS_ARRAY_INDEX - 1) * 10 + 1 ))
A slice of ten
Chained jobs
Stages that must happen in order, each submitted now but starting only when the one before it ends.
A=$(qsubstage_a.pbs)qsub-Wdepend=afterok:$Astage_b.pbs# only if A ends with 0qsub-Wdepend=afterany:$Areport.pbs# when A ends, either way
A dependent job sits in H until its condition is met, and the server deletes it if the condition can never be met.
Checkpointed job
Work that outlives one walltime. PBS takes the job back at the limit and runs the script again, so the program has to resume itself.
PBS requeues at the walltime and runs the script again from the top, up to 21 attempts. The program must save as it goes, resume from its own checkpoint, and aim at a total rather than a duration.