A PBS Survival Guide
A guide nobody asked for, but everyone eventually needs.
Welcome to Walltime Chronicles — a personal documentation project capturing all the weird, confusing, and occasionally infuriating challenges I have faced while using QUT's HPC systems (powered by PBS).
If you have ever:
- Submitted a job that instantly failed for reasons only known to the cluster gods,
- Fought with
walltimelimits like you were disarming a bomb, - Wondered why your script runs perfectly except when submitted through
qsub, - Or simply stared into the abyss of PBS logs…
Then this is the right place.
Crash Course Café — onboarding for newcomers
New to Aqua? The Crash Course Café takes you from your first SSH to running your own work on the cluster, in eight lessons.
Where to Start
Pick your entry point. Each card is one shipped page.
-
New to QUT Aqua?
Start with the prerequisites checklist, then take the lessons in order.
-
Picking hardware?
Three-tier field guide to every node on Aqua — H100s, A100 MIG slices, large-memory boxes, the watchdog queue. Tells you which queue backs which silicon.
-
Sizing walltime?
The 2× rule, queue ceilings, scaling rules by workload (CPU complexity classes, GPU architectures), and a worked ResNet finetune example.
-
Writing batch scripts?
Copy-paste recipe that sequentially runs N experiments inside one PBS job — timestamped names, log banners, and
qsub. -
Inspecting past jobs?
A user-friendly alternative to
qjobs -x— walltime usage %, CPU saturation, GPU utilisation, memory footprint, all in one tasting-notes table. -
macOS remote-dev pain?
Survival paths when VS Code Remote-SSH isn't an option, plus the standalone guide to taming
.DS_Storeand._*files when Finder mounts the HPC. -
Want to poke the scheduler?
The site's interactive pieces at full size: the score explorer, and the backfill animation on all thirteen H100 nodes.
What You'll Find Here
- A crash course for newcomers — eight lessons that take you from your first SSH to running your own work on Aqua.
- Mystery errors & how I solved them — real issues with real fixes (and real frustration).
- Tips, workarounds, and gotchas — things that should have been in the official docs.
- PBS scripts & snippets — copy-paste-friendly templates with helpful comments.
- Experiments & mistakes — because learning is messy.
What You'll Not Find Here
- A Linux course — the prerequisites checklist lists the shell commands the lessons assume and links QUT's own shell tutorial. Everything else here assumes you can use a terminal.
- A PBS reference — the course teaches the directives you will actually use. For the full list, start with the official docs1.
- HPC mastery — the course gets you to a working loop; the rest is a collection of "oh, that's why it failed!" moments. Neither claims to cover the whole field.
- System administration — I'm not your sysadmin. If you need to configure the cluster, that's above my pay grade.
- Debugging your code — your Python script is throwing errors? That's between you and your debugger. I'm here for PBS-related mysteries only.
- Performance optimisation — want to make your code run faster? I'll share some tricks that might work, but no promises. This isn't a magic wand for your algorithms, just PBS-specific tweaks that occasionally make things less slow.
Disclaimer
Before you start
Before you start using this guide, please read the QUT HPC "Aqua" Official Documentation1 and Altair's PBS Pro Reference Guide thoroughly.
This is not an official QUT HPC "Aqua" guide. It is just my (occasionally ranty) collection of notes, meant to help others avoid the black holes I fell into. Use at your own risk — and sanity.
Happy queueing, and may your jobs always run on the first try.
Contributing
If you have any suggestions or corrections, please feel free to open an issue or a pull request on GitHub.
License
This project is licensed under the MIT License. See the LICENSE file for details.