Lesson 3: Working Interactively
Mission Statement
"Interactive is where you develop. Batch is where you run." 🧪
Lesson 1's interactive job was a round-trip with nothing in the middle. This lesson puts real work there: request a shell sized for it, run a training script on a compute node by hand, keep the session alive if your connection drops, and recognise when the work should become a batch job.
📋 What You'll Accomplish
By the end of this 15–20 minute lesson, you'll have:
- Sized an interactive request — cores, memory and walltime, inside the queue caps
- Installed PyTorch and fetched
train_mnist.pywith its data, on the login node - Run it by hand on a compute node inside
qsub -I - Kept a session alive across a dropped connection with
tmux - Known when to stop being interactive and let Lesson 4 take over
You need Lesson 2's environment
Everything below assumes ~/hello-aqua/.venv (uv) or the hello-aqua env (Miniforge / micromamba) from Lesson 2 exists and works. If not, do the Test Drive there first; it takes two minutes.
⚖️ Part 1: Size the request (~3 min)
Lesson 1 asked for one core and one gigabyte because the point was to arrive somewhere. This time the point is to work, so the request has to fit the work, and it has to fit the queue.
PBS routes qsub -I to one of two interactive queues by whether you asked for a GPU:
| Queue | You get | Per job | Per user, all your interactive jobs | Walltime |
|---|---|---|---|---|
cpu_inter_exec |
a shell on cpu1n001, the single interactive CPU node |
1–8 cores, 1–34 GB | 8 cores, 34 GB | ≤ 12 h |
gpu_inter_exec (:ngpus=1) |
a shell plus MIG slices, ~10 GB of an H100 or ~20 GB of an A100 each, not whole cards | 1–12 cores, 1–68 GB, 1–2 slices | 12 cores, 68 GB, 2 slices | ≤ 12 h |
This lesson stays on CPU. For the GPU queue, Know Your Nodes explains the slices and Recipe 7 in Walltime by Recipe has the ready-made line.
The caps come from the queues themselves (qstat -Qf cpu_inter_exec); the eResearch queue page1 rounds them to 32 and 64 GB. Ask for more than the cap and the job is rejected at submit time.
Interactive jobs hold what they asked for until you leave
A batch job frees its node when the script ends. An interactive job frees it when you exit or the walltime runs out, and every user on Aqua shares the one interactive CPU node. Request what the session needs, not the cap, and one or two hours of walltime, not twelve.
This lesson's request
ncpus=4→ the script reads$NCPUSand uses every core PBS gives itmem=8GB→ memory for the whole job, all processes together; enough for the script with room to turn the knobs upwalltime=01:00:00→ an hour at the keyboard, not the 12 h cap
You submit this in Part 3. Part 2 comes first, because installing and downloading belongs on the login node.
📦 Part 2: Get the script, its dependency, and its data (~4 min)
train_mnist.py trains a small network to recognise handwritten digits from MNIST, the 70,000-image dataset every machine-learning course starts with. It finishes in under a minute on a few cores, reads $NCPUS for its thread count, and writes its runtime and peak memory to results.json.
It needs only PyTorch. Install that into your Lesson 2 environment, download the script, and let the script fetch its data (12 MB, once). All of this is network and disk work, not compute, so the login node is the right place for it:
PyTorch's CPU build lives on its own package index (the PyPI one is the CUDA build, gigabytes of it, only useful on a GPU node). Following uv's PyTorch guide, tell the project about that index and use it for torch only. Append to ~/hello-aqua/pyproject.toml:
[[tool.uv.index]]
name = "pytorch-cpu"
url = "https://download.pytorch.org/whl/cpu"
explicit = true
[tool.uv.sources]
torch = { index = "pytorch-cpu" }
cd ~/hello-aqua
uv add torch # about 800 MB installed
# The script, straight from this site's repository
wget https://raw.githubusercontent.com/ZhipengHe/Walltime-Chronicles/main/docs/tutorials/scripts/train_mnist.py
# Fetch the data now, so the compute node never has to. --epochs 0 downloads and exits.
uv run python train_mnist.py --epochs 0
uv add puts torch>=2.14.0 under dependencies, and uv.lock records the exact CPU build from that index while everything else, pandas included, keeps coming from PyPI. uv sync --frozen rebuilds all of it.
Same-filesystem rule
If you followed Lesson 2's tip and moved UV_CACHE_DIR to /scratch, this venv on /home is now on a different filesystem from the cache, and uv will warn Failed to hardlink files; falling back to full copy. It still works, just slower. The fix is to keep cache and venv together: uv on Aqua.
Expected output of the data fetch
host aquarius02
device cpu threads 1 seed 0
download https://ossci-datasets.s3.amazonaws.com/mnist/train-images-idx3-ubyte.gz
download https://ossci-datasets.s3.amazonaws.com/mnist/train-labels-idx1-ubyte.gz
download https://ossci-datasets.s3.amazonaws.com/mnist/t10k-images-idx3-ubyte.gz
download https://ossci-datasets.s3.amazonaws.com/mnist/t10k-labels-idx1-ubyte.gz
data 60,000 training images, 10,000 test (188 MB)
done test accuracy None in 19.8s -> results.json
Four files land in ~/hello-aqua/data/mnist/. Accuracy is None because nothing was trained.
Download the script, or read it here:
train_mnist.py
"""Train a small network to recognise handwritten digits (MNIST).
Its knobs map to what a job asks PBS for: epochs to walltime, samples and
width to memory, threads or a GPU to the hardware request. The seed fixes
shuffling and initialisation: one value repeats a run, different values give
different runs. A checkpoint lets a long run resume.
python train_mnist.py # ~half a minute on one core
python train_mnist.py --epochs 20 # longer
python train_mnist.py --width 4096 # heavier on memory and compute
python train_mnist.py --device cuda # use the GPU PBS gave you
python train_mnist.py --seed 7 --out run7.json
python train_mnist.py --checkpoint ckpt.pt # resumes if the file exists
It needs only PyTorch. The data (60,000 training and 10,000 test images,
about 12 MB compressed) is downloaded once into --data-dir and read straight
from the original IDX files; no torchvision, no numpy.
"""
import argparse
import gzip
import json
import os
import sys
import time
import urllib.request
import torch
from torch import nn
MIRROR = "https://ossci-datasets.s3.amazonaws.com/mnist/"
FILES = {
"train_images": "train-images-idx3-ubyte.gz",
"train_labels": "train-labels-idx1-ubyte.gz",
"test_images": "t10k-images-idx3-ubyte.gz",
"test_labels": "t10k-labels-idx1-ubyte.gz",
}
def parse_args():
"""Parse the command line and reject sizes the training loop cannot run with."""
p = argparse.ArgumentParser(description=__doc__.splitlines()[0])
p.add_argument("--data-dir", default="data/mnist", help="where the four MNIST files live (downloaded if missing)")
p.add_argument("--samples", type=int, default=60_000, help="training images to use, up to 60,000 (memory)")
p.add_argument("--width", type=int, default=1024, help="hidden layer width (memory and compute per step)")
p.add_argument("--epochs", type=int, default=8, help="passes over the data (runtime)")
p.add_argument("--batch", type=int, default=128, help="images per optimisation step")
p.add_argument("--threads", type=int, default=0, help="CPU threads; 0 = PBS's $NCPUS or 1")
p.add_argument("--device", default="auto", choices=["auto", "cpu", "cuda"])
p.add_argument("--seed", type=int, default=0, help="controls shuffling and initialisation")
p.add_argument("--checkpoint", default=None, help="path to save after each epoch and resume from")
p.add_argument("--out", default="results.json", help="where the summary is written")
args = p.parse_args()
if not 1 <= args.samples <= 60_000:
p.error("--samples must be between 1 and 60000")
if args.width < 1 or args.batch < 1:
p.error("--width and --batch must be at least 1")
if args.epochs < 0 or args.threads < 0:
p.error("--epochs and --threads cannot be negative")
return args
def fetch(data_dir):
"""Download the four MNIST files once. Network work: do it on a login node."""
os.makedirs(data_dir, exist_ok=True)
for name in FILES.values():
path = os.path.join(data_dir, name)
if not os.path.exists(path):
print(f"download {MIRROR}{name}")
urllib.request.urlretrieve(MIRROR + name, path)
def read_idx(path):
"""Read one IDX file (the original MNIST format) into a uint8 tensor."""
with gzip.open(path, "rb") as f:
data = f.read()
ndim = data[3] # third byte of the magic number: 1 for labels, 3 for images
header = 4 + 4 * ndim
shape = [int.from_bytes(data[4 + 4 * i : 8 + 4 * i], "big") for i in range(ndim)]
return torch.frombuffer(bytearray(data[header:]), dtype=torch.uint8).view(*shape)
def load(data_dir, samples, device):
"""Return train and test tensors on `device`, using the first `samples` training images."""
fetch(data_dir)
x_train = read_idx(os.path.join(data_dir, FILES["train_images"]))[:samples]
y_train = read_idx(os.path.join(data_dir, FILES["train_labels"]))[:samples]
x_test = read_idx(os.path.join(data_dir, FILES["test_images"]))
y_test = read_idx(os.path.join(data_dir, FILES["test_labels"]))
# Pixels to floats in [0, 1], flattened to 784 per image. This conversion
# is where the memory goes: 60,000 x 784 x 4 bytes is about 188 MB.
to_float = lambda t: t.reshape(t.shape[0], -1).float().div_(255).to(device)
return to_float(x_train), y_train.long().to(device), to_float(x_test), y_test.long().to(device)
def pick_device(name):
"""Resolve --device: `auto` takes a GPU if one is visible, `cuda` insists on one."""
if name == "cuda" or (name == "auto" and torch.cuda.is_available()):
if not torch.cuda.is_available():
sys.exit("ERROR: --device cuda requested but no GPU is visible (did you ask PBS for ngpus=1?)")
return torch.device("cuda")
return torch.device("cpu")
def peak_memory_mb():
"""Peak resident memory of this process, in MB. Linux and macOS only."""
try:
import resource
except ImportError: # Windows
return None
kb = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
return kb / 1024 if sys.platform != "darwin" else kb / (1024 * 1024)
@torch.no_grad()
def accuracy(model, x, y, batch=1000):
"""Fraction of `x` classified correctly, evaluated in batches to bound memory."""
correct = 0
for start in range(0, x.shape[0], batch):
correct += (model(x[start : start + batch]).argmax(1) == y[start : start + batch]).sum().item()
return correct / x.shape[0]
def main():
"""Train, report per-epoch accuracy, and write the JSON summary."""
args = parse_args()
started = time.time()
# PBS exports NCPUS inside a job; using exactly that many threads is what
# makes the job's cpupercent match its request.
threads = args.threads or int(os.environ.get("NCPUS", "1"))
torch.set_num_threads(threads)
torch.manual_seed(args.seed)
device = pick_device(args.device)
print(f"host {os.uname().nodename if hasattr(os, 'uname') else 'unknown'}")
print(f"device {device} threads {threads} seed {args.seed}")
x_train, y_train, x_test, y_test = load(args.data_dir, args.samples, device)
print(f"data {x_train.shape[0]:,} training images, {x_test.shape[0]:,} test ({x_train.numel() * 4 / 1e6:.0f} MB)")
model = nn.Sequential(
nn.Linear(784, args.width),
nn.ReLU(),
nn.Linear(args.width, 10),
).to(device)
optimiser = torch.optim.Adam(model.parameters(), lr=1e-3)
loss_fn = nn.CrossEntropyLoss()
# Resume if a checkpoint exists. Lesson 8 relies on this: a job that runs
# out of walltime can be resubmitted and carries on from the last epoch.
first_epoch = 0
if args.checkpoint and os.path.exists(args.checkpoint):
state = torch.load(args.checkpoint, map_location=device, weights_only=True)
model.load_state_dict(state["model"])
optimiser.load_state_dict(state["optimiser"])
first_epoch = state["epoch"] + 1
print(f"resumed from {args.checkpoint} at epoch {first_epoch}")
test_acc = float("nan")
for epoch in range(first_epoch, args.epochs):
epoch_started = time.time()
model.train()
order = torch.randperm(x_train.shape[0], device=device)
for start in range(0, x_train.shape[0], args.batch):
idx = order[start : start + args.batch]
optimiser.zero_grad()
loss = loss_fn(model(x_train[idx]), y_train[idx])
loss.backward()
optimiser.step()
model.eval()
test_acc = accuracy(model, x_test, y_test)
print(f"epoch {epoch + 1:>3}/{args.epochs} loss {loss.item():.4f} test accuracy {test_acc:.4f} {time.time() - epoch_started:5.1f}s")
if args.checkpoint:
torch.save(
{"model": model.state_dict(), "optimiser": optimiser.state_dict(), "epoch": epoch},
args.checkpoint,
)
elapsed = time.time() - started
summary = {
"seed": args.seed,
"samples": int(x_train.shape[0]),
"width": args.width,
"epochs": args.epochs,
"device": str(device),
"threads": threads,
"test_accuracy": None if test_acc != test_acc else round(test_acc, 4), # None when no epoch ran
"elapsed_s": round(elapsed, 1),
"peak_rss_mb": None if peak_memory_mb() is None else round(peak_memory_mb()),
"job_id": os.environ.get("PBS_JOBID"),
}
with open(args.out, "w") as f:
json.dump(summary, f, indent=2)
print(f"done test accuracy {summary['test_accuracy']} in {elapsed:.1f}s -> {args.out}")
if __name__ == "__main__":
main()
🛠️ Part 3: Run it by hand on a compute node (~5 min)
Now the round-trip from Lesson 1, with work in the middle.
Step 1: Request the node
Wait for the prompt to change to cpu1n001.
Step 2: Activate and run
cd ~/hello-aqua
source .venv/bin/activate # or: conda activate hello-aqua / micromamba activate hello-aqua
echo $NCPUS # → 4
python train_mnist.py
$NCPUS is set by PBS inside every job; the script reads it for its thread count.
What you should see
host cpu1n001
device cpu threads 4 seed 0
data 60,000 training images, 10,000 test (188 MB)
epoch 1/8 loss 0.0939 test accuracy 0.9613 5.3s
epoch 2/8 loss 0.0546 test accuracy 0.9738 1.8s
epoch 3/8 loss 0.0304 test accuracy 0.9749 1.8s
epoch 4/8 loss 0.0272 test accuracy 0.9799 1.7s
epoch 5/8 loss 0.0762 test accuracy 0.9787 1.7s
epoch 6/8 loss 0.0680 test accuracy 0.9790 1.7s
epoch 7/8 loss 0.0098 test accuracy 0.9789 1.7s
epoch 8/8 loss 0.0136 test accuracy 0.9820 1.7s
done test accuracy 0.982 in 25.2s -> results.json
Eight passes over 60,000 digits and it reads 98% of the 10,000 it has never seen. The accuracies will match to a few decimals (the seed is fixed). The times will vary with whoever else is on cpu1n001, and the first epoch is slower while PyTorch warms up.
Step 3: Read the summary
{
"seed": 0,
"samples": 60000,
"width": 1024,
"epochs": 8,
"device": "cpu",
"threads": 4,
"test_accuracy": 0.982,
"elapsed_s": 25.2,
"peak_rss_mb": 494,
"job_id": "12345678.aqua"
}
job_id is $PBS_JOBID, also set by PBS. elapsed_s and peak_rss_mb are what the run actually used: about 25 s and 500 MB, against a request of 1 h and 8 GB.
🔌 Part 4: Survive a dropped connection (~3 min)
An interactive job's shell is your SSH session. Lose the connection and PBS ends the job, along with anything running in it. The cure is tmux on the login node, so the thing holding your job is not your Wi-Fi. You are still inside the job from Part 3, so leave it first:
exit # back on the login node
tmux new -s dev # a session that outlives your connection
# Inside tmux, the same request as before:
qsub -I -l select=1:ncpus=4:mem=8GB -l walltime=01:00:00 -P ABCDEF1234
Detach with Ctrl+B then D; the job keeps running. Reconnect later and reattach:
Those two keys and two commands are all this lesson needs; the tmux cheat sheet has the rest (windows, panes, scrolling).
If your session seems to have vanished
- Aqua has more than one login node (
aquarius01,aquarius02, …), andssh aqua.qut.edu.aulands you on one of them. A tmux session lives on the node where you started it, sotmux lson the other node shows nothing. Runhostnamewhen you start the session, and check it again before assuming the session is gone. - Sessions do not survive a login-node reboot. Maintenance is the third Wednesday of each month;
time_until_outage.shtells you how far away it is.
tmux costs the login node nothing. What runs inside it still follows Lesson 1's rule: editing, git and qsub yes, the training run no.
Interactive or batch?
Before you start running things in an interactive job, ask: am I waiting on the machine, or is it waiting on me?
- If you are typing, reading output, changing a parameter and running again, this is interactive work. Stay.
- If you have set something going and are watching it, or would like to walk away, or intend to run it more than once with different inputs, it is a batch job. That is Lesson 4.
🎯 Key Takeaways
You now know
⚖️ Interactive requests are sized for the session, not the maximum — the caps are 8 cores / 34 GB (CPU) and 12 cores / 68 GB / 2 MIG slices (GPU), 12 h, and everything you ask for is held until you leave
🛠️ PBS tells your job what it got — $NCPUS for the thread count, $PBS_JOBID for the record
🔌 tmux on the login node keeps an interactive job alive across a dropped connection
🛑 Interactive is for developing; batch is for running — the question is who is waiting on whom
🔗 What's Next?
→ Lesson 4: Your First Batch Job — write the request and the commands in a file, and let PBS run them with nobody watching.
Stuck?
qsub -Isits at "waiting for job to start"? There is one interactive CPU node and it may be full. Try fewer cores (ncpus=2:mem=4GB), or checkpbsnodeinfo | grep cpu1n001to see how busy it is.command not found: pythonafter the prompt changes? You haven't activated the environment on the compute node.source ~/hello-aqua/.venv/bin/activate(or the conda equivalent) is per shell.- The script tries to download on the compute node? You skipped the
--epochs 0fetch in Part 2, or ran from a different directory. Fetch once on the login node and keepdata/mnist/next to the script. - PyTorch install is slow or warns about hardlinks? Cache and venv are on different filesystems. uv on Aqua has the three placements that avoid it.
- Want VS Code or Jupyter inside the interactive job instead of a bare shell? Surviving without VS Code Remote SSH covers the tunnel and the port-forwarded Jupyter Lab, both of which run inside exactly the
qsub -Iyou just used. - Curious what the interactive node actually is? Know Your Nodes, "CPU Interactive — the appetiser".
📝 Quick Reference
python train_mnist.py --epochs 0 # fetch the data and exit
python train_mnist.py --epochs 30 # longer run
python train_mnist.py --width 8192 # more memory and compute
python train_mnist.py --threads 8 # more cores
python train_mnist.py --device cuda # the GPU PBS gave you
python train_mnist.py --seed 7 --out run7.json # one result per seed
python train_mnist.py --checkpoint ckpt.pt # resume if the file exists
-
Access only in QUT network. Please use VPN to access the documentation when off-campus. ↩