CUDA Containers
This page covers the use of CUDA-enabled containers.
1. Overview
Setonix-Q provides curated Singularity (SIF) containers optimised for NVIDIA Grace Hopper (GH200) nodes (Arm64).
These images are built and published for common quantum workflows on Setonix-Q, including:
PennyLane + NVIDIA GPU simulators (Lightning GPU)
NVIDIA cuQuantum Appliance (cuQuantum + NVIDIA-optimised Qiskit + benchmarks)
CUDA-Quantum toolchain/runtime
MPI CUDA MPICH base for downstream container builds (MPI guidance TBD)
2. Available Container Images (Summary Table)
Image | Tag / Version | What it provides | Registry | SIF (S3) | Recipe / Source |
|---|---|---|---|---|---|
PennyLane GraceHopper |
| PennyLane + PennyLane-Lightning GPU backend (CUDA) + JupyterLab (in venv) |
|
|
|
cuQuantum Appliance |
| cuQuantum + NVIDIA-optimised Qiskit components + cuQuantum benchmarks |
|
| NGC-based (recipe pending from NVIDIA) |
cuQuantum Appliance |
| cuQuantum + NVIDIA-optimised Qiskit components + cuQuantum benchmarks |
|
| NGC-based |
cuQuantum Appliance |
| cuQuantum + NVIDIA-optimised Qiskit components + cuQuantum benchmarks |
|
| NGC-based |
CUDA-Quantum |
| CUDA-Quantum toolchain/runtime |
|
| NGC-based |
MPI CUDA MPICH Base |
| Base runtime (CUDA + MPICH) for downstream images |
|
|
|
3. Accessing Setonix-Q (GH200)
3.1 Allocate a Quantum node
salloc -p quantum -N 1 --gres=gpu:1 --time=01:00:003.2 Recommended module baseline
module purge
module load pawsey pawseytools pawseyenv PrgEnv-nvidia craype-arm-grace
module load gcc-native-mixed/12.3
module load pawseyenv
export MODULEPATH=$MODULEPATH:$LMOD_CUSTOM_COMPILER_GNU_12_0_PREFIX
module load pawsey4. Container usage quick start
4.1 cuQuantum Appliance (most common user workflow)
This image is widely used because it includes:
cuQuantum libraries
cuQuantum benchmark suite
NVIDIA-optimised Qiskit components (important for performance on NVIDIA GPUs)
Why we highlight this:
Many users run Qiskit workloads, and this image is designed specifically to provide a GPU-accelerated / NVIDIA-tuned Qiskit stack together with cuQuantum.
Example sanity checks:
singularity exec --nv <cuquantum_appliance>.sif python -c "import qiskit; print(qiskit.__version__)"
singularity exec --nv <cuquantum_appliance>.sif nv-quantum-benchmarks --helpIf you rely on Qiskit performance on GH200, this is the recommended starting point.
4.2 PennyLane GraceHopper: venv is required for examples
The PennyLane image installs its Python stack into a virtual environment:
Virtual env path:
/opt/venvThe build sets
VIRTUAL_ENV=/opt/venvand updatesPATHaccordingly.cuQuantum-related libs are also added into
LD_LIBRARY_PATH(via the venv site-packages path).
From the recipe (excerpted design intent):
creates venv:
python -m venv /opt/venvinstalls CUDA backend (
custatevec-cu12)installs
pennylane_lightningwheels and PennyLane from GitHubincludes
matplotlibandjupyterlab
Practical rule for users:
When you start an interactive shell, explicitly activate the venv before running PennyLane examples.
Example:
singularity exec --nv pennylane-gracehopper_0.0.3-arm.sif bash -lc '
source /opt/venv/bin/activate
python -c "import pennylane as qml; import pennylane_lightning; print(qml.__version__)'
Tip: Even if the image sets PATH for the venv during build, using
bash -lc+source /opt/venv/bin/activateavoids surprises across different Singularity invocation patterns.
5. Critical Singularity Behaviour on Setonix-Q (Root cause of “missing examples”)
5.1 Root cause: Singularity default bind of $HOME overrides container /home/*
By default, Singularity bind-mounts your host $HOME into the container.
This is usually convenient — but it causes a specific issue for NVIDIA’s cuQuantum and CUDA-Quantum container layouts:
cuQuantum places entrypoint assumptions and/or examples under:
/home/cuquantumCUDA-Quantum places entrypoint assumptions and/or examples under:
/home/cudaq
When Singularity’s default home bind happens, the container’s original /home tree is effectively not what you see anymore, so these directories can appear missing or overwritten.
5.2 Why we can’t rely on fakeroot here
Setonix-Q does not provide fakeroot capability for users.
So users cannot take the common “fakeroot-based” approach to extract/restore /home/cuquantum or /home/cudaq content from inside the image and rebuild/fix it locally.
5.3 Recommended workflow (robust)
Do not rely on container-bundled examples under /home/cuquantum or /home/cudaq.
Instead, clone example repositories into your real (host) $HOME (which is writable and intended for user workflows):
cd $HOME
git clone https://github.com/NVIDIA/cuquantum
git clone https://github.com/NVIDIA/cuda-quantumThis avoids:
Singularity home bind overriding container
/home/*read-only SIF filesystem constraints
fakeroot dependency
5.4 Driver issue
Currently, users may encounter the following error when running GPU-enabled workloads:
cupy_backends.cuda.api.runtime.CUDARuntimeError:
cudaErrorInsufficientDriver: CUDA driver version is insufficient for CUDA runtime versionThis is caused by a CUDA runtime–driver mismatch between the container images and the current Setonix-Q host driver. This is a known system-level issue and cannot be resolved by users. A CUDA driver upgrade via HPE is planned for March, after which this issue is expected to be resolved.
6. MPI (TBD)
MPI usage guidance on multi-node jobs (srun vs mpirun, PMI/PMIx compatibility, recommended patterns) will be added after validation on Setonix-Q.