Setting up COCAL#

This page documents the configuration and build-related steps for COCAL. It covers the per-machine config.sh file, the provided sw-build (prebuilt Kokkos installs), how to compile on a compute node, and how to add a new backend. Sections about running the solver, choosing physics, and reproducing tests are intentionally excluded; see the run and testing pages for those.

The idea#

Three things vary, and each has one home:

What varies — Lives in — Set with

Machine paths and SLURM (account, partitions) — config.sh — edit this file
Compiler and Kokkos arch flags per backend — scripts/cocal_build_env.sh — COCAL_KOKKOS_VARIANT
(Run-time physics is passed at run-time and is out of scope here.)

Create your machine copy of the configuration from the tracked template:

cp config.template.sh config.sh

The tracked config.template.sh provides the fields and helpful find: comments; your personal config.sh is intentionally gitignored so your cluster-specific paths and allocations are never committed.

At minimum, set COCAL_KOKKOS_ROOT to your copy of the provided sw-build and COCAL_SLURM_ACCOUNT to your allocation. You may also export any of these values in your shell; exported values override the file.

If you have an older clone where config.sh is tracked, migrate it to the template pattern like this:

cp config.sh ../my-config-backup.sh
git checkout -- config.sh
git pull
cp ../my-config-backup.sh config.sh

This preserves your settings while making them untracked and ignored.

Backends#

Choose a backend with COCAL_KOKKOS_VARIANT. Each backend corresponds to a Kokkos install (Kokkos allows only one GPU arch per install), so builds are kept separate per variant.

Variant	Hardware	Compiler
`cpu`	Any host (Serial + OpenMP)	`g++`
`cpu<N>`	Same cpu build; run with N threads (`cpu1`, `cpu35`)	`g++`
`ada89`	NVIDIA Ada (`sm_89`) — e.g. RTX 4080	`nvcc_wrapper`
`hopper90`	NVIDIA Hopper (`sm_90`) — H100 / H200	`nvcc_wrapper`
`mi100`	AMD Instinct (`gfx908`) — MI100	`hipcc`

Note: cpu, cpu1 and cpu35 share a single build tree (build-cpu). The numeric suffix affects only the run-time thread count and does not require recompilation.

The provided sw-build#

The repository builds against prebuilt Kokkos 5.1.1 installs, one per backend. The expected layout is:

sw-build/install-cpu/
sw-build/install-hopper90/
sw-build/install-mi100/

Copy the sw-build you were given into a directory you control and point COCAL_KOKKOS_ROOT at it. For example:

cp -r /path/you/were/given/sw-build "$WORK/sw-build"
# then set in config.sh: COCAL_KOKKOS_ROOT=$WORK/sw-build

Keep your own copy rather than using someone else’s directly: the build writes a small dependency cache (.cpm-cache/) into the tree and you should have write access.

Compiling COCAL#

With config.sh pointed at your sw-build and the appropriate COCAL_KOKKOS_VARIANT set, compile the solver using the helper script that submits the job to a compute node:

cp config.template.sh config.sh        # once per clone; edit COCAL_KOKKOS_ROOT
export COCAL_KOKKOS_VARIANT=hopper90   # cpu | cpu35 | hopper90 | mi100

./scripts/submit.sh build

Note

submit.sh submits the build to a compute node (it should not be run on a login node). The wrapper is used to avoid running heavy compiler workloads on interactive/login nodes where they can be killed for resource overuse.

Preparing a network-backed dependency cache (optional)#

If compute nodes do not have network access, warm the CPM dependency cache on a login node once so that subsequent compute-node configures do not attempt to download packages:

source scripts/cocal_build_env.sh
cmake -S . -B /tmp/cocal-cfg $(cocal_cmake_common_args)

This configure step downloads fmt and cxxopts into the CPM_SOURCE_CACHE and typically ends with a Kokkos-not-found error — that is expected and means the dependencies were fetched.

Building without SLURM#

On machines without SLURM (local laptops or development boxes) you must prepare the environment for a manual build. Requirements:

A Kokkos install for your backend (there may be no install-ada89 in the provided sw-build). You can build one using scripts/cocal_kokkos.sbatch as a recipe (it is a plain CMake invocation), or point COCAL_KOKKOS_DIR at an existing install.
gfortran available on your PATH: a small Fortran helper is built as part of the staging process.

Example environment configuration for a local CUDA Ada build:

export COCAL_KOKKOS_VARIANT=ada89
export COCAL_CUDA_HOME=/usr/local/cuda
export COCAL_KOKKOS_DIR=/path/to/install-ada89/lib/cmake/Kokkos
source scripts/cocal_build_env.sh

Be careful with temporary assignments and source — prefer export VAR=... on its own line rather than VAR=... source ... which may not persist the value after source returns.

Adding a backend#

To add a new GPU architecture variant (example: blackwell100 using sm_100), update the environment script and your configuration template:

Add a case arm to scripts/cocal_build_env.sh mirroring the style of the existing GPU entries and set the appropriate -DKokkos_ARCH_* flag. For example:

blackwell100)
    COCAL_CXX_COMPILER="${COCAL_CXX_COMPILER:-${COCAL_KOKKOS_SRC}/bin/nvcc_wrapper}"
    COCAL_KOKKOS_BUILD_FLAGS=(
        -DKokkos_ENABLE_SERIAL=ON
        -DKokkos_ENABLE_CUDA=ON
        -DKokkos_ENABLE_CUDA_LAMBDA=ON
        -DKokkos_ARCH_BLACKWELL100=ON
    )
    : "${COCAL_CUDA_HOME:?cuda backend needs COCAL_CUDA_HOME -- set it in config.sh}"
    export PATH="${COCAL_CUDA_HOME}/bin:${PATH}"
    export LD_LIBRARY_PATH="${COCAL_CUDA_HOME}/lib64:${LD_LIBRARY_PATH:-}"
    export NVCC_WRAPPER_DEFAULT_COMPILER="${NVCC_WRAPPER_DEFAULT_COMPILER:-g++}"
    ;;

Add two SLURM lines in both config.template.sh and your local config.sh: COCAL_SLURM_COMPILE_BLACKWELL100 and COCAL_SLURM_RUN_BLACKWELL100. Each is a string with "partition gres mem cpus time" describing the resources for compile and run jobs respectively.
Grep the tree for existing variant names and update any human-facing lists (comments, docs, tables) that enumerate supported backends:

grep -rn "hopper90" --include='*.sh' --include='*.sbatch' --include='*.md' .

Files overview#

File	Purpose
`config.template.sh`	The tracked template; copy it to `config.sh` and edit per machine
`config.sh`	Your machine-local values (gitignored)
`scripts/cocal_build_env.sh`	Maps Backend → compiler + CMake flags
`scripts/cocal_kokkos.sbatch`	Builds a Kokkos install for a backend (CMake recipe)
`scripts/cocal_build.sbatch`	Builds `pRNS_WL` without launching the solver