Running on a Cluster#

This page documents submitting and running COCAL on a cluster (via ./scripts/submit.sh), how to choose run-time physics, where outputs land, reproducing validated test cases, and common troubleshooting notes.

Basic run#

Set run-time environment variables and submit a run job:

export COCAL_ROT_LAW=DR COCAL_NUM_SOL_SEQ=35
./scripts/submit.sh run

The helper submit.sh accepts additional sbatch options after --; these are passed through directly. Example:

./scripts/submit.sh run -- --partition=gpuMI100x8 --time=04:00:00

Build vs run#

There are two submission modes used during development:

  • ./scripts/submit.sh build — submits a compute-node compile-only job to

    perform the slow cold build of pRNS_WL (useful to warm the build tree).

  • ./scripts/submit.sh run — performs a fast incremental rebuild and then

    launches the solver.

These are separate sbatch submissions. Wait for a completed build job before submitting run if you rely on the cold build; otherwise both jobs may compile into the same build-<variant>/ tree and race, causing intermittent failures. Monitor with:

squeue -u $USER

Choosing the physics#

The defaults are uniform rotation (UR) and one solution. Override them before submitting when needed, for example:

export COCAL_ROT_LAW=DR COCAL_NUM_SOL_SEQ=35
./scripts/submit.sh run

See Rapidly Rotating Neutron Star for supported rotation laws and the other physics options.

Where output goes#

Two places hold result information:

  • logs/ at the repo root — SLURM job stdout/stderr named <job-name>_<job-id>.

  • The run directory under the build tree (build-<variant>/...) — staged

    parameter files and solution directories.

Typical layout:

logs/cocal-DR-hopper90_20253856.out
logs/cocal-DR-hopper90_20253856.err
build-hopper90/RNSH3_K180.7_run_20253856/
      run.log -> ../../logs/cocal-DR-hopper90_20253856.out
      work_area_RNS_seq/sol_01/ ... sol_35/

The SLURM log contains everything printed by the job (incremental compile, solver progress, and final messages). The run directory is named from the parameter set and job id and contains the staged COCAL tree and solution files (.las field files under work_area_RNS_seq/sol_NN/).

Troubleshooting and notes#

  • Do not compile on a login node; compile on a compute node via

    ./scripts/submit.sh build to avoid the compiler being killed.

  • COCAL_KEEP_RUN_DIR: the staging script tests for non-empty, not truthy

    semantics — setting it to 0 still keeps the directory; to delete the run directory, unset COCAL_KEEP_RUN_DIR.

  • If you see cocal_build_env: no config.sh at /path/to/config.sh make a

    local copy from the template:

    cp config.template.sh config.sh
  • If COCAL_KOKKOS_ROOT is unset you copied the template but did not edit

    it; set it to point at your sw-build copy.

  • If a job reports no Kokkos install at .../lib/cmake/Kokkos, run

    ./scripts/submit.sh kokkos for the missing backend to build an install or point COCAL_KOKKOS_DIR at an existing install.

  • compiler 'hipcc' not found indicates the job landed on a node without the

    required toolchain; check the partition in the corresponding COCAL_SLURM_* line in config.sh.

  • Variables set as temporary assignments with VAR=x source ... may read

    back empty; prefer export VAR=x on its own line.