Black Hole Torus#

The black hole torus (BHT) gallery exercises the same solver stack on a stationary accretion-torus configuration. It is the second major end-to-end physics example after the RNS sequence and is used to validate the waveless sector in the full spacetime+matter setup that is closer to the astrophysical accretion-torus problem.

This page is written as a self-contained guide: it explains the supported module, its environment variables, the deck-staging model, and the exact local and cluster launch commands used in practice.

Quick start#

The established working BHT path is the waveless module:

export COCAL_MODULE=bht-wl
export COCAL_KOKKOS_VARIANT=hopper90
export COCAL_NSOL=70
./scripts/submit.sh run

This is the commonly used Delta/SLURM workflow. The BHT module is selected by COCAL_MODULE and the backend is selected by COCAL_KOKKOS_VARIANT. The sequence length is controlled by COCAL_NSOL.

BHT module and runner layout#

The project currently exposes three solver modules, of which the BHT family is:

  • BHT-WL: black hole torus, waveless - executable: pBHT_WL - local runner: scripts/riemann_bht_wl.sh - SLURM runner: scripts/delta_bht_wl.sbatch

The BHT workhorse is the waveless formulation. The first successful GPU runs were made on the delta_bht_wl.sbatch path with both H200 and MI100 backends, each returning a successful solution with rc=0 and 67 iterations under the waveless sector.

Local execution (Riemann runner)#

For a detached local run, launch the Riemann runner directly:

COCAL_SW_WI=1 COCAL_SW_HIJ=1 \
nohup scripts/riemann_bht_wl.sh parallel 35 > /tmp/bht-wl.log 2>&1 &

This is the same physical configuration used for a production BHT run, but with plain detached-process scheduling instead of the job scheduler.

The COCAL_SW_* variables are not boolean values to be compared; they are presence-based switches. The implementation intentionally checks whether a variable exists, not whether it equals zero. In practice:

  • COCAL_SW_HIJ=0 turns the waveless sector on.

  • unset COCAL_SW_HIJ turns it off.

  • the code path is selected by presence, so paired runs cannot disagree about whether the waveless formulation is active.

Delta / SLURM execution#

Delta routes BHT through the shared scheduler script and a single config file:

export COCAL_MODULE=bht-wl
export COCAL_KOKKOS_VARIANT=hopper90    # cpu<N> | ada89 | hopper90 | mi100
export COCAL_NSOL=70
./scripts/submit.sh run

Common backend choices are:

  • cpu / cpu<N> for host runs

  • ada89 for NVIDIA Ada-class GPUs

  • hopper90 for H100/H200-class GPUs

  • mi100 for AMD MI100 GPUs

The module name, the Kokkos backend, and the sequence length remain the main control variables for the BHT job.

BHT-specific runtime variables#

The BHT family has a few physics switches and staging controls beyond the standard RNS variables:

  • COCAL_SW_WI: Waveless sector switch for the WI piece.

  • COCAL_SW_HIJ: Waveless sector switch for the HIJ piece.

  • COCAL_BHT_BCTYPE: black-hole torus boundary choice; default is KS.

The key point is that the code tests whether these variables are present, not whether they hold a specific numeric value. This keeps the local and SLURM paths consistent when switching between formulations or staging the same deck across multiple machines.

Deck staging and parameter menu#

The BHT deck is staged from a menu rather than from a single hard-coded run. The supporting logic treats:

parameter_sample/BHT_WL/D2.0

as a menu definition, then expands it into the actual run configuration under D2.0/AJS/K1/KS. This ensures that Riemann and Delta use the same deck meaning instead of drifting on a local convention.

The staging step is therefore an example of the repository’s shared build/run hygiene: it normalizes a physics menu into a single resolved input family before the solver starts.

Expected outputs and validation#

The BHT run writes the usual project outputs:

  • solver log in logs/ at the repo root

  • run directory under the active build tree

  • per-solution directories with the staged inputs and final solution data

For validation, use the same project workflow as the other examples:

  1. Run the fast unit tests with ctest --test-dir build -L fast.

  2. Run the full end-to-end check with bash tests/e2e/run_e2e.sh.

  3. Run the SLURM GPU gate with sbatch ci/slurm_e2e_gpu.sbatch.

The BHT gallery is intended to reproduce the physically validated waveless accretion-torus sequence across supported GPU backends, with the waveless sector switches kept consistent across local and cluster execution.