Skip to main content

Usage

How to invoke zisk-worker, the stateless proving process that dials the coordinator and proves assigned segments. Covers its three configuration sources and override precedence, the operationally relevant defaults, signal handling, compute-unit sizing, and worked examples.

A worker is the proving process. It dials the coordinator, registers a compute capacity, and proves the segments it's assigned. Workers are stateless across jobs — add, remove, or restart them at will. A worker host needs ~32 GB RAM and, optionally, a CUDA-capable GPU.

zisk-worker [OPTIONS]

Configuration​

zisk-worker reads its settings from three sources, plus built-in defaults for anything you don't set:

  • TOML config file passed via --config <PATH>. The canonical home for long-lived settings: coordinator URL, advertised capacity, connection tuning, telemetry. The full schema is documented on Configuration.
  • Environment variables. The binary reads only ZISK_WORKER_CONFIG_PATH (the config file when --config is unset), plus ZISK_HOME / ZISK_CACHE_DIR for its paths. It ignores RUST_LOG. The other ZISK_WORKER_* variables are read by the install script, which turns them into flags. Full mapping on CLI & environment.
  • Command-line flags for identity overrides, proving keys, backend selection, performance tuning, and diagnostics. The narrowest layer; use for one-off testing, container entrypoints, or the systemd unit's ExecStart. Full flag list on CLI & environment.

Override precedence​

Settings are resolved in three layers, with later sources winning:

  1. Built-in defaults.
  2. TOML config file (--config, else ZISK_WORKER_CONFIG_PATH).
  3. Command-line flags.

Example: a TOML with [worker].compute_capacity.compute_units = 10 is shifted to 32 by --compute-capacity 32. Setting ZISK_WORKER_COMPUTE_CAPACITY=16 in the worker's environment has no effect: only the install script reads it, and it writes it into the service as --compute-capacity 16. Neither RUST_LOG nor [logging] changes the worker's log level; only -v (debug) and -vv (trace) do.


Defaults​

If you start zisk-worker with no flags and no config file, the built-in defaults take over. The full schema is on Configuration; the most operationally relevant defaults are:

SettingDefaultOverride
Coordinator URLhttp://127.0.0.1:50051[coordinator].url / --coordinator-url
Compute units advertised10[worker].compute_capacity / --compute-capacity
Worker identifierrandom UUID[worker].worker_id / --worker-id
STARK proving key path${ZISK_HOME}/provingKey ($HOME/.zisk/provingKey; /opt/zisk/provingKey for a system install)--proving-key / ZISK_HOME
SNARK proving key path${ZISK_HOME}/provingKeySnark, read only with --snark--proving-key-snark / ZISK_HOME
Final-SNARK jobsoff--snark
BackendAssembly (unless --emulator is set; on macOS the emulator is selected automatically)--asm <PATH> / --emulator / --gpu
Inputs folder. (current dir; under the install script's unit, its working directory /var/lib/zisk-worker)[worker].inputs_folder
Reconnect interval5s[connection].reconnect_interval_seconds
Telemetryon, every 10s[telemetry].enabled / --no-telemetry, [telemetry].interval_seconds
Log levelinfo-v (debug) / -vv (trace)

Signals​

SignalEffect
SIGTERM / SIGINT / SIGUSR1Stop now. The worker cancels the job in flight, kills its ASM services, unlinks their shared memory and semaphores, and exits with status 0, within about 5 s. Past 5 s it exits anyway and the next start unlinks what is left. Every MPI rank stops on its own.
SIGKILLForced exit with no clean-up. Shared memory stays in /dev/shm until the worker starts again.

A stopped worker doesn't finish its work. If it was computing, its stream drops and, once reconnect_grace_period_ms passes, the coordinator fails the job with Worker <id> connection dropped and parks the job's other workers until they recover. The coordinator doesn't reassign the job; the client has to submit it again. Stop workers between jobs when you can.

Under MPI​

mpirun can't pass SIGTERM on: it treats it as an abort, sends the ranks SIGTERM and then SIGKILL, and exits with status 1. Send SIGUSR1 to mpirun instead. It forwards the signal to every rank, and with --mca orte_allowed_exit_without_sync 1 (Open MPI 4) it exits with status 0 once they all have. The service unit the install script writes does both:

/etc/systemd/system/zisk-worker.service (excerpt)
ExecStart=<path to mpirun> ... --mca orte_allowed_exit_without_sync 1 ... /usr/local/bin/zisk-worker ...
KillMode=mixed
KillSignal=SIGUSR1
TimeoutStopSec=60

A unit written by an older install script doesn't. Re-run the install script to pick up the new unit.


Exit status​

StatusMeaning
0Stopped by a signal (see above).
75EX_TEMPFAIL: the worker ended itself because it can't recover in-process and needs a restart. It happens when a cancellation handshake fails, when recovery takes more than 300 s, or when a panic hits a worker with several MPI ranks. Every rank is aborted.
otherA start-up error, such as a bad config or a missing proving key.

Before exiting with 75 the worker logs, at ERROR and on stderr, <reason>; aborting the worker so that it restarts. On its next start it reports the reason to the coordinator (see Telemetry). Run the worker under a supervisor that restarts it on failure: the systemd unit uses Restart=on-failure and the launchd plist KeepAlive. A worker that keeps exiting with 75 points to a deeper problem.


Compute units explained​

Compute units are an abstract weight the coordinator uses when splitting a job. A worker advertising compute_units = 10 gets roughly twice the load of one advertising 5.


Examples​

Run with the config installed by the deploy script:

zisk-worker --config /etc/zisk/worker.toml

Connect to a remote coordinator with a stable worker id and a larger capacity than the default:

zisk-worker --config /etc/zisk/worker.toml \
--coordinator-url http://coord.lan:50051 \
--worker-id worker-a-gpu1 --compute-capacity 32

Use the GPU build with four concurrent streams:

zisk-worker --config /etc/zisk/worker.toml --gpu --max-streams 4

Serve final-SNARK jobs. --snark is required; the SNARK key must be on disk (for example installed with --with-snark), and every worker in the cluster must use a key of the same protocol:

zisk-worker --config /etc/zisk/worker.toml \
--snark --preload-snark \
--proving-key-snark /opt/zisk/provingKeySnark