Deploy as services
A step-by-step guideline for the production prover deployment:
each binary runs under a real service manager (systemd on Linux,
launchd on macOS) behind a dedicated service user, with config
in /etc/zisk/ and state on its own writable path. By the end of
this page a coordinator and one worker are running as managed
services and have proved a real gcd job.
When to pick this
Pick this path when the prover needs to run 24/7 under a real
service manager, survive reboots, restart on failure, and route
logs to journald (Linux) or /var/log/zisk-*/ (macOS). Right
shape for:
- A bare-metal or VM host that runs the coordinator full-time.
- One or more worker hosts that need to register at boot.
- Anything where you want the binary launched by
pid 1instead of by your shell.
For shorter-lived workflows where you launch the binaries by hand, see Deploy the binaries directly; for a containerised stack with Prometheus and Grafana bundled in, see Deploy with Docker.
Prerequisites
| Requirement | Notes |
|---|---|
| Linux / macOS | Linux uses systemd; macOS uses launchd. No Windows. |
sudo access | The install scripts write to /usr/local/bin, /etc/zisk, and the service manager directory. |
| Internet access | The scripts self-bootstrap from raw.githubusercontent.com and pull ziskup for the bundle. |
| ~32 GB RAM (worker hosts) | The Assembly emulator preallocates large shared regions on the worker side. |
Open ports 7000, 50051, 9090 | Public API, cluster gRPC, and metrics on the coordinator host. Workers only need outbound to :50051. |
System dependencies (apt / brew packages) live in the developer install guides: Linux, macOS.
How the install scripts work
Two scripts ship in the upstream repo, one per binary, and both are curl-pipe-able from a fresh host:
distributed/deploy/scripts/coordinator/install.sh
distributed/deploy/scripts/worker/install.sh
Each script detects the OS and lays things out conceptually the same way, with platform-specific homes:
| Asset | Linux (systemd) | macOS (launchd) |
|---|---|---|
| Binary | /usr/local/bin/zisk-{coordinator,worker} | /usr/local/bin/zisk-{coordinator,worker} |
| Config | /etc/zisk/{coordinator,worker}.toml | /etc/zisk/{coordinator,worker}.toml |
| Service unit | /etc/systemd/system/zisk-*.service | /Library/LaunchDaemons/com.zisk.*.plist |
| Writable state | /var/lib/zisk-{coordinator,worker}/ | /usr/local/var/zisk-{coordinator,worker}/ |
| Shared bundle | /opt/zisk/ | /Library/Application Support/ZisK/ |
| Logs | journald (journalctl -u zisk-*) | /var/log/zisk-*/zisk-*.log (rotated by newsyslog) |
| Service user | zisk-coordinator / zisk-worker | zisk-coordinator / zisk-worker |
Install the coordinator
Run the coordinator install script on the host you've chosen as the coordinator. The OS detection and behaviour differ slightly, so the verification commands branch — pick your OS below.
1. Run the installer.
curl -fsL https://raw.githubusercontent.com/0xPolygonHermez/zisk/main/distributed/deploy/scripts/coordinator/install.sh \
| sudo bash -s -- --api-port 7000
The script, in one pass:
- Pulls the shared toolchain into
/opt/zisk/viaziskup --nokey(the coordinator doesn't need the proving key). - Creates the
zisk-coordinatorsystem user (home/var/empty, no login). - Installs
/usr/local/bin/zisk-coordinatorand the example config at/etc/zisk/coordinator.toml. - Provisions
/var/lib/zisk-coordinator/for writable state (includes acache/subdir for registered guest ELFs). - Writes a hardened systemd unit at
/etc/systemd/system/zisk-coordinator.servicewithNoNewPrivileges,ProtectSystem=strict,ProtectHome,LimitNOFILE=65535, andNice=-10. - Runs
systemctl enable zisk-coordinatorand thensystemctl restart zisk-coordinator— the service is up immediately and survives reboots.
2. Verify the service is up.
sudo systemctl status zisk-coordinator
You should see active (running). Tail the logs to confirm three
"listening on" lines for ports 7000, 50051, and 9090:
sudo journalctl -u zisk-coordinator -f
3. Probe the health endpoint.
curl -i http://127.0.0.1:9090/health # → HTTP/1.1 200 OK
Install the worker
Run the worker install script on each host you've chosen as a
worker, including the coordinator host if it also proves. Point
it at the coordinator's cluster port (:50051). The examples
below use http://127.0.0.1:50051, which only works on the
coordinator host itself; on a dedicated worker host, pass the
coordinator's address instead.
1. Run the installer.
curl -fsL https://raw.githubusercontent.com/0xPolygonHermez/zisk/main/distributed/deploy/scripts/worker/install.sh \
| sudo bash -s -- --coordinator-url http://127.0.0.1:50051
The script mirrors the coordinator install but:
- Pulls the bundle with the proving key (workers need it to generate proofs).
- Creates the
zisk-workersystem user. - Drops
/usr/local/bin/zisk-workerand the stock example config at/etc/zisk/worker.toml. The file is not edited: itscoordinator.urlstayshttp://127.0.0.1:50051, and the URL you supplied is passed as--coordinator-urlin the unit'sExecStart, where it overrides the file. - Provisions
/var/lib/zisk-worker/(the unit's working directory) for input staging. Logs go to journald; there is no on-disk log directory on Linux. - Writes
/etc/systemd/system/zisk-worker.servicewithRestart=on-failure. - Runs
systemctl enable zisk-workerand thensystemctl restart zisk-worker.
2. Verify the worker registered.
sudo systemctl status zisk-worker
sudo journalctl -u zisk-worker -f
In the coordinator's journal (on the coordinator host) you should see:
<timestamp> INFO: Registered worker: WorkerId(<id>…) (total: 1 CC: 10CU ACC: 0CU)
<timestamp> INFO: WorkerId(<id>…) registered successfully
Both scripts accept the same operational knobs regardless of OS.
Pass them after bash -s --:
| Flag | Effect |
|---|---|
--binary PATH | Install a pre-built binary instead of pulling via ziskup. |
--config PATH | Install your own TOML instead of the bundled sample. |
--no-start | Install but don't start (useful for staged rollouts). |
--uninstall -y | Reverse the install (see Uninstalling). |
Coordinator-only: --api-port, --cluster-port, --metrics-port.
Worker-only: --coordinator-url, --worker-id, --compute-capacity, --proving-key, --with-snark, --gpu / --cpu, --mpi / --no-mpi.
The worker installer accepts --log-level and also reads
RUST_LOG from the environment or from an --env file (the
sample deploy/scripts/examples/.env.example sets
RUST_LOG=info). Either one adds --log-level to the worker's
ExecStart, a flag zisk-worker doesn't accept: the service
crash-loops with error: unexpected argument '--log-level' found.
Leave both unset for the worker; its verbosity is set only with
-v / -vv. The coordinator installer handles --log-level
correctly.
--with-snark downloads the SNARK proving key into the bundle and
adds --snark --proving-key-snark to the worker's service, so the
worker serves final-SNARK jobs. See
zisk-worker / Proving keys.
Configure the coordinator
The install script wrote a working example at
/etc/zisk/coordinator.toml. The tables it reads:
| Table | Purpose |
|---|---|
[service] | Identity used in log labels. environment is a free-form tag (development / staging / production). |
[server] | The public gRPC API where clients submit jobs. shutdown_timeout_seconds is the drain window applied on SIGTERM. |
[coordinator] | The cluster gRPC port workers dial to register and stream segments. |
[metrics] | GET /metrics returns Prometheus text; GET /health returns 200 OK while alive. |
[logging] | level sets the log level. format has no effect in 1.4.0-alpha: pretty, json and compact print the same lines. |
[backend] | Internal — keep coordinator. |
Edit /etc/zisk/coordinator.toml with sudo nano (or your
editor of choice), then restart the service to pick up the
change:
sudo nano /etc/zisk/coordinator.toml
sudo systemctl restart zisk-coordinator
sudo journalctl -u zisk-coordinator -n 20 # check it came back clean
Configure the worker
The install script wrote the stock /etc/zisk/worker.toml. Its
coordinator.url is http://127.0.0.1:50051; the URL you passed
on the command line lives in the unit's ExecStart as
--coordinator-url and wins over the file. The tables it reads:
| Table | Purpose |
|---|---|
[worker] | Identity + capacity. compute_units is an abstract weight the coordinator uses to assign segments. |
[coordinator] | Where the worker dials. Overridden by the --coordinator-url in ExecStart. |
[connection] | How the worker reacts to network blips. Only reconnect_interval_seconds is used; heartbeat_timeout_seconds is accepted but unused. |
[telemetry] | Host and process telemetry the worker sends with its heartbeats. |
[logging] | Accepted but not read: zisk-worker ignores [logging] and RUST_LOG. Raise verbosity with -v / -vv in ExecStart. |
There's intentionally no [backend] table — backend selection
(Assembly emulator, Rust emulator, GPU, Plonk) is done via flags
in the unit's ExecStart. See
zisk-worker for every flag.
Edit and restart:
sudo nano /etc/zisk/worker.toml
sudo systemctl restart zisk-worker
sudo journalctl -u zisk-worker -n 20
Submit a smoke-test job
The repo ships a gcd example whose remote-host binary submits
a real proving job over the public gRPC API. Run it from any
host that can reach the coordinator's :7000 — typically a
developer machine, not the service-managed coordinator /
worker host.
Then clone the repo if you haven't already and run the example:
# Skip the clone if you already have one
[ -d zisk ] || git clone https://github.com/0xPolygonHermez/zisk.git
cd zisk/examples/gcd/host
cargo run --release --bin remote-host
The binary builds a
ProverClient::remote("http://localhost:7000")
internally — the same code path your own host applications use.
The URL is hard-coded: on a client host that isn't the
coordinator, edit it in examples/gcd/host/src/prover-clients/remote.rs
to point at the coordinator's address before running.
While it runs, watch the coordinator and worker journals:
- The coordinator journal shows the job going through its
phases, from
[Job] Started JobId(…)through[Phase1],[Phase2]and[Phase3] Aggregation started …to[Job] Finished JobId(…) successfully ✔. - The worker journal shows witness generation and proof
generation, and ends with
Aggregation task completed for JobId(…).
When remote-host exits cleanly it prints
Proof was verified successfully. and gcd(5, 10) => 5. It
verifies the proof but does not save it; the sha-hasher
example's prove-remote binary writes one to
tmp/sha_hasher_proof.bin.
Manage the services
Day-two operations are handled by the service manager directly,
not by re-running the install script. The commands below show
both binaries' coordinator examples — swap zisk-coordinator
for zisk-worker (Linux) or com.zisk.coordinator for
com.zisk.worker (macOS) to operate on the worker side.
Inspect status
Check whether the service is running, when it last started, and the exit status of its previous run.
sudo systemctl status zisk-coordinator
Look for Active: active (running) and a recent Started
timestamp. A failed state means the unit exited and didn't
restart — the journal tail below shows why.
Tail the logs
Watch the service's stdout / stderr in real time. Use this when running a smoke test, debugging a misconfig, or tracking down why a worker isn't registering.
stdout/stderr go to journald:
sudo journalctl -u zisk-coordinator -f
Start
Bring a stopped service up. Both install scripts auto-start the
service, so you only reach for this after a manual stop or an
install run with --no-start.
sudo systemctl start zisk-coordinator
Stop
Shut the service down. On the coordinator, the service manager
sends SIGTERM: the coordinator stops accepting new jobs and
drains in-flight requests for up to shutdown_timeout_seconds
(see zisk-coordinator / Signals).
The worker doesn't drain. It cancels the job in flight, kills its
ASM services, unlinks their shared memory and exits with status
0 within about 5 s; a job it was computing fails on the
coordinator and must be submitted again (see
zisk-worker / Signals).
Under MPI the systemd unit sends SIGUSR1 to mpirun alone,
which passes it to every rank:
ExecStart=<mpirun> ... --mca orte_allowed_exit_without_sync 1 ... /usr/local/bin/zisk-worker ...
KillMode=mixed
KillSignal=SIGUSR1
TimeoutStopSec=60
Restart=on-failure
Restart=on-failure (and KeepAlive in the launchd plist) also
restarts a worker that exits with status 75, which is how a
worker that can't recover in-process gets back into the cluster.
A unit written by an install script older than 1.4.0-alpha lacks
these settings: re-run the install script to rewrite it.
sudo systemctl stop zisk-coordinator
Restart
Pick up a config edit or push a newly-installed binary into the
running process. Equivalent to a stop followed by a start,
issued as one command so the gap is minimised.
sudo systemctl restart zisk-coordinator
Disable / re-enable on boot
Stop the service from starting at boot, but leave the unit / plist on disk so you can re-enable later without re-running the installer.
sudo systemctl disable zisk-coordinator
sudo systemctl enable zisk-coordinator # re-enable for next boot
disable removes the symlinks from /etc/systemd/system/*.target.wants/
but leaves the unit file in place. Pair with stop if the
service is currently running.
Updating
Both install scripts are idempotent — re-running them in place
picks up the latest binary, refreshes the bundle via ziskup,
and rewrites the unit/plist. Existing config under /etc/zisk/
is not touched. Pass --no-start to stage the new binary
without bouncing the service, then restart on your own schedule.
The unit/plist is rewritten from the flags of the current run
only. Re-running the installer with just --no-start drops every
flag you passed at install time: --api-port on the coordinator,
and --coordinator-url, --worker-id, --compute-capacity and
--with-snark on the worker (--gpu comes back on its own from
the bundle's GPU/CPU variant). Pass the same flags again, as the
commands below do for the install examples on this page.
# Coordinator
curl -fsL https://raw.githubusercontent.com/0xPolygonHermez/zisk/main/distributed/deploy/scripts/coordinator/install.sh \
| sudo bash -s -- --no-start --api-port 7000
sudo systemctl restart zisk-coordinator
# Each worker host
curl -fsL https://raw.githubusercontent.com/0xPolygonHermez/zisk/main/distributed/deploy/scripts/worker/install.sh \
| sudo bash -s -- --no-start --coordinator-url http://127.0.0.1:50051
sudo systemctl restart zisk-worker
For a rolling fleet update, take one worker at a time: wait until
it is idle (coordinator_worker_state is 1, ready, rather than
3, computing), upgrade and restart it, then move on. Stopping a
worker that is computing fails its job; nothing finishes it or
moves it to another worker. Every worker must keep the same
proving keys and --snark setting as the rest of the cluster, or
the coordinator rejects it when it registers again.
Uninstalling
Each install script accepts --uninstall, which stops and
disables the service, removes the unit/plist, deletes the binary,
removes the service user, and deletes the state directory
(/var/lib/zisk-* or /usr/local/var/zisk-*). It removes only
its own TOML under /etc/zisk/, and /etc/zisk/ itself only
once it is empty.
Uninstall deletes the state directory whatever it holds, even if
it existed before the install. Install also runs chown -R on an
existing state directory. Don't point the scripts at a directory
other software uses.
# Coordinator
curl -fsL https://raw.githubusercontent.com/0xPolygonHermez/zisk/main/distributed/deploy/scripts/coordinator/install.sh \
| sudo bash -s -- --uninstall -y
# Each worker host
curl -fsL https://raw.githubusercontent.com/0xPolygonHermez/zisk/main/distributed/deploy/scripts/worker/install.sh \
| sudo bash -s -- --uninstall -y
-y skips the confirmation prompt. The shared bundle under
/opt/zisk/ (Linux) or /Library/Application Support/ZisK/
(macOS) is left in place because another binary on the host may
still depend on it. The script ends with a hint to run
sudo ziskup --uninstall --system; that command refuses while
any zisk-*.service unit (or ZisK plist) is still installed, so
run it only after uninstalling every ZisK service on the host.
Alternatively, remove the bundle by hand:
sudo rm -rf /opt/zisk # Linux
sudo rm -rf "/Library/Application Support/ZisK" # macOS