— Operators · one process per role
Run a node
on a chain you do not own
A Ztakora node is a Node process, one per role, talking JSON-RPC to a chain. It has four inputs — an endpoint, a signing key, an address book and a state directory — and reads everything else from the network: the epoch, the phase, the issuer set, the verifier set and the parameter set in force. A daemon that had to be told its own environment is a daemon that will be told it wrong, so there is nothing to configure beyond those four lines.
This page is the public form of docs/ops.md, the file the node's deploy.ts and store.ts point at. Where this page and the contracts disagree, the contracts are right and this page has a bug.
— Inputs
Four inputs, and one of them is handled differently
Three of the four are flags you can type in a transcript. The fourth cannot be, which is the only asymmetry in the whole interface.
| Input | Flag | Environment | Notes |
|---|---|---|---|
| Chain endpoint | --rpc <url> | ZTAKORA_RPC_URL | defaults to http://127.0.0.1:8545 |
| Signing key | — | ZTAKORA_KEY_<ROLE> | never a flag, never a file in the repository |
| Address book | --book <path> | ZTAKORA_BOOK | defaults to state/address-book.json |
| State directory | --state <dir> | ZTAKORA_STATE | defaults to state |
A command-line argument is visible in ps, in shell history, and in whatever incident-ticketing system captures the process tree. The address book is plain text on disk and is meant to be copied between machines, so it holds addresses and no secrets. That restraint is the whole attack surface of the operator's box.
— Keys
One variable per on-chain role
The variables are named after the role the key plays on-chain, because that is the address a bond, a slash or a settlement event is filed against.
ZTAKORA_KEY_DEPLOYER owner of Ztakora: pause, watcher list, gate terms, ownership
ZTAKORA_KEY_ISSUER declares a book, mints and burns its own series
ZTAKORA_KEY_VERIFIER1 bonded attester — commit and reveal
ZTAKORA_KEY_VERIFIER2 second attester
ZTAKORA_KEY_VERIFIER3 third attester
ZTAKORA_KEY_OPS aggregator: advance, settle, punish, record
ZTAKORA_KEY_WATCHER audits settlements; a shadow report needs a bond
ZTAKORA_KEY_BACKSTOP contributes to and withdraws from the reserveA daemon refuses to start without its own key
Even if the other seven are present. The code's fallback is the deterministic devnet account, which is right on a fixture and wrong anywhere else — those private keys are published in the repository. On a real endpoint that fallback is only reachable by asking for it by name with --allow-fixture-keys, and the refusal message says so.
- One key, one process. The reliable way to get a nonce self-collision is two processes signing the same key; this run learned that by doing it, which is why six role daemons are deliberately stopped while a funding driver holds the same keys.
- An empty variable counts as unset.
ZTAKORA_KEY_VERIFIER1=is a line in most environment files; a daemon that signs with an empty key is worse than one that refuses to start. - The supervisor prints what it cannot sign for before it spawns anything, and says what the gap does to quorum — see the playbook below.
— First run
Rehearse, then deploy, then supervise
npm run demo is the rehearsal: an in-process EVM, all four roles, four epochs. If the demo is not producing what you expect, do not deploy.
npm ci # solc 0.8.26, @ethereumjs/vm 10, viem 2
npm run compile # build/artifacts/*.json — aborts on a stale source
npm test # 55 tests; green before anything else
npm run demo # four rehearsed epochs, one process, in memory# the deployment: prints the requirement it derived before it signs anything
ZTAKORA_KEY_DEPLOYER=… node node/src/cli.ts deploy \
--rpc https://YOUR-ENDPOINT --chain-id 84532 --book state/base-sepolia.json \
--fast --collateral 0.00024 --bond 0.000249290657234351 \
--cover 0.00015 --fund 0.00015# the network: six roles, one supervised process
set -a; source /etc/ztakora/network.env; set +a
node build/network.mjs \
--rpc https://YOUR-ENDPOINT --book state/base-sepolia.json \
--state state/base-sepolia --chain-id 84532 \
--target 0.000045 --stagger 3000 --respawn 3Six roles, one variable each: ISSUER, VERIFIER1, VERIFIER2, VERIFIER3, OPS and WATCHER. There is no --env flag; passing one is refused rather than ignored. Restart is the same command — the journal is keyed by chain id, so a role that comes back finds its own work recorded as done instead of doing it twice. --stagger exists because six processes connecting in the same millisecond is one request burst, and a public endpoint answers a burst with 429. Ctrl-C stops all six; the supervisor forwards the signal, so nothing is left orphaned and arguing over one nonce.
--collateral fixes the bond every verifier has to clear and --fund is what a faucet-sized deployer pays the rest of the cast with. Four amounts, one arithmetic — required = 0.831 × collateral, printed before the first signature. deploy refuses a bond below it rather than bringing up a network that is locked out after its first settlement.
# read the network without signing anything — safe on a machine with no keys
node node/src/cli.ts status --rpc https://YOUR-ENDPOINT --book state/base-sepolia.jsonstatus and audit never sign, so they are safe to hand to an auditor or to run from a monitoring cron. Point them at the endpoint rather than --devnet: an in-process EVM exists only inside the process that started it, so status --devnet would deploy a fresh chain and report that one — a healthy-looking network of zeros, which is worse than an error. The CLI refuses to do it.
— The clock
Who has to act, and what happens when they do not
An epoch is blocksPerEpoch blocks; the phase is derived from block.number, not from a timer in your process. A node whose clock is wrong is a node that is simply late.
| Phase | Who acts | What happens if they do not |
|---|---|---|
| Commit · 240 blocks | attester | the bond is slashed for a skip, and its price is missing from the quorum |
| Reveal · 240 blocks | attester | the same slash; the pre-image it never revealed is why it is stored on disk |
| Settle · 240 blocks | aggregator | nothing is applied — and the line of every dormant book decays anyway |
| Grace · 20 blocks | anyone | slack before a phase is declared missed, so one slow block does not become a slashed verifier |
The node waits for the boundary; it does not force one
The scheduler polls and waits. On a chain you do not own there is no evm_mine, and the client refuses to call it. issuerLoop does its issuance in Settle, not in the commit window, because the ceiling a mint is checked against is the one this epoch agreed — a node that mints first and reads the line second is guessing.
Pick the epoch length with the block time in mind
At 720 blocks a one-second chain settles every 12 minutes; a twelve-second chain takes 2.4 hours, with every daemon waiting the boundary either way. The clock is measured in blocks precisely so that a verifier cannot choose which calendar week it benefits from by withholding a reveal.
— State
The three files a role writes
All JSON, all written atomically — a temp file and a rename. Keep the directory off shared mounts: it is mode 0600 for the pre-images, and a network filesystem is where those permissions go to be ignored.
| File | Holds | Losing it costs |
|---|---|---|
<role>.jsonl | append-only journal, one line per action | history; the chain still has the truth |
<role>.done.json | which once(key) work has already run | a replayed action — safe, everything is idempotent, but noisy |
<role>.state.json | pending blind commitments: cycle, ceiling, salt | 1% of the bond, and the only file here an attacker wants before reveal |
The .done.json keys are namespaced by chain id, so a restart onto a different network cannot inherit a journal's idea of what it already did. They are written after success, deliberately: a crash between the transaction landing and the write replays the action, and every settlement and enforcement entry point is idempotent, whereas recording first would silently skip an epoch whose transaction never landed.
The five verifier slots are five attester processes with five different key variables and, on a shared box, five different state directories — the attester's state file is named after its address precisely so that two of them can coexist.
tail -n 20 state/aggregator.jsonl | jq -c '{epoch, action, ok, error}'
grep -c '"ok":false' state/*.jsonlThe journal needs no parser. One of the three lines above is how a silent role is found before the epoch closes: a refusal is logged with the numbers the node demanded, and the network still looks idle.
— Funding
The arithmetic that stops a testnet run
On an OP Stack chain the sequencer charges an L1 data fee from the same L2 balance, and the acceptance test is gasLimit × gasPrice + L1 fee + value ≤ balance. No rule in the protocol caused this deployment to go quiet; this did.
# measured on this deployment at gasPrice = 6.6e6 wei
21 000-gas transfer, 68 bytes want 1.73e12 wei (the L2 part is 1.4e11 — 8% of it)
60 000-gas contract call want 1.89e12 wei
declare of a four-asset book want 5.4e12 weieth_estimateGas on Base answers in L2 gas including the L1 component, so a client that signs at the estimate plus 30% lands on top of a number that is already padded. The transport used to sign at a 15-million-gas default, which made a verifier holding 98 968 937 297 061 wei unable to sign a commitment that would have spent a tenth of that.
Three consequences
- Size a key for epochs, not for one transaction. A Base Sepolia epoch at the size of this deployment is ~14 signed transactions, so 1.6e13 wei per epoch:
--fundis epochs you want to survive × transactions per epoch × the current want. - A key too poor to sign is a silent role, not an errored one. The journal line to look for is
insufficient funds for gas * price + value, and the number worth reading is thewant, not thehave. - Once every key is below the L1 fee the network cannot even be drained — a sweep is a transaction too. The route back is through the contracts, which hold the capital:
LiabilityRegistry.requestCoverWithdraw(bytes32(0), amount)thenclaimCoverWithdraw()aftercoverDelay.
The fee is not a constant
It tracks Ethereum mainnet. The same call was measured at 1.574e12, 1.153e12, 1.005e12 and 0.844e12 wei across the twenty-five minutes from 16:56 to 17:21 UTC on the run that produced the numbers above — a 2-fold swing inside the time it takes an epoch to reach its settle window. A waiting role recovers by itself, because the transport hands back a nonce the chain refused to take; a daemon that keeps printing the same want is waiting on the fee market rather than broken.
A refusal is free, and it is the cheapest measurement on the chain
eth_sendRawTransaction rejects before the transaction exists, so nothing is spent and the node still returns the whole reservation in the error text. A loop that asks "can this key pay for this call now?" by making the call costs zero while the answer is no, and one transaction the moment it is yes. That is how this deployment's treasury was found to be locked rather than gone: the issuer's cover in LiabilityRegistry is several times the sum of every key balance in the run, and it leaves through the contract's own two-step gate.
What this run could not buy, stated plainly: at the fee level it ended at, the two cover calls together need ~1.9e12 wei of headroom on the issuer key and it holds 0.49e12, and moving money from the keys that can still sign costs almost as much as it moves. That is why the network's aggregate ceiling is currently zero — not because a rule failed, but because no epoch has been paid for yet. The reading is on the network page, the addresses are on the contracts page, and the quorum arithmetic is below.
— Playbook
Each failure the contracts can produce
Every entry is a message a node actually printed. Several of them are not incidents; the distinction is the useful part of this section.
An address that "has nothing on it" is usually an unverified one
A Blockscout page for a deployed but unverified contract answers is_verified: false with no name, no ABI and an empty Read tab, and shows 20 KB of runtime bytecode where a reviewer expected a contract — which reads, from the link, exactly like "nothing was deployed". Three things settle it without any explorer: the creation transaction (eth_getTransactionReceipt of the deploy), the length of eth_getCode compared to deployedBytecode in build/artifacts/<Name>.json — this deployment is 20 215 bytes on both sides — and status reading state out of the contract itself. Verification is a publication decision, not a build step: see the verification section.
no key for attester (on-chain role verifier1)
The variable named is the one this process has, which is not the same question as whether a file on disk contains it. Three ways this happens, in order of how often: an --env <file> that the supervisor never read (it now refuses with the list of flags it does take — load the file in the shell instead); an attester started straight from the CLI, since every attester signs as VERIFIER1 and it is the supervisor that hands the second and third their own keys under that name in their child environments; and a key that is present but empty.
Quorum is why the supervisor prints this before spawning anything. A bonded quorum is two thirds of the stake that posted a bond — with three equal bonds, an epoch with two attesters settles nothing while the books keep publishing and the epochs keep advancing. A network missing an attester looks idle. It is not idle.
address book is for chain 31337, this endpoint is chain 84532
You are holding the rehearsal book and pointing at a real chain, or the reverse. The book records its chain id for exactly this moment; the addresses exist on both, which is why this is a refusal rather than a warning. deploy also refuses an endpoint whose chain id is not the one you named — it does not guess.
Backstop has no code at 0x0…0 on chain 84532
A book that was never deployed, or a deploy that did not get mined. Do not patch the file: redeploy or find the real book. Every component is checked for code before load returns, so this failure means one of eight addresses is decorative.
mint refused: AboveLoadLine(…) — not an incident
The node computed a want against the headroom it read, and the chain re-read its own line inside the same transaction; settlement landed in between. It is in the journal as mint with ok:false and the three numbers the contract reported. Read the second argument as the current line. issuance held back: want …, headroom … is the same rule seen a moment earlier: the target exceeds what the line supports, so nothing was signed. Raise --target only if the book supports it.
no declared book — nothing to settle, and its line decays anyway
The aggregator's normal log for an issuer that has gone quiet. A dormant book is skipped, not settled by force; settle on an undeclared manifest reverts ManifestMismatch(0, 0) and the node does not send it. When that silence is a funding problem, the preceding line is the issuer's failed commit with a want larger than its balance — left alone, that looks exactly like an idle network: epochs advance, the aggregator reports no book, the watcher audits a spread of 0 ppm, and every one of those lines is honest about a book that stopped being published because its key could not pay the reservation.
submitted book does not hash to the declared manifest
The issuer's publication is two transactions, declare then submitBook, and a verifier whose commit window opened between them reads half a book: this epoch's manifest on top of last epoch's collateral. The node refuses both, because committing to a book you have not actually read is not a cautious version of attesting — it is attesting to whichever half the RPC answered first. So it looks again every three seconds until eight blocks remain, and only then abstains. Those wait lines are the normal sound of six processes on one public clock, loudest straight after a restart.
already declared — this node has the work recorded as done
The issuer restarted into a journal from the same chain. Nothing is wrong. If the chain id had changed the key would not have matched, so this message is proof the journal is being used the way it was designed.
An attester missed the commit window
Nothing to recover. It did not participate, its price is absent from the quorum, and if it committed and then failed to reveal, that is the slash. The salt and pre-image it stored is what lets it reveal after a crash — if that state file is gone, the commitment is unrecoverable and the bond pays. This is the one failure in the system that costs money rather than time.
A transaction is stuck, or a nonce is too low
Each send takes the chain's current nonce and keeps a local watermark above it, so two actions in one tick cannot collide. When a transaction is never mined the client says so and says what to do: no receipt for 0x… after 20000ms at height 1841 — the transaction is pending or dropped; resubmit at the same nonce with a higher gas price. The reliable way to get a self-collision is to run two processes with the same key. The same two messages arrive from a restart: the killed process left a transaction in the pool, so the new one's first send reuses a nonce the chain has moved past. That is one failed tick, not a stuck node — the watermark rebuilds from the chain and the next boundary works. Both appeared in this deployment's journals at its two restarts, and nothing needed recovering after them.
recordShadow comes back NotBonded(0x…)
The gate's signal input is restricted to bonded addresses, because a shadow report is a veto and an unfounded veto has to come from capital that is at risk in the epochs it would freeze. A watching node that is not bonded can still audit and still print its digest — the loop logs the refusal rather than assuming success — but it cannot speak to governance. Bond the watcher, or accept that its reports are advisory.
The aggregator is down
advance and settle are permissionless: any node may roll the epoch boundary, and the first caller wins the gas and nothing else. A watcher or a verifier can do the aggregator's job if it has to; it will just not be logged under aggregator.jsonl. What you cannot get from anyone else is punish and record, which need the ops key.
You need to stop the protocol
setPaused(true) from the deployer key is the only kill switch and it is owner-only; the test suite proves a stranger's attempt reverts NotAuthorised(addr,"ztakora"). A paused chain still advances epochs — settlement of the epoch already in progress cannot be stopped, which is intentional: the line you already agreed is the line that applies.
— Parameters
Changing anything is slower than that on purpose
The parameter set is not a dial an operator turns. A candidate goes through GovernanceGate.
- Three independent endorsements.
- Four epochs of observation.
- A shadow-report signal above 50% of the weight.
- Every moved field within ±15% of its current value.
requiredMaskandblocksPerEpochare not movable at all — one is a policy question, the other is the unit the whole clock is counted in, and a set that renumbers epochs would rewrite which past epoch it claims to have been in force.
Those four numbers are set, not compiled in. The gate's constructor defaults are 2 endorsements, 30 observation epochs and a ±25% band; deploy replaces them with 3, 4, 50% and ±15% as its last wiring step, before it writes the address book. So a deployment that stopped early leaves no book rather than a loose gate — but a network you brought up by hand, or with a script that is not this one, is live at the constructor defaults until you check. They are public getters: requiredEndorsements, requiredObservation, requiredSignalPpm, bandPpm.
Key rotation is proposeOwnership plus acceptOwnership: two signatures, and a proposal is not a transfer.
— Boundaries
What this runbook does not cover
Stating the boundary is part of the document; the last line of the repository README is honest about what remains before anything here touches real collateral.
No managed images, no key management, no monitoring
There is no managed-image story, no key-management deployment and no monitoring integration in this repository. deploy/env.example and the systemd templates under deploy/systemd/ are the current boundary.
No breach procedure, because nothing here can drive one
Ztakora.reportLoss is callable by the owner, a registered watcher or the liability registry, and no daemon in the node calls it — there is no report-loss command, by design rather than by omission: a node that could decide a loss unilaterally would be a node holding a red button. Until a liquidator procedure is written, a breach is an out-of-band act. What each of the first, third and fourth claims contributed is in the Waterfall event of the call itself; step two is the issuer's unearmarked cover and leaves no trace but its absence, so check issuerCover afterwards, then read the outstanding shortfall and whether settlement is blocked with status.