Skip to main content

Compute node lifecycle

A compute node (CN) does not have one lifecycle. It has two, and they run on different keys.

  • Cn.state tracks the control-plane registration, keyed by the SmartOS server_uuid the node reports from /usr/bin/sysinfo. It answers "does tritond consider this machine part of the fleet, and does it hold a credential?"
  • BootRecord.state tracks the netboot identity of one admin-network MAC, keyed by that MAC. It answers "when this NIC sends a DHCP DISCOVER, what does triton-booter serve it?"

A single machine has exactly one Cn row and one BootRecord per admin NIC. The two advance independently and are only stitched together at two moments: when setup completes, and when the node is reset or removed. Reading a fleet problem as one state machine is the most common way to misdiagnose it.

Registration states​

CnState has exactly five variants. The wire names below are what the API returns and what the operator console displays.

StateWhat it meansWhat the control plane holdsLeaves via
pendingtritonagent self-registered and is waiting for approvalAn active claim_code and a poll_token. No bound credentialClaim-code approval or an open auto-approve window, disable, or delete
approvedActive member of the fleetThe bound Agent-scoped API key that authenticates the rest of /v1/agent/*disable, factory reset
disabledAn operator took the node out of serviceThe row is kept for audit visibilityAnonymous re-registration re-arms it to pending; factory reset; delete
awaiting_setupA bare node announced itself over the boot bus but is not running tritonagentAnnounced sysinfo (including the disk inventory) and last_seen. No claim code, no long-poller, no credentialOperator approve-setup, or delete
setting_upAn operator authorized the destructive install and the on-node work is in flightA minted bound API key plus three ticket keys stashed for one-shot delivery, a setup_epoch, a setup_deadline, and a pinned assigned_pisetup-complete, setup-failed, or the deadline reconciler

CnState is #[non_exhaustive] in the store types, so a client that pattern matches on it needs a fallback arm.

Two edges in that diagram are worth reading twice. disabled --> pending is the supported "re-enable with fresh credentials" path: a fresh anonymous POST /v1/agent/register from the same server_uuid drops the record back to pending, mints a new claim code and poll token, and clears any bound credential. The disable event stays in the audit chain. And there is no edge into setting_up other than the operator one.

Netboot states​

BootState has exactly four variants, plus an orthogonal BootMode of normal (the default) or rescue.

StateWhat triton-booter servesNotes
discoveredNothing. The booter stays silentCreated on first sighting. A record can also be synthesized on read for a MAC that was never reported, without writing a row
enrollingThe enrollment platform image from BootPolicy.enrollment_piUnservable when enrollment_pi is unset, fails the stamp charset gate, or is not staged on the booter
enrolledThe record's own assigned_piUnservable without a staged assigned_pi. This boot also carries the CN root password hash as -B root_shadow= when one is set
decommissionedNothingTerminal until an operator reactivates the record

Rescue mode sits outside this diagram on purpose. BootMode::Rescue is honored in every state, decommissioned included, and the rescue render drops the record's kernel arguments, the boot-archive hash, and the networking module. A broken per-node config cannot block recovery, and a retired machine can still be booted to be inspected.

The booter never sends a DHCP NAK. Every negative outcome is silence: a non-PXE-shaped client when answer_only_pxe is true (the default), a discovered or decommissioned record, a servable record with no reserved address, or a REQUEST that names another server in option 54. Silence means you diagnose netboot from the booter's own logs and counters, not from the client.

How a node enters the fleet​

Two entry paths exist, and they enter the registration state machine at different points.

Network boot​

A bare machine on the admin network PXE-boots, the booter reports the sighting to tritond as discovered, and the site BootPolicy decides what happens next. auto_enroll defaults to true, because reachability of a dedicated admin DHCP segment is the enrollment trust boundary. When it is on and the optional allowlist matches, tritond atomically reserves the lowest free address from BootPolicy.admin_network, writes the NetworkingSpec, and moves the record to enrolling. An empty allowlist with auto_enroll on admits everything on that segment. Allowlist entries take exactly two forms: an exact iDRAC service_tag matched case-insensitively, or a canonical-form mac_prefix.

The node then boots the enrollment platform image, whose stock ur-agent announces over the booter's AMQP subset. The booter validates identity before it calls tritond at all: sysinfo has to be a JSON object with a parseable UUID, the nil UUID and the all-0xFF unprogrammed SMBIOS value are rejected, and the topic-derived routing UUID has to equal the UUID in the body. tritond independently rejects those two reserved UUIDs again at its own boundary. The resulting Cn row lands in awaiting_setup: inert, with no credential and no claim code, holding only the announced sysinfo that the disk inventory is read from.

Turning auto_enroll off restores strict mode. A new server_uuid can then create an awaiting_setup record only while an operator has a UrEnrollWindow open with a live slot; otherwise the announce is rejected with 403. Window duration is clamped server-side to a 24-hour hard cap. Opening a window never authorizes a disk wipe.

Manual agent install​

A machine already running a SmartOS global zone skips netboot entirely. tritonagent self-registers with POST /v1/agent/register, and tritond creates the record directly in pending with a six-character claim code drawn from a Crockford base32 alphabet that excludes the console-ambiguous letters I, L, O, and U. The code is displayed as XXX-XXX and is valid for one hour. The agent prints the code, mirrors it best-effort to /var/lib/tritonagent/claim-code, and long-polls GET /v1/agent/register/status for at most one hour before exiting. An operator approving the code moves the record to approved, and the agent takes the per-CN API key on its next poll. That plaintext key is delivered to the agent exactly once and is never shown to the operator, who sees only bound_api_key_id.

An auto-approve window skips the claim code entirely: register_cn consumes a slot atomically and creates the record already approved. The window is bounded by wall time and by an optional remaining-count budget, and its duration is clamped to the same 24-hour cap.

Because this path never touches BootRecord, a manually installed node has no netboot identity until one is created for it.

The one gate that matters​

POST /v1/cns/{server_uuid}/approve-setup is the load-bearing security seam of the whole lifecycle. It is the only mutation in the system that produces setting_up, and setting_up is the only state the booter will drain a credential bundle for.

danger

approve-setup authorizes a destructive pool build on the node's disks. The on-node script runs zpool create -n as a dry run first, then the real zpool create for the pool named zones. Approving the wrong server_uuid wipes the wrong machine.

The gate is enforced three ways at once:

  • It requires the operator action CnSetupApprove, which is denied to every scoped API key. ApiKeyScope has six values: full, read_only, audit_only, agent, booter, and ur_enroll. The booter holds two keys with deliberately different blast radii, and its ur_enroll key cannot reach this endpoint or open an enrollment window.
  • It gates on state before doing any work: any state other than awaiting_setup returns 409, so a replayed approval on an in-flight node conflicts rather than restarting an install.
  • It validates the operator's inputs at the boundary rather than on the node. The requested layout is realized against the announced disks using the disk layout library's own rules, so a bad topology is a 400 that never mints a credential. An operator-supplied hostname has to be an RFC-1123 label, because the value is written into node.config, which the platform config loader evaluates at every boot. disk_compress is checked against the same allowlist the on-node script uses.

What the approval writes is as important as what it refuses. It mints the bound Agent key and three HS256 ticket keys into a one-shot bundle, stamps a per-attempt setup_epoch, stamps a setup_deadline of the approval time plus 45 minutes, and pins assigned_pi from the boot policy's enrollment_pi as it stands at that instant. Pinning at approval time makes the node's future steady-state platform image immune to a policy edit during the minutes-long install.

There is no saga behind any of this. The durable CnState column is the state machine, and the operation_id in the approval response is an opaque correlation id for the audit trail, not something you can follow.

What "installing" actually means​

Once a node is setting_up, the booter's approval-poll loop takes the consume-once credential bundle only after it already holds a live connection to that node, and drives at most one install per node at a time. The script it ships is byte-identical for every node; everything node-specific arrives in an environment map.

Two properties of that script shape how failures present to you:

  • It sets the wall clock first, before the pool build and before any signature or TLS check, and fails closed if it cannot. A node whose clock cannot be set never reaches its disks.
  • It refuses rather than wipes. If a pool named zones already exists but its org.smartos:server_uuid stamp does not match the approved server UUID, setup stops. Enrollment never destroys an unrecognized pool.

The authoritative success signal is a sentinel line on the wire, not the exit status: the transport coerces the script's reboot exit code, so absence of the sentinel is treated as failure and the node is never marked approved on an unproven run.

How setup ends​

Both terminal transitions are gated on the setup_epoch the booter echoes back. A mismatch is a 409, so a delayed report from a superseded attempt cannot drive a freshly re-approved row.

  • Success. setup-complete moves setting_up to approved and, in the same atomic unit, flips every enrolling boot record among the node's MACs to enrolled, binding server_uuid and stamping assigned_pi. This is the one place the two state machines are joined. No boot-record write happens when assigned_pi is absent, so the node re-enrolls on its next PXE rather than becoming an enrolled record with nothing to serve.
  • Failure. setup-failed moves setting_up back to awaiting_setup, records the reason (capped at 512 bytes) in last_setup_error, and deletes the bound API key along with the pending credential and all three ticket keys. A failed install leaves no live key material anywhere; a re-approval mints fresh.
  • Silence. A reconciler reverts any setting_up row past its setup_deadline through the same failure path, with a fixed reason recorded on the row. A setting_up row carrying no deadline is reverted immediately. The guarantee is that a node never wedges in setting_up.

There is a second, quieter route from setting_up to approved. After the node reboots into its installed tritonagent, the agent authenticates a registration refresh with its bound key. When the setup bundle has already been consumed, that refresh promotes the row to approved on its own. This is the crash-recovery proof: it covers the case where the install finished but the booter's report did not land. A refresh arriving while the credential is still unconsumed conflicts, because the installer has not proven it took the bundle.

Anonymous registration cannot do any of this. It is refused for approved, setting_up, and awaiting_setup records, so an attacker on the admin network cannot replace a live node's sysinfo or its console TLS pin.

Leaving the fleet, and coming back​

Three operations remove capability, and they differ in how much they undo.

Disable flips the record to disabled. The node stops being a placement target and the record survives for audit. This is the reversible one: a fresh anonymous registration re-arms it to pending.

warning

tritonadm cn disable does not revoke the bound API key today. The handler logs a placeholder and the revocation code is unimplemented, even though the command help and the API documentation both claim revocation. Factory reset and setup failure do delete the key, inside the store transaction. Treat disable as "stop scheduling here", not as credential revocation.

Factory reset moves approved or disabled back to awaiting_setup. It destroys no data itself. What it does is re-arm the netboot pipeline: it deletes the bound key and the ticket keys, clears assigned_pi, disk_layout, approved_at, and every setup field, and flips every enrolled boot record bound to that server back to enrolling while keeping the record's address, networking, and provenance. The node's next boot re-enters enrollment, and the wipe happens only if you approve setup again. Reset is refused with 409 when instances are placed on the node, when a placement reservation is in flight, when a non-terminal migration references the node as source or target, or when no valid enrollment platform image is configured, because a reset node with nothing to boot is a dead end.

Delete is disable-first by contract: it returns 409 unless the row is disabled, pending, or awaiting_setup. A live approved node has to be deliberately disabled first, and a setting_up node has to finish or fail first. Deletion removes the Cn row, its index entries, the bound key rows, and the per-node satellite rows (capacity, placement, load summary, reservations, NIC-tag inventory, legacy-VM discovery). It also decommissions the node's boot records so the hardware cannot PXE back in as a dead identity. Audit chain entries, migration records, and ClickHouse series are deliberately kept, and deleting an already-absent node succeeds.

A decommissioned boot record is reversed with a reactivate, which returns it to discovered and clears the bound server_uuid while keeping the address, networking, and provenance. Hand-deleting the record instead loses the reserved admin address and produces a re-enrollment dead end.

Where the controls live​

The registration side of the lifecycle is partly covered by tritonadm cn: list, show, approve, disable, label set, reservoir set and reservoir clear, and auto-approve status, open, and close.

The rest is not. There is no tritonadm verb for approve-setup, factory reset, delete, setting the CN root password, boot records, the boot policy, rescue mode, or the enrollment window. Those live on the tritond HTTP API and in the operator console. tritonadm cn list --state also accepts only pending, approved, and disabled, so a node moving through netboot enrollment is invisible to that filter and shows up only in the unfiltered list, in --json output, or in the console.

Next steps​