Reset or remove a compute node
Two operations take a compute node (CN) out of service, and they are not interchangeable. Factory reset keeps the node's record and re-arms the netboot enrollment pipeline so the node's next boot lands back in setup. Remove deletes the record and decommissions the node's boot MACs so the hardware stops getting a boot answer.
Both operations delete the node's bound API key inside the store transaction, so
the node's tritonagent can no longer authenticate. Remove additionally deletes
the compute node record and its capacity, placement, load-summary, reservation,
NIC-tag, and legacy-VM rows. Neither operation is undone by the other: a reset
node has to be re-approved for setup, and a removed node has to be reactivated
per MAC and enrolled from the start.
Which one you want
| Factory reset | Remove | |
|---|---|---|
| Endpoint | POST /v1/cns/{server_uuid}/factory-reset | DELETE /v1/cns/{server_uuid} |
tritonadm verb | none | none |
| Accepted starting states | approved, disabled | disabled, pending, awaiting_setup |
| Ending state | awaiting_setup | record deleted |
| Bound API key | deleted | deleted |
| Boot records | enrolled back to enrolling, address and networking kept | any state to decommissioned |
| Data on the node's disks | untouched | untouched |
| Audit history | kept | kept |
Neither call touches the node itself. The node keeps running until you reboot it, and its ZFS pool and datasets are left exactly as they are.
Before you start
Both actions require a root operator. A full-scope API key clears the scope
gate; the read_only, audit_only, agent, booter, and ur_enroll scopes
are refused there. Authorization then comes from the root-operator Cedar rule,
the same seam that protects setup approval.
Both are refused with 409 Conflict while instances are placed on the node, while
a placement reservation is in flight, or while a non-terminal migration names the
node as source or target. Move or delete the instances first. To see where each
hosted instance would land, POST /v1/cns/{server_uuid}/drain/preview runs the
placement engine read-only and returns the placeable and not-placeable sets.
There is no tritonadm subcommand for either operation. tritonadm cn covers
list, show, approve, disable, label, reservoir, and auto-approve
only. Use the HTTP API or the operator console.
The tritonadm cn commands are control-plane calls, so they run from any host
where tritonadm is signed in to the cluster. The examples below use the head
node. On the head node, as root, load the endpoint and token into the shell:
eval "$(tritonadm env)"
That sets TRITONADM_ENDPOINT and TRITONADM_ACCESS_TOKEN, which the curl
commands below use.
Factory reset a compute node
Use this when the same hardware is going back into this cloud: a rebuild onto a different disk layout, a new hostname, or a node whose control-plane state you want rebuilt from scratch.
Step 1: confirm the node hosts no instances
On the head node, as root:
tritonadm system cn-instances <CN_UUID>
<CN_UUID> is the node's server_uuid, the first column of tritonadm cn list.
Step 2: check that an enrollment platform image is configured
Factory reset refuses when the site boot policy has no valid enrollment_pi,
because the reset node would land in enrolling with nothing to boot. On the
head node, as root:
curl -sS -H "Authorization: Bearer $TRITONADM_ACCESS_TOKEN" \
"$TRITONADM_ENDPOINT/v2/cn-boot-policy"
The enrollment_pi field of the response carries the platform image stamp
served to enrolling nodes.
Step 3: reset the node
On the head node, as root:
curl -sS -X POST -H "Authorization: Bearer $TRITONADM_ACCESS_TOKEN" \
"$TRITONADM_ENDPOINT/v1/cns/<CN_UUID>/factory-reset"
The response is the updated compute node record. In one transaction the store
moves the node to awaiting_setup, deletes the bound API key, clears
bound_api_key_id, pending_credential, the three per-node ticket keys,
assigned_pi, disk_layout, approved_at, and every setup field, and records
the reason factory reset by operator in last_setup_error.
It also flips every boot record still bound to this node from enrolled back to
enrolling. Records are found by their server_uuid binding rather than by
sysinfo, so a NIC the node no longer reports is re-armed too. Each record keeps
its address, networking spec, and provenance.
Step 4: reboot the node
Nothing reboots the node for you, and the node does not re-enter enrollment
until it boots again. Reboot it from its console, or drive its BMC. On the host
where triton-booter is installed, as root:
triton-booter redfish --host <BMC_HOST> --user <BMC_USER> \
--password-file <PASSWORD_FILE> set-onetime-boot --target uefi-http
triton-booter redfish --host <BMC_HOST> --user <BMC_USER> \
--password-file <PASSWORD_FILE> power-cycle
--target accepts uefi-http (the default) or pxe. Add
--accept-invalid-certs when the BMC serves a self-signed certificate. The
password is read from <PASSWORD_FILE>, never from the command line.
Step 5: re-approve the node for setup
Wait until the node is back in the enrollment platform image and announcing
before you approve. The boot service takes the one-shot setup bundle only once
it has a live connection to the node, and approval stamps a deadline of
approved_at plus 45 minutes. The reconciler reverts any node still in
setting_up past that deadline to awaiting_setup with the reason cn setup exceeded its deadline without reporting complete/failed; reverted by reconciler.
Approving is the same call as a first-time enrollment. See Enroll a compute node for the disk-layout inputs and the layout preview endpoint.
What the next setup run does to the disks
Factory reset destroys no data. What happens to the pool depends on what the setup script finds on the node when it runs:
State of the pool named zones | What setup does |
|---|---|
| Absent | Runs disklayout and the destructive zpool create, building a new pool from the approved layout. |
Present, org.smartos:server_uuid on zones/var matches the approved server UUID | Resumes. The zpool create block is skipped entirely and the existing datasets are kept. |
| Present, stamp missing or belonging to another server | Refuses at step refuse_existing_pool and exits 20. Enrollment never destroys an unrecognized pool. |
To rebuild a node onto a different disk layout, destroy the existing zones
pool yourself first. Set rescue mode on one of the node's MACs with
POST /v2/cn-boot/{mac}/rescue, boot the node, and destroy the pool from there.
Re-approval on its own resumes the pool that is already present.
Factory reset does not clear the node's stored global-zone root password hash.
The hash stays on the compute node record and is handed to the next setup run
in the setup bundle. To clear it, POST /v1/cns/{server_uuid}/root-password
with both password and password_hash omitted or empty before you reset.
Remove a compute node
Use this when the hardware is leaving this cloud.
Step 1: disable the node
Remove is disable-first. An approved node is refused until you disable it. On
the head node, as root:
tritonadm cn disable <CN_UUID>
Output:
Disabled CN <CN_UUID> (state=disabled)
Disable flips the record to disabled and clears the claim code and any pending
credential. A node in pending or awaiting_setup is already removable and does
not need this step. A node in setting_up has to finish or fail first.
Disable does not delete the node's bound API key. The store transaction leaves
bound_api_key_id in place and the handler's revocation step is a placeholder
that only logs. Factory reset and remove both delete the key; disable alone does
not. Treat a disabled node's agent as still credentialed.
Step 2: remove the record
On the head node, as root:
curl -sS -i -X DELETE -H "Authorization: Bearer $TRITONADM_ACCESS_TOKEN" \
"$TRITONADM_ENDPOINT/v1/cns/<CN_UUID>"
Success is 204 No Content. Removing a node that is already absent also
succeeds, so the call is safe to repeat.
The store deletes the compute node row, its state, claim-code, and poll-token index entries, the bound API key rows, and the per-node satellite rows: capacity, placement, load summary, reservations, NIC-tag inventory, and this node's legacy-VM discovery rows. Audit chain entries, migration records, and the node's ClickHouse series are deliberately kept.
It then decommissions the node's boot records from two sources: every record
still bound to this server_uuid, plus every unbound record whose MAC appears in
the node's sysinfo. A MAC bound to a different server is left alone.
Step 3: confirm the boot records are silent
On the head node, as root:
curl -sS -H "Authorization: Bearer $TRITONADM_ACCESS_TOKEN" \
"$TRITONADM_ENDPOINT/v2/cn-boot"
Each of the removed node's MACs reads "state": "decommissioned". The boot
service gives a decommissioned MAC no DHCP reply and no boot script, so the
hardware cannot PXE back into the fleet. Rescue mode is the one exception: it is
served in every state, including decommissioned.
Bringing removed hardware back
Reactivate each MAC, then enroll the node again. On the head node, as root:
curl -sS -X POST -H "Authorization: Bearer $TRITONADM_ACCESS_TOKEN" \
"$TRITONADM_ENDPOINT/v2/cn-boot/<MAC>/reactivate"
<MAC> is the record's canonical lowercase colon-separated form, for example
0c:c4:7a:11:22:33. Colon, hyphen, dot, and unseparated input are all accepted.
Reactivate returns the record to discovered and clears its bound
server_uuid, while keeping its address, networking spec, and provenance. The
address matters: the boot service sends no DHCP reply to a record that has no
address, and the address is tritond's to assign, not a lease pool the boot
service falls back to.
Reactivate is idempotent on a record already in discovered, and returns 409
Conflict from an active (enrolling or enrolled) record.
When the call is refused
Every guard below returns 409 Conflict with the message shown.
| Message | Which call | What to do |
|---|---|---|
instances are still placed on this node; delete or migrate them first | both | Migrate or delete the hosted instances. |
a placement reservation is in flight for this node | both | Wait for the reservation to settle. |
a migration involving this node is in flight; wait for it to finish or abort it | both | Wait, or abort the migration. |
no valid enrollment PI is configured in the boot policy; the reset node could not netboot back into enrollment | factory reset | Set a valid enrollment_pi in the boot policy and stage it on the boot service. |
cn is <STATE>; only an approved or disabled node can be factory reset | factory reset | Reset applies to approved and disabled nodes only. |
cn is approved; disable it before removing | remove | Run tritonadm cn disable first. |
cn is <STATE>; wait for setup to finish or fail before removing | remove | A setting_up node has to reach a terminal outcome first. |
See also
- Enroll a compute node covers the approval that a reset node needs next.
- Compute node lifecycle explains the two state machines these operations move.
- tritonadm cn is the reference for the CLI commands used here.
- Installation overview puts compute node enrollment in the order of a full install.