Skip to main content

Reset or remove a compute node

Two operations take a compute node (CN) out of service, and they are not interchangeable. Factory reset keeps the node's record and re-arms the netboot enrollment pipeline so the node's next boot lands back in setup. Remove deletes the record and decommissions the node's boot MACs so the hardware stops getting a boot answer. Neither operation is undone by the other.

Which one you want​

Factory resetRemove
EndpointPOST /v1/system/hardware/cns/{cn}/factory-resetDELETE /v1/system/hardware/cns/{cn}
tritonadm verbnonenone
Accepted starting statesapproved, disableddisabled, pending, awaiting_setup
Ending stateawaiting_setuprecord deleted
Bound API keydeleteddeleted
Boot recordsenrolled back to enrolling, binding cleared, address and networking keptany state to decommissioned
Data on the node's disksuntoucheduntouched
Audit historykeptkept

Neither call touches the node itself. The node keeps running until you reboot it, and its ZFS pool and datasets are left exactly as they are.

Before you start​

Both actions require a root operator. A full-scope API key clears the scope gate. Every other scope (read_only, tenant_write, audit_only, agent, booter, ur_enroll, dnsedge, and certs) is refused there. Authorization then comes from the root-operator Cedar rule, the same seam that protects setup approval.

Both are refused with 409 Conflict while instances are placed on the node, while a placement reservation is in flight, while a non-terminal migration names the node as source or target, or while the node holds or coordinates MantaBlock volumes. Move or delete the instances first. To see where each hosted instance would land, POST /v1/system/hardware/cns/{cn}/drain/preview runs the placement engine read-only and returns the placeable and not-placeable sets.

Factory reset, remove, rescue mode, and boot-record reactivate have no tritonadm subcommand. Use the HTTP API for those. The other steps use tritonadm.

The tritonadm commands are control-plane calls, so they run from any host where tritonadm is signed in to the cluster. The examples below use the head node. <CN> is the node's hostname, UUID, or a UUID prefix. <CN_UUID> is its full server_uuid, which the API paths need. Print it with tritonadm cn show <CN>. On the head node, as root, load the endpoint and token into the shell:

eval "$(tritonadm env)"

That sets TRITONADM_ENDPOINT and TRITONADM_ACCESS_TOKEN, which the curl commands below use.

Factory reset a compute node​

Use this when the same hardware is going back into this cloud: a rebuild onto a different disk layout, a new hostname, or a node whose control-plane state you want rebuilt from scratch.

Step 1: confirm the node hosts no instances​

On the head node, as root:

tritonadm instance list --cn <CN>

An empty node prints the header row only.

Step 2: check that an enrollment platform image is configured​

Factory reset refuses when the site boot policy has no valid enrollment_pi, because the reset node would land in enrolling with nothing to boot. On the head node, as root:

tritonadm cn boot policy show

The enrollment_pi line carries the platform image stamp served to enrolling nodes. - means none is set. Set one with tritonadm cn boot policy set --enrollment-pi <PI_STAMP>.

Step 3: reset the node​

On the head node, as root:

danger

Factory reset deletes the node's bound API key, so the node's tritonagent can no longer authenticate. There is no undo. The node returns to service only after you re-approve it for setup.

curl -sS -X POST -H "Authorization: Bearer $TRITONADM_ACCESS_TOKEN" \
  "$TRITONADM_ENDPOINT/v1/system/hardware/cns/<CN_UUID>/factory-reset"

The response is the updated compute node record. In one transaction the store moves the node to awaiting_setup, deletes the bound API key, clears bound_api_key_id, pending_credential, the three per-node ticket keys, assigned_pi, disk_layout, approved_at, and every setup field, and records the reason factory reset by operator in last_setup_error.

It also flips every boot record still bound to this node from enrolled back to enrolling. Records are found by their server_uuid binding rather than by sysinfo, so a NIC the node no longer reports is re-armed too. Each record loses its server_uuid binding and its assigned_pi, and keeps its address, networking spec, and provenance.

Step 4: reboot the node​

Nothing reboots the node for you, and the node does not re-enter enrollment until it network-boots. A plain reboot is not enough: setup installed a local boot tree and boot blocks on the node's zones pool, so a plain reboot boots that pool. Set a one-time network boot from the firmware boot menu on its console, or drive its BMC. On the host where triton-booter is installed, as root:

triton-booter redfish --host <BMC_HOST> --user <BMC_USER> \
  --password-file <PASSWORD_FILE> set-onetime-boot --target uefi-http
triton-booter redfish --host <BMC_HOST> --user <BMC_USER> \
  --password-file <PASSWORD_FILE> power-cycle

--target accepts uefi-http (the default) or pxe. Add --accept-invalid-certs when the BMC serves a self-signed certificate. The password is read from <PASSWORD_FILE>, never from the command line.

Step 5: re-approve the node for setup​

Wait until the node is back in the enrollment platform image and announcing before you approve. The boot service takes the one-shot setup bundle only once it has a live connection to the node, and approval stamps a deadline of approved_at plus 45 minutes. The reconciler reverts any node still in setting_up past that deadline to awaiting_setup with the reason cn setup exceeded its deadline without reporting complete/failed; reverted by reconciler.

Approving is the same command as a first-time enrollment.

danger

Approving setup authorizes the installer to build a pool on the node's disks. When no zones pool is present, the confirmed layout's disks are wiped. There is no undo, and the command does not ask for confirmation.

On the head node, as root, preview the layout, then approve:

tritonadm cn setup plan <CN>
tritonadm cn setup approve <CN> --layout <LAYOUT>

See Enroll a compute node for the disk-layout flags and what each rejection means.

What the next setup run does to the disks​

Factory reset destroys no data. What happens to the pool depends on what the setup script finds on the node when it runs:

State of the pool named zonesWhat setup does
AbsentRuns disklayout and the destructive zpool create, building a new pool from the approved layout.
Present, org.smartos:server_uuid on zones/var matches the approved server UUIDResumes. The zpool create block is skipped entirely and the existing datasets are kept.
Present, stamp missing or belonging to another serverRefuses at step refuse_existing_pool and exits 20. Enrollment never destroys an unrecognized pool.

To rebuild a node onto a different disk layout, destroy the existing zones pool yourself first. Set rescue mode on one of the node's MACs with POST /v1/system/hardware/cn-boot/{mac}/rescue, boot the node, and destroy the pool from there. Re-approval on its own resumes the pool that is already present.

Factory reset does not clear the node's stored global-zone root password hash. The hash stays on the compute node record, and the next setup run reads it when it takes the setup bundle. To clear it, run tritonadm cn root-password <CN> --clear before you re-approve setup.

Remove a compute node​

Use this when the hardware is leaving this cloud.

Step 1: disable the node​

Remove is disable-first. An approved node is refused until you disable it. On the head node, as root:

tritonadm cn disable <CN>

Output:

This will make the following changes:
    disable CN <CN_HOSTNAME> (<CN_SHORTID>), currently approved; no new instances place there
Would you like to continue? [y/N]

Answer y. In a script, pass -y instead; without a terminal the command refuses to run unless -y is given.

Disable flips the record to disabled and clears the claim code and any pending credential. A node in pending or awaiting_setup is already removable and does not need this step. A node in setting_up has to finish or fail first.

Disable also revokes the node's credential: tritond refuses the bound API key of a node that is not approved or setting_up. The key record stays until remove or factory reset deletes it.

Step 2: remove the record​

On the head node, as root:

danger

Remove deletes the compute node record, its bound API key, and its capacity, placement, load-summary, reservation, NIC-tag, and legacy-VM rows. There is no undo. To bring the hardware back, reactivate each MAC and enroll the node from the start.

curl -sS -i -X DELETE -H "Authorization: Bearer $TRITONADM_ACCESS_TOKEN" \
  "$TRITONADM_ENDPOINT/v1/system/hardware/cns/<CN_UUID>"

Success is 204 No Content. Removing a node that is already absent also succeeds, so the call is safe to repeat.

The store deletes the compute node row, its state, claim-code, and poll-token index entries, the bound API key rows, and the per-node satellite rows: capacity, placement, load summary, reservations, NIC-tag inventory, and this node's legacy-VM discovery rows. Audit chain entries, migration records, and the node's ClickHouse series are deliberately kept.

It then decommissions the node's boot records from two sources: every record still bound to this server_uuid, plus every unbound record whose MAC appears in the node's sysinfo. A MAC bound to a different server is left alone.

Step 3: confirm the boot records are silent​

On the head node, as root:

tritonadm cn boot list

Each of the removed node's MACs reads decommissioned in the STATE column. The boot service gives a decommissioned MAC no DHCP reply and no boot script, so the hardware cannot PXE back into the fleet. Rescue mode is the one exception: it is served in every state, including decommissioned.

Bringing removed hardware back​

Reactivate each MAC, then enroll the node again. On the head node, as root:

curl -sS -X POST -H "Authorization: Bearer $TRITONADM_ACCESS_TOKEN" \
  "$TRITONADM_ENDPOINT/v1/system/hardware/cn-boot/<MAC>/reactivate"

<MAC> is the record's canonical lowercase colon-separated form, for example 0c:c4:7a:11:22:33. Colon, hyphen, dot, and unseparated input are all accepted.

Reactivate returns the record to discovered and clears its bound server_uuid, while keeping its address, networking spec, and provenance. The address matters: the boot service sends no DHCP reply to a record that has no address, and the address is tritond's to assign, not a lease pool the boot service falls back to.

Reactivate is idempotent on a record already in discovered, and returns 409 Conflict from an active (enrolling or enrolled) record.

When the call is refused​

Every guard below returns 409 Conflict with the message shown.

MessageWhich callWhat to do
instances are still placed on this node; delete or migrate them firstbothMigrate or delete the hosted instances.
a placement reservation is in flight for this nodebothWait for the reservation to settle.
a migration involving this node is in flight; wait for it to finish or abort itbothWait, or abort the migration.
the node is registered for MantaBlock and still takes new volumes; disable it firstbothRun tritonadm cn disable first.
the node holds or coordinates <N> MantaBlock volumes; move or delete them firstbothMove or delete the volumes.
no valid enrollment PI is configured in the boot policy; the reset node could not netboot back into enrollmentfactory resetSet a valid enrollment_pi in the boot policy and stage it on the boot service.
cn is <STATE>; only an approved or disabled node can be factory resetfactory resetReset applies to approved and disabled nodes only. <STATE> prints in Rust debug form, for example SettingUp or AwaitingSetup.
cn is approved; disable it before removingremoveRun tritonadm cn disable first.
cn is SettingUp; wait for setup to finish or fail before removingremoveA setting_up node has to reach a terminal outcome first.

See also​