Skip to main content

Apply an update

This guide moves a running cloud to the newest build on the channel it follows: first the control plane and the agents, then the other control-plane zones, then tritonadm itself. At the end, tritonadm versions reports each of those components as current.

Before you start, you need:

  • A root shell on the head node, in the global zone. Every command on this page runs there.
  • A tritonadm that can reach tritond, through a signed-in session or the head node's local socket.
  • The cluster's channel.upstream_url set. See Updates and release channels.

The order matters. tritond refuses a tritonadm built against a newer admin API version than it serves, so the CLI you already have updates the control plane, and the CLI is replaced last.

1. See what is out of date​

On the head node, as root:

tritonadm versions

Each row compares what a component runs with the channel's target, and marks it current or drift:

COMPONENT      WHERE            RUNNING           TARGET            STATE
agent          <CN_HOSTNAME>    <OLD_STAMP>       <NEW_STAMP>       drift
control plane  serving build    <OLD_STAMP>       <NEW_STAMP>       drift
control plane  triton-tritond1  <OLD_STAMP>       <NEW_STAMP>       drift
booter         triton-booter0   <OLD_STAMP>       <NEW_STAMP>       drift
fdb            triton-fdb1      <OLD_STAMP>       <NEW_STAMP>       drift
portal         triton-portal0   <OLD_STAMP>       <NEW_STAMP>       drift
...
tritonadm      this host        <OLD_STAMP>       <NEW_STAMP>       drift

A row whose TARGET is - has nothing in the channel to compare against.

2. Update the control plane and the agents​

danger

The control plane API is unavailable for about a minute while the tritond zone is reprovisioned. An agent that speaks no fleet API version the new control plane serves has some of its calls refused until its own refresh, later in the same run. Running instances keep running.

On the head node, as root:

tritonadm update tritond agents

The command prints the channel, the revision, and a plan, then asks Proceed?. Output:

channel edge, revision <NEW_STAMP>
COMPONENT      WHERE            NOW          TARGET       IMPACT
control plane  triton-tritond1  <OLD_STAMP>  <NEW_STAMP>  API unavailable for about a minute; snapshot zones/<ZONE_UUID>@update-<NEW_STAMP>
agents         8 of 8 nodes     <OLD_STAMP>  <NEW_STAMP>  agent restart on each node
Proceed? [y/N]

Answer y. The run publishes the channel in the cluster, waits for the boot service mirror, snapshots and reprovisions the tritond zone, refreshes the agent on each node, and prints one row per node. Every row reads updated. Output:

== verify
NODE           AGENT        ONLINE  SEEN    RESULT
<CN_HOSTNAME>  <NEW_STAMP>  yes     2s ago  updated
...
control plane build <NEW_STAMP>

A node that reads refresh failed or not on target is listed again after the table, with the last line of the refresh script's error. The script installs the new agent only after it has fetched and checked it, so most failures leave the old agent running, and running the same command again retries that node.

To tell which case you have, on that node, as root:

svcs -xv svc:/site/tritonagent:default

If the service is online, run the same command again. If it is offline or in maintenance, the new agent is installed but did not start. A second run skips that node, because its installed build already matches, so repair the service on the node. The previous agent is kept in /opt/triton/tritonagent.prev.<PID>.

Nodes already on the target are left out of a second run. A node whose agent has gone quiet is skipped as unreachable; add --cn <CN_HOSTNAME> to try it anyway.

If the plan is refused​

The planner refuses a run that would leave an agent with no fleet API version in common with the new control plane:

incompatible: <CN_HOSTNAME>: its agent speaks <VERSIONS> and the new control plane serves <VERSIONS>; it is cut off until its agent is refreshed
tritonadm update: error (Error): the plan leaves a node unable to talk to the control plane; --force runs it anyway

The control plane serves the current fleet API version and the one before it. An agent two versions behind has none in common with it, which happens when a cluster skips an update.

When the planner prints a safe instead: line, follow it. When the refusal also reads the target agent speaks <VERSIONS> and the new control plane serves <VERSIONS>, stop: the new agent cannot talk to the new control plane, and --force does not change that. Otherwise, on the head node, as root, run the same command with --force:

tritonadm update tritond agents --force

The run refreshes each selected agent over SSH from the head node, not through tritond, so an agent that is cut off still receives its refresh. Keep agents in the command: tritonadm update tritond --force refreshes no agents, and a node that is held or unreachable is not refreshed either.

If tritonadm itself is refused​

A tritonadm built against a newer admin API version than the running tritond serves fails every request with exit status 5:

tritonadm update: error (ApiVersionUnsupported): server does not support this API version: <VERSION>

This happens when tritonadm was updated before tritond. Put the previous binary back, then run step 2 again. On the head node, as root:

tritonadm self-update --rollback
tritonadm update tritond agents

On a build of tritonadm without --rollback, run the previous binary by its path instead: /opt/triton/bin/tritonadm.prev update tritond agents.

3. Update the other control-plane zones​

The portal, the boot service, DNS, ClickHouse, FoundationDB, and object storage zones update one component at a time. Preview a component first. On the head node, as root:

tritonadm update portal --dry-run

The preview names the image change. For the portal, it also names the hand-copied UI that the portal stops serving:

zone:triton-portal0
  = zone
  ~ image
      before: <OLD_IMAGE_UUID>
      after:  <NEW_IMAGE_UUID>
...
  portal: will serve the image's UI instead of /data/portal-ui/<DIR> (clears triton:portal_static_dir, reboots the zone)

Apply it. On the head node, as root:

tritonadm update portal

The command asks Apply the updates above? This reprovisions/restarts running components. and then reprovisions the zone. The portal is unavailable until the zone has booted, twice when the UI line appears in the preview. The command ends with == done: <N> updated, 0 error(s) ==.

If the reboot fails after the reprovision, the error names vmadm reboot <ZONE_UUID>. Run that command, then run tritonadm update portal again. The second run finds the setting already cleared, does not reboot the zone, and records the zone's new image so that tritonadm versions reports it.

Use the same two commands for booter, dnsedge, clickhouse, fdb, and mantad. A component that is already on the target reports so and changes nothing. Reprovisioning dnsedge stops authoritative DNS until the zone is back.

4. Update tritonadm​

On the head node, as root:

tritonadm self-update

Output:

installed: <OLD_STAMP>
candidate: <NEW_STAMP> (<BYTES> bytes)
tritonadm updated at /opt/triton/bin/tritonadm
previous binary preserved at /opt/triton/bin/tritonadm.prev; `tritonadm self-update --rollback` puts it back

Before it replaces the binary, self-update asks the running tritond whether it serves the new build's admin API. After step 2 it does. When it does not, the command refuses and changes nothing:

the candidate speaks admin API <VERSION>, which the running tritond does not serve yet. Update tritond first (`tritonadm update tritond`), then self-update; --force swaps it anyway

5. Confirm​

On the head node, as root:

tritonadm versions

Every row for the agents, the control plane, the zones you updated in step 3, and tritonadm reads current. This page does not update these, so their rows can still read drift:

  • notifyd, which tritonadm update has no component for.
  • mantad zones on a cluster with no managed mantad deployment. The host-local walk updates them but does not record their image, so their rows keep the old one.
  • The MantaFS zones, which tritonadm infra mantafs deploy upgrade updates.
  • NAT gateway zones and sandbox images.
  • A portal, boot service, ClickHouse, or DNS edge zone on a node other than this head node. FoundationDB and mantad members on other nodes are updated through tritond.

See also​