Skip to main content

Upgrade sandbox components

This guide upgrades the pieces that sandboxes run on, in the order that keeps live sandboxes safe. It ends with the notes for the release that added package traits.

The release channel carries every sandbox piece: the runtime bundle (the virtual machine monitor, the guest kernel, and the guest agent), the four built-in images, and the NAT gateway image. tritonadm update copies a sealed revision into the cluster's channel and rolls it out.

Before you start​

On the head node, in the global zone, as root, drain the warm pool, so no pooled zone holds old pieces:

tritonadm config set sandbox.pool_depth 0

Wait until the pooled zones are gone from each sandbox CN. In each CN's global zone, as root, count the running VMs before and after each step. The counts must match:

vmadm list -H state=running | wc -l

Check the front door keys​

Before you update tritond to a build with the triton:front_door switch, confirm that each tritond zone's triton:front_door and triton:identityd_tls_bind_address are consistent. Read them with the check in Check a zone's front door keys. tritonadm update reprovisions the zone and keeps its zone metadata.

If a zone shows triton:identityd_tls_bind_address=none and has no triton:front_door key, do not update it yet. Under the new identityd start method that combination stops identityd.

Step 1: Upgrade tritond​

On the head node, in the global zone, as root, plan the control-plane update:

tritonadm update tritond --check
danger

tritonadm update tritond snapshots the control-plane zone and reprovisions it. The API is down for about a minute while the new zone starts. Run it in a maintenance window. If the control plane does not come back, the snapshot holds its previous data.

Apply the update, and answer the Proceed? prompt:

tritonadm update tritond

The run snapshots the control-plane zone, reprovisions it from the channel's triton-platform image, and waits until the control plane answers. It prints the snapshot name and control plane answering, build <BUILD>. See Updates and channels.

Check that https://<API_NAME>/v1/health answers, and that tritonctl whoami shows the new server build.

Step 2: Upgrade tritonagent and the sandbox runtime bundle​

tritonadm update agents refreshes the agent on every CN, then installs the sandbox runtime bundle on each CN that runs sandboxes or already has a bundle. Each CN runs tritonagent-refresh and sandbox-runtime-refresh against the boot service's channel mirror, so you do not run the refresh scripts by hand. --hold <CN> leaves a CN's agent and bundle as they are.

On the head node, in the global zone, as root, plan the update, then apply it and answer the Proceed? prompt:

tritonadm update agents --check
tritonadm update agents

The agent refresh keeps etc/agent.env and the credentials. On each CN, the log is /var/log/tritonagent/agent.out.

A sandbox zone mounts the bundle, so the bundle refresh refuses on a CN with a running sandbox. Drain the pool and delete live sandboxes on that CN, then run the update again. A CN that is already current reports success while its sandboxes run.

Check that:

  • The running VM count on each CN is unchanged.
  • tritonadm cn list -j shows the sandbox features in last_status.agent_features.
  • tritonadm versions shows sandbox runtime as current on each sandbox CN. The agent checks the bundle again every 5 minutes.

Step 3: Check the images​

The intent reconciler registers each built-in image and moves its pin, unless you hold it. On the head node, in the global zone, as root, check the sandbox image rows:

tritonadm versions

Step 4: Fill the pool again​

On the head node, in the global zone, as root:

tritonadm config set sandbox.pool_depth 2

Check that a pooled create of the default package answers in about 0.4 seconds, and that tritonadm op list shows no stuck sandbox operations.

Roll back​

tritonadm channel revisions lists the sealed revisions. Drain the pool, then put the cluster's channel, the runtime bundles, and the image pins back.

danger

A run with no component also covers the control plane. When the revision's control-plane image differs from the running one, the run snapshots and reprovisions the control-plane zone, and the API is down for about a minute.

On the head node, in the global zone, as root:

tritonadm update --revision <REVISION>

An existing NAT gateway keeps its image until you make it again. To stop a pilot at once, turn every gate to off. Live sandboxes keep running until their deadline or until you delete them.

Upgrading to package traits​

The release that added sandbox packages also enforces CN traits. Read these notes before you upgrade.

danger

Upgrade every tritond before any create runs on the new one. An older tritond cannot read a bool or list CN trait, and a row it rewrites loses an instance's traits or a CN's labels. Do not roll back to an older tritond after a CN has a bool or list trait.

  • At start, tritond moves image-build, cluster, rack, and each silo's NoCloud build tag from CN traits to labels, once per CN. It logs moved descriptive CN traits to labels. Other trait keys stay traits.
  • At start, tritond gives each instance made before traits were enforced the traits a create gives now. A packaged instance takes its package version's traits. A sandbox made before packages takes the default sandbox package's traits. It logs instance takes the traits a create gives it for each one. A VM made before packages keeps none and places by placement.default_vm_traits.
  • A move that fails is retried with backoff until it is done. The gauge triton.upgrade_backfill_pending, with series cn_labels and instance_traits, stays at 1 until then.
  • After the upgrade, every seeded sandbox package carries sandbox=true, and a CN with no traits takes no seeded sandbox. Set up a layout before tenants create sandboxes. See Place sandboxes with CN traits and labels.
  • A sandbox create that gives sizes as numbers is now refused. Tell tenants to use --package.

See also​