Troubleshooting an install
Each section names a symptom you can see, then the check that confirms it, then the fix. Start with the first section. It applies to every other one.
Read the saga, not the summary
tritonadm operation list is a summary. It shows every finished saga as
finished, whether it succeeded, failed, or unwound. With --json, the same
sagas carry done. When anything looks wrong, read the real state. On the founder, in the global zone, as root:
tritonadm operation show <OPERATION> --json
Read state: pending, running, unwinding, succeeded, failed,
unwound, or stuck. Then walk steps to the first step that is not
succeeded. unwound means the saga failed and rolled back, whatever the
list showed. stuck means an undo failed and needs an operator.
The founder pivot fails
Symptom: genesis ends with
D6 founder pivot failed — genesis is NOT complete.
The lines above the error name the pivot step that stopped. Find the operation it was waiting on, and read it. On the founder, as root:
tritonadm operation list
tritonadm operation show <OPERATION> --json
tritonadm fdb status --break-glass --zone triton-fdb1
Correct the cause, then run setup bootstrap again with the same
arguments. When both replacements and both founder zones still exist,
bootstrap verifies the replacements and resumes the retirements. When only
some of them exist, bootstrap refuses with
refusing to re-run genesis: pivot descendant zone(s) ... exist. Inspect
the zones and the operation records before you change anything.
converge cannot resolve the release channel
Symptom:
resolve the release channel for converge; a cluster installed from
--image-dir must pass --image-dir again, or declare an upstream with
`tritonadm config set channel.upstream_url <url>`
Genesis from the channel records channel.upstream_url, and converge
reads it when you pass no --channel. The error has two causes. A genesis
from --image-dir records no channel; pass --image-dir again. A genesis
that printed
WARNING: could not record channel.upstream_url (...); run `tritonadm config set channel.upstream_url <URL>` before converge
did not record it; set it once, on the founder, as root:
tritonadm config set channel.upstream_url https://tritoncloud.nyc3.digitaloceanspaces.com/channels/edge.json
Compute nodes never get a DHCP lease
Symptom: the BMC reports the NIC link up and the machine tries to network
boot. The booter log shows no DHCP DISCOVER for its MAC, and
tritonadm cn boot declared never shows it as booted.
The booter never sends a DHCP NAK, so every negative outcome is silence.
Rule out admission first. A machine that matches no entry in the boot policy
allowlist is discovered, and the booter serves it nothing. With
auto-enrollment on and an empty allowlist, every machine is admitted, as
described in
Enroll compute nodes and converge.
Check the boot record. On the founder, in the global zone, as root:
tritonadm cn boot show <MAC>
If the record is enrolling and no lease arrives, the broadcast does not
reach the booter. A PXE DISCOVER is an untagged layer-2 broadcast on the
admin VLAN. When the founder's admin network uses an LACP bond, some switch
fabrics deliver native-VLAN broadcast to one bond member only, or to
neither. Tagged VLANs arrive on both. Confirm with a capture. On the founder, in the global zone, as root:
snoop -d aggr0 "ether broadcast"
Zero admin-VLAN broadcast while the machine retries is the signature. The immediate workaround is to reduce the bond to a single member on the founder. This removes admin-link redundancy until the switch is corrected. Read the member ports and their link state first:
dladm show-aggr -x aggr0
<SECOND_MEMBER> is the member port to remove. On the founder, in the global zone, as root:
If you remove the member that still carries admin traffic, the founder loses its admin link, and a session over that link drops with it. Run the command from the console or the BMC.
dladm remove-aggr -l <SECOND_MEMBER> aggr0
The durable fix is on the switch. Make native-VLAN handling identical on every member port of the founder's LAG, and check storm control. The founder never network-boots, but it hosts the booter. Every other machine's enrollment depends on the native-VLAN broadcast receive path of the founder's bond.
A machine stops in POST after a Redfish boot override
Symptom: after the boot-override request and the reset, the BMC event log
shows System CPU Resetting but no NIC link-up. The machine never reaches
the network boot loader.
On iDRAC9 a request that includes BootSourceOverrideMode stages a BIOS
job through the Lifecycle Controller. If that controller is busy, the job
stays in Scheduling and never completes. Clear the job queue. Then reset
again with the two-field override shown in
Enroll compute nodes and converge:
POST https://<BMC>/redfish/v1/Managers/iDRAC.Embedded.1/Oem/Dell/DellJobService/Actions/DellJobService.DeleteJobQueue
{"JobID":"JID_CLEARALL_FORCE"}
A node stays in setting_up and never completes
Symptom: tritonadm cn show <CN> --json shows state of setting_up and
a last_seen older than approved_at.
The agent session from the enrollment platform image to the booter can drop
without an error, and it never reconnects. Reset the machine so that it
network-boots again. The installer recognizes a pool that it already built
by the zones pool stamp, and it does not erase that pool a second time.
When a node reports neither outcome before its 45-minute deadline, the
reconciler returns it to awaiting_setup and records the reason in
last_setup_error. Correct the cause and approve setup again.
The booter never serves
Symptom: booter-serving blocks, and the booter log in the booter zone
repeats:
channel mirror refresh failed ... fetch https://s3.<DC_DOMAIN>:7443/triton-images/channel/install.sh
Check whether the cluster's own DNS serves the s3 name. On the founder, in the global zone, as root:
nslookup s3.<DC_DOMAIN> <DNSEDGE_ADMIN_IP>
NXDOMAIN while tritond resolves means the DNS zone serves a stale
snapshot. Compare what the store holds with what DNS answers. tritonadm
has no command that reads the DNS record feed, so read it from the HTTPS
endpoint. On a host that resolves the cluster's names and trusts its
certificates:
curl -H "Authorization: Bearer <API_KEY>" \
https://tritond.<DC_DOMAIN>:8443/v1/dns/zones/<DC_DOMAIN>/records
A store generation far ahead of what DNS answers means the DNS feed fails. The DNS zone log shows this:
zone list refresh failed url=https://triton-tritond1.<DC_DOMAIN>:8443
The DNS zone pins that URL to an address but verifies the certificate under
the name. The cause is trust, not resolution. With an ACME staging
directory, the chain needs the data center trust anchors at
/data/tls/anchors.pem in the DNS zone. The durable fix is a public
certificate authority. Its chain ends at a root that the compiled-in
Mozilla set already contains.
Certificates never issue
The cause is almost always the delegation. Check it against the parent's
authoritative servers as shown in
Requirements. Do not use a recursive
resolver, because it can serve a cached SERVFAIL. When the records are
correct, the certificate reconciler continues without another install.
The install log has a "no native root certificates" warning
On a platform image with no CA bundle, each tritonadm process prints a
six-line no native root certificates found; using bundled Mozilla roots
advisory once. The advisory is harmless. Filter it:
grep -vE 'SSL_CERT|certificate store|trust additional|no native root' bootstrap.log
Known gaps
Current as of the prod-01 bring-up.
| Gap | Effect |
|---|---|
| No install media | You prepare node zero by hand. |
No stable channel published | The documented one-line install fails without TRITON_CHANNEL=edge. |
| No CA bundle in the platform image | The curl bootstrap cannot do TLS. |
operation list state | Every finished saga shows as finished, whatever its outcome. |
answers.secrets | Declared, validated, displayed, never used. |
| Serial mismatch | A wrong declared serial is not reported when the UUID matched. |
| Both nameservers in one zone | ns0 and ns1 share one DNS zone. There is no real redundancy. |
tritonadm builds | Published for illumos only. There is no workstation build. |