Troubleshooting an install
Each section names a symptom you can see, then the check that confirms it, then the fix. Start with the first section. It applies to every other one.
Read the saga, not the summary
tritonadm operation list is a summary. It shows an unwound saga as
done. When anything looks wrong, read the real state:
tritonadm operation show <OPERATION> --json
Read state (running, done, failed, unwound). Then walk steps to
the first step that is not succeeded. unwound means the saga failed and
rolled back, whatever the list showed.
The founder pivot fails at the last step
Symptom: genesis ends with
D6 founder pivot failed — genesis is NOT complete.
Before you assume the cluster is broken, check whether the pivot did its work:
tritonadm fdb status
tritonadm operation show <FDB_DECOMMISSION_OPERATION> --json
A known defect makes verify_converged, the last of ten steps, fail on a
cluster that converged correctly. The signature is this. Steps 0 through 8
are succeeded. fdb status reports available, quorum ok, data healthy,
and fully_recovered. Step 9 is failed after approximately ten minutes.
The unwind is safe, because it does not exclude a destroyed member a second
time. Genesis still does not complete. Run a control plane that has the
corrected predicate, then run setup bootstrap again.
converge cannot resolve the release channel
Symptom:
resolve the release channel for converge; a cluster installed from
--image-dir must pass --image-dir again, or declare an upstream with
`tritonadm config set channel.upstream_url <url>`
Genesis does not record the channel it installed from. The error occurs
even when the install used --channel, and the message names the one cause
that does not apply. Set the upstream once:
tritonadm config set channel.upstream_url https://tritoncloud.nyc3.digitaloceanspaces.com/channels/edge.json
Compute nodes never get a DHCP lease
Symptom: the BMC reports the NIC link up and the machine tries to network
boot. The booter log shows no DHCP DISCOVER for its MAC, and
tritonadm cn boot declared never shows it as booted.
The booter never sends a DHCP NAK, so every negative outcome is silence.
Rule out admission first. A machine whose identity matches no declared node
is discovered, and the booter serves it nothing. Check the boot record:
tritonadm cn boot show <MAC>
If the record is enrolling and no lease arrives, the broadcast does not
reach the booter. A PXE DISCOVER is an untagged layer-2 broadcast on the
admin VLAN. When the founder's admin network uses an LACP bond, some switch
fabrics deliver native-VLAN broadcast to one bond member only, or to
neither. Tagged VLANs arrive on both. Confirm with a capture on the founder:
snoop -d aggr0 "ether broadcast"
Zero admin-VLAN broadcast while the machine retries is the signature. The immediate workaround is to reduce the bond to a single member on the founder. This removes admin-link redundancy until the switch is corrected:
dladm remove-aggr -l <SECOND_MEMBER> aggr0
The durable fix is on the switch. Make native-VLAN handling identical on every member port of the founder's LAG, and check storm control. The founder never network-boots, but it hosts the booter. Every other machine's enrollment depends on the native-VLAN broadcast receive path of the founder's bond.
A machine stops in POST after a Redfish boot override
Symptom: after the boot-override request and the reset, the BMC event log
shows System CPU Resetting but no NIC link-up. The machine never reaches
the network boot loader.
On iDRAC9 a request that includes BootSourceOverrideMode stages a BIOS
job through the Lifecycle Controller. If that controller is busy, the job
stays in Scheduling and never completes. Clear the job queue. Then reset
again with the two-field override shown in
Enroll compute nodes and converge:
POST https://<BMC>/redfish/v1/Managers/iDRAC.Embedded.1/Oem/Dell/DellJobService/Actions/DellJobService.DeleteJobQueue
{"JobID":"JID_CLEARALL_FORCE"}
A node stays in setting_up and never completes
Symptom: tritonadm cn show <CN> --json shows state of setting_up and
a last_seen older than approved_at.
The agent session from the enrollment platform image to the booter can drop
without an error, and it never reconnects. Reset the machine so that it
network-boots again. The installer recognizes a pool that it already built
by the zones pool stamp, and it does not erase that pool a second time.
When a node reports neither outcome before its 45-minute deadline, the
reconciler returns it to awaiting_setup and records the reason in
last_setup_error. Correct the cause and approve setup again.
The booter never serves
Symptom: booter-serving blocks, and the booter log in the booter zone
repeats:
channel mirror refresh failed ... fetch https://s3.<DC_DOMAIN>:7443/triton-images/channel/install.sh
Check whether the cluster's own DNS serves the s3 name. On the founder:
nslookup s3.<DC_DOMAIN> <DNSEDGE_ADMIN_IP>
NXDOMAIN while tritond resolves means the DNS zone serves a stale
snapshot. Compare what the store holds with what DNS answers:
curl -H "Authorization: Bearer <API_KEY>" \
http://<TRITOND_IP>:8080/v1/dns/zones/<DC_DOMAIN>/records
A store generation far ahead of what DNS answers means the DNS feed fails. The DNS zone log shows this:
zone list refresh failed url=https://triton-tritond1.<DC_DOMAIN>:8443
The DNS zone pins that URL to an address but verifies the certificate under
the name. The cause is trust, not resolution. With an ACME staging
directory, the chain needs the data center trust anchors at
/data/tls/anchors.pem in the DNS zone. The durable fix is a public
certificate authority. Its chain ends at a root that the compiled-in
Mozilla set already contains.
Certificates never issue
The cause is almost always the delegation. Check it against the parent's
authoritative servers as shown in
Requirements. Do not use a recursive
resolver, because it can serve a cached SERVFAIL. When the records are
correct, the certificate reconciler continues without another install.
The install log is unreadable
The TLS client prints a six-line "no native root certificates" advisory each time it constructs a client. This happens many times in one install. The advisory is harmless. Filter it:
grep -vE 'SSL_CERT|certificate store|trust additional|no native root' bootstrap.log
Known gaps
Current as of the prod-01 bring-up.
| Gap | Effect |
|---|---|
| No install media | You prepare node zero by hand. |
| Channel not recorded at genesis | converge cannot resolve the channel after a --channel genesis. |
No stable channel published | The documented one-line install fails without TRITON_CHANNEL=edge. |
| No CA bundle in the platform image | The curl bootstrap cannot do TLS. |
verify_converged defect | Every founder pivot fails at its last step. |
operation list state | An unwound saga shows as done. |
answers.secrets | Declared, validated, displayed, never used. |
| Serial mismatch | A wrong declared serial is not reported when the UUID matched. |
| Both nameservers in one zone | ns0 and ns1 share one DNS zone. There is no real redundancy. |
tritonadm builds | Published for illumos only. There is no workstation build. |