Architecture

This page explains which services make up Triton Cloud, where each one runs, what state it owns, which direction the calls flow, and how an API request becomes a change on a compute node.

The shape to hold in your head: one control plane that writes all durable state to FoundationDB, and an agent on every compute node that pulls its jobs from it. The control plane dials an agent for a narrower set of calls: console and SSH tunnel streams, sandbox exec, and sandbox lifecycle operations such as stop, start, reshape, and disk snapshots.

Compute nodes and zones

A Triton Cloud datacenter is built from SmartOS compute nodes (CNs), and each one runs tritonagent in its global zone as SMF service site/tritonagent. The founder is the first compute node, the one tritonadm setup bootstrap turns into a control plane. Later control-plane zones, such as new FoundationDB members, are provisioned onto other compute nodes by their agents.

The control plane runs in zones. Every control-plane zone clones one image, triton-platform, which carries FoundationDB, tritond, triton-admin, identityd, triton-portal, triton-notifyd, mantad, triton-booter, triton-dnsedge, triton-certs, and ClickHouse. An image built with the MantaFS binaries also carries the MantaFS master, storage, and chunk daemons; an image built without them has no MantaFS roles. A zone's first-boot user-script imports and enables only the SMF services of its own role. Each zone has a delegated /data dataset that survives reprovisioning, and all durable cluster state lives in FoundationDB, so an upgrade can replace the zone root without losing data.

Zone role	SMF services	Listeners
FoundationDB	site/triton-fdb	4500 plaintext, 4501 TLS, and both on a member in mixed TLS mode
tritond	site/triton-identityd, site/triton-tritond, site/triton-admin	See the tritond zone
ClickHouse	site/triton-clickhouse	8123 HTTP, 9000 native
Booter	site/triton-booter	DHCP 67, HTTP 80, HTTPS 443, TFTP 69
DNS edge	site/triton-dnsedge	DNS, 8053 health
Notifications	site/triton-notifyd	127.0.0.1:8092 plaintext, 0.0.0.0:8443 TLS
Tenant portal	site/triton-portal	8088 HTTP, and 443 HTTPS by default
Object storage	site/triton-mantad	7443 S3, 127.0.0.1:7101 admin
MantaFS master	site/triton-mantafs-master, site/triton-mantafs-certs:master
MantaFS storage node	site/triton-mantafs-storage, site/triton-mantafs-certs:storage-daemon, plus site/triton-mantafs-chunkd and site/triton-mantafs-certs:chunkd for chunk-path volumes

The triton-certs agent ships in every platform zone. It stays dormant unless the zone's first-boot metadata seeds it. When enabled, it redeems a one-shot token against tritond, installs the zone's certificate bundle under /data/tls, and polls for renewals. MantaFS zones get their daemon certificates from their own service, site/triton-mantafs-certs, which runs one instance per daemon and redeems that daemon's certificate stream against tritond. triton-notifyd is disabled in its manifest, so upgrading the image cannot start mail delivery.

tritond models each control-plane zone as an infra zone. The kinds, as tritonadm prints and accepts them, are fdb, tritond, mantad, booter, clickhouse, dnsedge, portal, notifyd, nat-gateway, mantafs-master, and mantafs-storage. The API wire form spells the multi-word kinds with underscores, such as nat_gateway. A NAT gateway zone is operator-owned like the others, but it realizes a tenant resource, and its record names the owner.

note

Installation spans more than one node. tritonadm setup bootstrap creates the founder FoundationDB and tritond zones and, on a multi-node datacenter, ends with the founder pivot, which leaves triton-fdb1 and triton-tritond1 in place of the founder pair. setup converge then drives the committed intent: ClickHouse, DNS and certificates, object storage, the booter, and the boot policy. tritonadm fdb grow and fdb decommission change FoundationDB membership, and tritonadm storage deploy places a multi-node mantad cluster on declared compute nodes.

What runs on a compute node

Every CN's global zone runs tritonagent. It drives vmadm, ZFS, and the proteus port lifecycle, reports compute node metrics and capacity, and serves two TLS listeners:

• The console listener, port 9101. It serves GET /console/{vm_uuid} for serial and VNC consoles, GET /tunnel/{relay_instance_id} for the agent's half of the SSH broker, and the /sandbox/{sandbox}/... routes. Those cover sandbox exec and the sandbox lifecycle calls: POST /power/stop, POST /power/start, POST /shape, PUT /boot, POST /disk, and POST /disk/snapshots. Each route requires a ticket that tritond minted.
• The live-migration listener, port 4568. The source CN's agent dials the target CN's agent at GET /migrate/{migration_id}. The control plane does not dial it.

The global zone also carries the dataplane and, where they are installed, the storage and sandbox runtimes:

Component	What it is
proteus kernel module and proteusadm	The virtual network dataplane and its low-level CLI. See The dataplane.
site/mantablock-engine	The MantaBlock engine, which serves each MantaBlock volume on a Unix socket in a per-zone directory. Its bundle also carries rshyve, the VMM of MantaBlock instances.
sandbox-runtime bundle	The firehyve VMM plus the guest kernel and initramfs that sandbox microVMs boot. The channel marks it optional, because a node that runs no sandbox needs none.

Besides tenant instances, CNs run the zones that realize tenant-facing resources. A saga places each one, and the agent provisions it:

• Gateway zones boot the triton-gateway image. One image serves three roles, and the zone's user-script enables exactly one: a NAT gateway (triton-natd, with triton-egressd beside it), a load balancer (triton-lbd and triton-adc), or an SSH relay (triton-zoneconnect).
• NAS gateway zones boot the triton-nas-gateway image and serve NFSv4 from MantaFS to one tenant VPC through triton-nasd and NFS-Ganesha.
• Storage zones: mantad members placed by tritonadm storage deploy, and MantaFS storage nodes, whose data lives on the zone's delegated dataset.

The services

Each row is a crate under services/ in the mariana-trench workspace.

Service	Runs in	Owns
tritond	tritond zone, site/triton-tritond	The API, authentication and authorization, the saga engine, the reconcilers, and every durable control-plane record
triton-admin	tritond zone, site/triton-admin	The operator console: an HTTP API under /api, the embedded React single-page app, and operator session state
identityd	tritond zone, site/triton-identityd	The OpenID Connect provider: users, groups, realms, clients, grants, signing keys, and sessions
triton-portal	Its own zone, site/triton-portal	The tenant portal backend, which bridges its React frontend to identityd and tritond
triton-notifyd	Its own zones, site/triton-notifyd	Transactional mail. Replicas run active-active against one FoundationDB keyspace.
triton-booter	Booter zone, site/triton-booter	Network boot on the admin network: DHCP authority, iPXE rendering, platform-image serving, the enrollment channel mirror, and the optional ur enrollment endpoint
triton-dnsedge	DNS edge zone, site/triton-dnsedge	Public authoritative DNS for the datacenter's zones, fed by tritond
triton-certs	Every platform zone, dormant by default	The zone's TLS bundle: one-shot bootstrap redeem, then the renewal poll
tritonagent	Global zone of every CN, site/tritonagent	Everything CN-local
triton-natd	NAT gateway zone	Renders and applies ipf and ipnat rules from typed metadata
triton-egressd	NAT gateway zone	Sandbox egress: a filtering DNS resolver and an SNI- and Host-checked proxy
triton-lbd	Load balancer zone	Installs compiled listener inputs from typed metadata
triton-zoneconnect	SSH relay zone	Opens connections in the zone's network context on behalf of the global zone
triton-nasd	NAS gateway zone	Applies the gateway configuration tritond put in metadata
triton-sandbox-agent	Inside each sandbox microVM	PID 1 and the guest agent

Three more components are part of a running cloud but are not crates here. FoundationDB is the durable store. ClickHouse is the optional time-series backend for metrics and placement history. mantad is the S3 object store, built from the mantas3 crates in the manta-storage repository. The dataplane, proteus, and the VMMs, rshyve and firehyve, live in their own repositories.

The tritond zone

tritond serves several listeners, each bound from the zone's first-boot metadata into /data/etc/tritond/config.toml, rendered from /opt/triton/tritond/etc/config.toml.tmpl:

Listener	Default	Serves
HTTP	0.0.0.0:8080	Every operation, gated by Cedar alone
HTTPS	0.0.0.0:8443	The same, once /data/tls holds a bundle
Cloud surface	0.0.0.0:8444	Tenant operations, the surface that faces the public network
Admin surface	0.0.0.0:8445	Operator operations, on the admin network
Fleet surface	0.0.0.0:8446	Fleet operations, on the admin network
Front door	Off. When on, 0.0.0.0:443	Optional customer listener that routes API names to a Cloud listener and forwards other names to local services
Local socket	/data/run/tritond.sock	A root-owned UNIX socket. The global zone reaches it under the zone root, and tritond authenticates a uid-0 peer as the operator.

Each surface listener serves only its own audience's operations and refuses every credential kind that is not that audience's. If any TCP listener exits, tritond exits, and SMF restarts the whole service. The local socket is best-effort and stays out of that rule.

triton-admin listens on 0.0.0.0:8081 and 0.0.0.0:9443 for TLS, and reaches tritond at http://127.0.0.1:8080. identityd listens on port 8090, bound to 127.0.0.1 for in-zone callers when its issuer URL is HTTPS and to 0.0.0.0 when the issuer is plaintext. It also listens on 0.0.0.0:443 for TLS, unless the front door owns port 443.

The front door is off by default. Operators turn it on or off with the zone metadata key triton:front_door. The tritond start method always exports that key as TRITOND_FRONT_DOOR, which overrides enabled in the [front_door] table of the config file. The identityd start method reads the same key to decide whether identityd binds port 443, so exactly one of the two holds it. The config file value and the environment variable do not reach identityd. The bind address defaults to 0.0.0.0:443, and setting it alone does not turn the front door on. Genesis writes triton:front_door=on when the identityd issuer is a DNS name whose host differs from the console's host, which is the case when the answers declare a dns block with ACME.

The front door picks the certificate and the route by the TLS server name. A connection for an API name or a forwarded name offers HTTP/2 (h2) and http/1.1. A browser can send requests for several of those names on one connection when one certificate covers them all, and each request goes to its own name's backend. A request whose host is not one of those names, or that the connection's certificate does not cover, gets 421. A preview connection speaks HTTP/1.1 only and serves its one host.

How the pieces talk

Solid arrows are required paths. Dotted arrows are optional or configuration dependent.

Five properties of that graph are deliberate.

The agent dials out for work. tritonagent needs one setting to reach the control plane, TRITONAGENT_ENDPOINT, and authenticates with a per-CN API key. It registers, claims jobs, fetches blueprints, reports outcomes, heartbeats, and posts status, capacity, and metrics, all under /internal/v1/agent/*. tritond holds no job-push channel back to the agent. What tritond does push goes to the console listener: console and tunnel streams, sandbox exec, and the sandbox lifecycle calls.

The agent's inbound listeners are TLS and ticket-gated. A console ticket is an HS256 token bound to (server_uuid, vm_uuid, kind). When tritond dials the console listener, it pins the SHA-256 of that listener's TLS SubjectPublicKeyInfo, reported at CN registration, so a hijacked admin IP cannot intercept the stream. The SSH tunnel and sandbox routes ride the same listener for the same reason: it already has the pinned identity and the per-CN ticket key their keys derive from. Migration tickets bind (source_cn, target_cn, vm_uuid, migration_id, role).

The console chain is three hops. Browser to triton-admin, which rewrites the tritond base URL's scheme from http/https to ws/wss and dials tritond, which dials the agent. Frames are copied verbatim at each hop, with no protocol parsing.

triton-booter only reads. It polls /v1/system/hardware/cn-boot/{mac} and /v1/system/hardware/cn-boot-policy, and reports to /internal/v1/cn-boot/{mac}/discovered and /internal/v1/cn-boot/{mac}/booted. Its API key carries the booter scope, which covers those four calls and health, and nothing else. tritond never dials the booter.

Agent transfers bypass tritond. An agent that builds or fetches an image holds no object-store credentials. tritond presigns a PUT or GET for it, and the agent moves the bytes directly to or from mantad. The agent's sha256 check at fetch time is the integrity boundary. The channel-publish saga is different: tritond itself downloads each release artifact, checks its sha256 when the release lists one, and uploads it to the in-cluster channel.

Where state lives

tritond is the only writer of durable control-plane state, and FoundationDB is where that state lives. At startup tritond opens one FoundationDB Database handle and shares it three ways: the Store (FdbStore), the audit chain (FdbChain), and the saga secondary-execution-coordinator store. Saga records sit under a saga/ key prefix that is disjoint from the store's keys, so one handle can read both without the two keyspaces colliding.

The other FoundationDB clients keep to their own first key segments. identityd roots under identity/ and writes its authentication audit chain under audit/identity/, triton-portal roots under portal/, and triton-admin owns admin/, with operator sessions under admin/session/. The triton-admin session store is durable only when ADMIN_FDB_CLUSTER_FILE is set and the binary was built with the foundationdb feature; otherwise sessions are in memory. mantad keeps its object metadata in FoundationDB as well.

The FoundationDB backend is compiled in behind the foundationdb cargo feature, because linking pulls in libfdb_c.so. The workspace pins the client to the fdb-7_3 feature. Without a configured cluster file tritond falls back to an in-memory store, audit chain, and saga engine, which is the posture tests and libfdb_c-less dev builds run in.

On every start, tritond runs an idempotent bootstrap step that mints these if they are absent and loads them if they are present: the JWT signing key, the per-deployment identity HMAC key, the root operator account, the default silo with its default tenant, and the default sandbox packages in every silo.

tritond reads only a small TOML bootstrap file, resolved from --config PATH, then $TRITOND_CONFIG, then /etc/tritond/config.toml. That file carries the listener addresses, the local socket path, the FoundationDB cluster file path, the log filter, and whether this process runs sagas. Everything else is cluster-wide settings stored in FoundationDB. Environment variables override stored settings, which override built-in defaults, and tritond logs a warning for every stored key an environment variable is shadowing.

Two components hold caches rather than state. triton-booter caches last-known-good boot records and channel artifacts on /data, so a compute node still boots, and the booter still serves channel artifacts, through a control-plane outage. A 5xx from tritond takes the cache path, and only a definitive 4xx is authoritative. Enrollment needs a live tritond: the booter reads the setup queue from it and takes each node's one-shot credential bundle from it.

The proteus driver remains the source of truth for realized dataplane state, which is why tritond compiles blueprints instead of tracking what is installed in the kernel.

Degraded modes are explicit rather than fatal. If ClickHouse is unreachable or its schema bootstrap fails, tritond logs a warning and falls back to an in-memory metrics ring buffer rather than refusing to start. If no ClickHouse URL resolves at all, the placement load materializer is never spawned and the history-based placement scorers contribute zero.

How intent reaches hardware

Every job-driven change to a compute node travels this path, and the direction never reverses:

1. A client calls tritond, which authenticates the caller and evaluates a Cedar policy. See Identity.
2. tritond starts a saga. The saga writes the records the operation needs, inside FoundationDB transactions.
3. The saga enqueues a job for an agent and waits for a terminal acknowledgement.
4. The agent on the target CN claims that job on its next poll, at POST /internal/v1/agent/claim, with a default poll interval of 5 seconds.
5. The agent applies the change locally with vmadm, ZFS, and proteus ioctls, then completes the job. The saga's wait ends.

Job kinds the agent executes include Provision, Start, Stop, Restart, Delete, and ApplyPortBlueprint. For network changes the split is strict: tritond owns the compiled per-port blueprint and a per-port generation that every blueprint-affecting change bumps, and the agent fetches and applies it rather than deriving one.

Sandbox lifecycle calls take a shorter path. A saga or handler in tritond calls the agent's console listener directly and reads the outcome from the response.

tritond still contains an in-process stub provisioner that completes jobs with no compute node behind them. It runs only against an in-memory store and only when provisioner.inprocess_disabled is false. A FoundationDB-backed tritond never runs it: it ignores that setting, logs a warning, and expects an external tritonagent.

A tritond process can also run without a saga executor. With saga_executor = false in the bootstrap file, the process submits sagas and another tritond runs them. A zone that serves the admin surface keeps it true.

Control-plane zones are provisioned through the same path. The infra-zone-provision saga writes the record, enqueues a ProvisionInfraZone job for an agent, waits for the terminal acknowledgement, then marks the zone active. FoundationDB members stay on the separate fdb-grow saga, because membership and coordinator changes require topology planning and approval.

The saga catalog

Every multi-resource operation, and every operation that enqueues work for an agent, runs as a saga registered in one catalog, services/tritond/src/sagas/mod.rs. The engine is the steno crate wrapped by tritond-saga, whose executor combines steno's client with a FoundationDB-backed store. Actions carry undos, so a failure unwinds instead of leaking half-created resources, and sagas recover after a tritond restart.

The catalog's sagas fall into these families:

Area	Saga names
Instances	instance-create, instance-delete, instance-start, instance-stop, instance-restart
Migration and MantaBlock moves	migrate-instance, migrate-instance-mantablock, migrate-instance-mantablock-recover, instance-mantablock-move, instance-mantablock-move-finish
Placement	designate
Addresses and subnets	external-ip-allocate, external-ip-release, external-subnet-gateway-mac, floating-ip-allocate, floating-ip-attach, floating-ip-detach, subnet-delete
Gateways	nat-gateway-create, nat-gateway-delete, nat-gateway-ensure, nat-gateway-teardown, load-balancer-create, load-balancer-ensure, load-balancer-update, load-balancer-delete, ssh-relay-ensure, ssh-relay-teardown
Shared filesystems	filesystem-create, filesystem-delete, filesystem-purge, filesystem-resize, filesystem-resize-settle, filesystem-attach, filesystem-detach, nas-gateway-set-ensure, nas-gateway-set-teardown, nas-gateway-remove, nas-gateway-replace
Object and file storage clusters	mantad-apply, mantad-grow, mantad-raise-rf, mantad-replace, mantad-scale-up, mantad-scale-down, mantad-update, mantafs-apply, mantafs-destroy, mantafs-upgrade
Images and channel	image-import, nocloud-build, channel-publish
Sandboxes	sandbox-create, sandbox-delete, sandbox-start, sandbox-stop, sandbox-snapshot, sandbox-pause, sandbox-resume
Fleet and control plane	node-join, infra-zone-provision, infra-zone-teardown, infra-zone-restart, fdb-grow, fdb-decommission, fdb-update
Accounts	self-signup, billing-onboarding

Compute node setup is the one lifecycle that is deliberately not a saga. The durable CnState column is the state machine, and the terminal transitions are performed directly. Compute node lifecycle covers why.

Scopes are the blast-radius control

API keys carry a scope, and the scopes encode the architecture's trust boundaries:

Scope	Reach
full	Equivalent to authenticating as the owning user
read_only	List and get on every resource, plus audit chain reads
tenant_write	read_only plus the writes a tenant performs on its own resources. The tier an ordinary customer session carries.
audit_only	Audit chain reads only
agent	The per-CN agent's key, used against tritond's /internal/v1/agent/* surface
booter	Boot-config reads plus the discovered and booted reports, and nothing else
ur_enroll	The ur enrollment endpoints, deliberately unable to approve a node for setup or open the enrollment window
dnsedge	The DNS zone list and record feed reads, and nothing else
certs	Certificate-bundle reads for the renewal poll, plus health

A scope is a ceiling on the credential, not the tenancy boundary. Cedar still decides which tenant a principal may touch.

Several scopes exist to make a leaked credential survivable. A stolen booter key cannot drive a lifecycle transition. A stolen ur_enroll key cannot authorize the disk wipe that compute-node setup performs, because that approval requires an operator action no scoped key can hold. A stolen dnsedge key, taken from the Internet-facing daemon, yields a read-only view shaped like DNS. certs is kept out of read_only because bundle reads carry private keys.

The dataplane

proteus is Triton Cloud's virtual network dataplane: an illumos kernel module providing per-port flow tables, a distributed firewall, NAT, a Geneve overlay, DHCP synthesis, an instance metadata service, and a cross-CN peer resolver. It is built on Oxide Computer's OPTE engine while providing its own driver interface.

It lives in its own repository, which builds the kernel module under kmod/proteus plus the crates proteus-core, proteus-api (wire types), proteus-ioctl, proteus-test-utils, proteusadm (a low-level operator CLI), and illumos-sys-hdrs, and the triton-vpc plugin. The repository's install manifest places the driver at /usr/kernel/drv/amd64/proteus and proteusadm at /usr/sbin/proteusadm. Its site/proteus-kmod SMF manifest is a development aid that loads a test build from /var/tmp/proteus_drv in place of the module the platform image ships. This workspace consumes proteus-api, proteus-ioctl, and triton-vpc as git dependencies pinned to one proteus revision, and also distributes proteusadm as a CLI-only agent bundle.

The division of labor matches the rest of the system. tritond compiles stored network intent into a per-port PortBlueprint. The agent applies that blueprint through ioctls on /dev/proteus, and the driver stays authoritative for what is actually realized.

The instance metadata service follows the same split. The kernel module punts a guest's metadata frames to tritonagent with the originating port stamped on each one, and the agent answers with the identity bound to that port, never one the guest supplies. The agent starts the service whenever its CN credential carries an IMDS token key, which tritond mints with every CN credential.

The VMMs

Two VMMs from the rust-bhyve repository take the place of the platform bhyve inside bhyve-branded zones for specific workloads:

• rshyve runs instances with MantaBlock disks. The zone's /bhyve.zone file names it, and the brand's init runs it with the brand's own arguments. It also implements the live-migration protocol for those instances: RAM pre-copy, a pause that drains the ring disks, a cutover gate that the agents answer with tritond's go, and the device state. The disks do not move; the registry hands each volume's coordinator to the target.
• firehyve is a microVM VMM over the same crates, with direct kernel boot and no UEFI, migration, or passthrough. It boots sandboxes, from the sandbox-runtime bundle.

The rust-bhyve README marks the project experimental and not production supported. The repository's open security findings record that live migration has no peer authentication or encryption of its own. Migration exports each vCPU's FPU state and restores the XSAVE area on the target, and a second open finding tracks that path until a migration between two compute nodes under a continuous AVX workload validates it. The README still lists both as gaps. In Triton Cloud, rshyve's stream runs over a Unix socket on each CN, and the agents carry it between CNs over the ticketed TLS WebSocket of the live-migration listener.

Object storage

mantad is the S3 service from the manta-storage repository's mantas3 crates, built into the triton-platform image. tritond uses it for image blobs, presigned uploads, the in-cluster release channel that the booter mirrors, and tenant buckets. Its /v1/storage/clusters/{id}/... routes forward typed admin calls to a registered mantad. mantad listens on 7443 for S3. Every mantad zone that tritond provisions, including a single-member cluster, binds its admin and peer listener on 0.0.0.0:7101. That listener is plain HTTP. Keep it off untrusted networks.

A production multi-node cluster comes from tritonadm storage deploy, which places one mantad zone on each declared compute node. It requires at least three nodes, an odd replication factor of at least 3, and distinct compute nodes. Object metadata lives in the Triton FoundationDB service, so the members share one metadata plane and replicate object data among themselves. An answers document with datacenter.manta.enabled takes the other path: tritonadm setup converge creates a one-member cluster at replication factor 1 and grows it, one member per pass, to 3 or 5 members at a matching replication factor, while a manta-grow standing approval is open. The tritonadm setup flags do not create mantad, and they refuse a production request for more than one mantad node.

To place a cluster, see Deploy object storage.

Identity

tritond issues and verifies its own operator credentials: HS256 JWTs signed with the cluster key, and tcadm_-prefixed API keys checked against bcrypt hashes in the store. Authorization is a Cedar policy evaluation either way.

identityd is the cluster's OpenID Connect provider. Setting TRITOND_IDENTITYD_ISSUER_URL, or the identityd_issuer_url cluster setting, turns on a relying-party verifier in tritond. It verifies each access token's RS256 signature against the issuing realm's published JWKS and binds every trusted issuer to the realm it may speak for, so one silo's realm cannot mint a token for another silo. Absent an issuer and any stored silo bindings, the verify path is skipped. tritond tries an identityd token before its own JWT path, and API keys bypass both.

Operators sign in to the console through identityd's system realm when triton-admin is configured for SSO. Tenants sign in with tritonctl login --sso, which uses the device flow against their silo's realm.

See also

• Compute node lifecycle explains the two state machines a compute node moves through and why the destructive transitions are operator-gated.
• Sandbox nodes covers what a compute node needs to run sandboxes.
• Installation overview puts genesis, convergence, enrollment, and growth in order.
• Genesis on the founder covers the bring-up sequence that creates the control-plane zones described here.
• Enroll a compute node takes a compute node from network boot to approved.
• tritonadm command index is the complete operator CLI reference.