Architecture

Triton Cloud is four Rust services, one FoundationDB cluster, and one illumos kernel module. This page explains what each piece owns, which direction the calls flow, and how an API request becomes a change on a compute node.

The shape to hold in your head: one control plane that writes all durable state, and agents that pull work from it. Nothing in the control plane pushes to a compute node except the console stream.

The head node and compute node split

A head node runs the control plane. Its services live in SmartOS zones on that host: tritond and triton-admin share one zone, FoundationDB runs in its own zone, and triton-booter runs in a third. Optional storage and metrics zones sit alongside them.

A compute node (CN) runs tenant instances. Its only Triton Cloud software is tritonagent, running in the global zone as SMF service site/tritonagent, plus the proteus kernel module that carries the instances' network traffic.

The head node holds no per-instance runtime state that it did not compute itself. The agent is the only component that mutates CN-local runtime state.

note

tritonadm setup apply builds a converged single-node head node: the control plane and one CN role on the same machine. The fdb-grow and fdb-decommission sagas exist to scale the control-plane FoundationDB cluster beyond that, but the installer's own scope is a single node.

The services

Each row is a crate in the mariana-trench workspace.

Service	Crate	Runs in	Owns
tritond	services/tritond	Head node control-plane zone, SMF site/triton-tritond	The API, authentication and authorization, the saga engine, and every durable control-plane record
triton-admin	services/triton-admin/backend	Same zone as tritond, SMF site/triton-admin	The operator console: an HTTP API under /api, the embedded React single-page app, and operator session state
triton-booter	services/triton-booter	Its own head node zone, SMF site/triton-booter	Network boot on the admin network: DHCP authority, iPXE script rendering, platform-image serving, and the optional ur enrollment endpoint
tritonagent	services/tritonagent	Global zone of every compute node, SMF site/tritonagent	Everything CN-local: vmadm, ZFS, proteus ports, host metrics, the console listener, and the live-migration listener

Three more components are part of a running cloud but are not crates here. FoundationDB is the durable store, running in its own zone (site/triton-fdb). ClickHouse is the optional time-series backend for metrics and placement history. mantad is the optional S3-compatible object store that tritond calls for image builds and storage-cluster operations.

proteus is the dataplane, described below. It lives in a sibling repository.

How the pieces talk

Solid arrows are required paths. Dotted arrows are optional or configuration dependent.

Four properties of that graph are deliberate.

The agent dials out, never in. tritonagent needs one setting to work, TRITONAGENT_ENDPOINT, and authenticates with a per-CN API key. It claims jobs, fetches blueprints, reports outcomes, heartbeats, and posts status, capacity, and metrics. tritond holds no job-push channel back to the agent.

The two inbound agent listeners are TLS and ticket-gated. The console listener on port 9101 accepts a WebSocket at GET /console/{vm_uuid} only with an HS256 ticket bound to (server_uuid, vm_uuid, kind), and tritond pins the SHA-256 of that listener's TLS SubjectPublicKeyInfo, exchanged at registration, so a hijacked admin IP cannot intercept the stream. The live-migration listener on port 4568 is dialed by the source CN's agent, not by the control plane.

triton-booter only reads. It polls /v2/cn-boot/{mac} and /v2/cn-boot-policy, and reports discovered and booted. Its API key carries the Booter scope, which covers boot-config reads plus those two reports and nothing else. tritond never dials the booter.

The console chain is three hops. Browser to triton-admin, which rewrites the tritond base URL's scheme from http/https to ws/wss and dials tritond, which dials the agent. Frames are copied verbatim at each hop, with no protocol parsing.

Where state lives

tritond is the only writer of durable control-plane state, and FoundationDB is where that state lives. At startup tritond opens one FoundationDB Database handle and shares it three ways: the Store (FdbStore), the audit chain (FdbChain), and the saga secondary-execution-coordinator store. Saga records sit under a saga/ key prefix that is disjoint from the store's keys, so one handle can read both without the two keyspaces colliding.

triton-admin keeps its own state and nothing else. Operator sessions live under the admin/session/ prefix, and triton-admin owns the admin/ first key segment exclusively. That store is durable only when ADMIN_FDB_CLUSTER_FILE is set and the binary was built with the foundationdb feature; otherwise sessions are in memory.

The FoundationDB backend is compiled in behind the foundationdb cargo feature, because linking pulls in libfdb_c.so. The workspace pins the client to the fdb-7_3 feature. Without a configured cluster file tritond falls back to an in-memory store, audit chain, and saga engine, which is the posture tests and libfdb_c-less dev builds run in.

On every start, tritond runs a bootstrap step that mints three things if they are absent and loads them if they are not: the JWT signing key, the per-deployment identity HMAC key, and the root operator account. It is idempotent.

tritond reads only a small TOML bootstrap file, resolved from --config PATH, then $TRITOND_CONFIG, then /etc/tritond/config.toml. That file carries the bind address, the FoundationDB cluster file path, and the log filter. Everything else is cluster-wide settings stored in FoundationDB. Environment variables override stored settings, which override built-in defaults, and tritond logs a warning for every stored key an environment variable is shadowing.

Two components hold caches rather than state. triton-booter caches last-known-good boot records on /data so a compute node still boots through a control-plane outage; a 5xx from tritond takes the cache path, and only a definitive 4xx is authoritative. The proteus driver remains the source of truth for realized dataplane state, which is why tritond compiles blueprints instead of tracking what is installed in the kernel.

Degraded modes are explicit rather than fatal. If ClickHouse is unreachable or its schema bootstrap fails, tritond logs a warning and falls back to an in-memory metrics ring buffer rather than refusing to start. If no ClickHouse URL resolves at all, the placement load materializer is never spawned and the history-based placement scorers contribute zero.

How intent reaches hardware

Every change to a compute node travels the same path, and the direction never reverses:

1. A client calls tritond. The Authorization: Bearer header is authenticated: tokens beginning with tcadm_ are looked up against bcrypt-hashed API key records, and other tokens are validated as HS256 JWTs. Authorization is a Cedar policy evaluation.
2. tritond starts a saga. The saga writes the records the operation needs, inside FoundationDB transactions.
3. The saga enqueues a provisioning job and waits for a terminal acknowledgement.
4. The agent on the target CN claims that job on its next poll, at POST /v1/agent/claim, with a default poll interval of 5 seconds.
5. The agent applies the change locally with vmadm, ZFS, and proteus ioctls, then completes the job. The saga's wait ends.

Job kinds the agent executes include Provision, Start, Stop, Restart, Delete, and ApplyPortBlueprint. For network changes the split is strict: tritond owns the compiled per-port blueprint and its monotonic generation, and the agent fetches and applies it rather than deriving one.

tritond still contains an in-process stub provisioner from an earlier phase that consumes the job queue itself. Production disables it with the provisioner.inprocess_disabled setting, at which point tritond logs that it expects an external tritonagent.

Control-plane zones are provisioned through the same spine. The infra-zone-provision saga writes the record, enqueues a ProvisionInfraZone job for an agent, waits for the terminal acknowledgement, then marks the zone active. The infra-zone kinds tritond models are fdb, tritond, mantad, booter, and clickhouse. FoundationDB members stay on the separate fdb-grow saga, because membership and coordinator changes require topology planning and approval.

The saga catalog

Every multi-resource operation, and every operation that enqueues work for an agent, runs as a saga registered in one catalog. The engine is the steno crate wrapped by tritond-saga, whose executor combines steno's client with a FoundationDB-backed store. Actions carry undos, so a failure unwinds instead of leaking half-created resources, and sagas recover after a tritond restart.

Eighteen sagas are registered:

Area	Saga names
Instances	instance-create, instance-delete, instance-start, instance-stop, instance-restart, migrate-instance
Placement	designate
Networking	floating-ip-allocate, floating-ip-attach, floating-ip-detach, nat-gateway-create, nat-gateway-delete
Images	image-import, nocloud-build
Fleet and control plane	node-join, infra-zone-provision, fdb-grow, fdb-decommission

The catalog is the design surface as well as the code: a module that does not exist is a gap, and an action without an undo is a leak.

Compute node setup is the one lifecycle that is deliberately not a saga. The durable CnState column is the state machine, and the terminal transitions are performed directly. Compute node lifecycle covers why.

Scopes are the blast-radius control

API keys carry a scope, and the scopes encode the architecture's trust boundaries:

Scope	Reach
full	Equivalent to authenticating as the owning user
read_only	List and get on every resource, plus audit chain reads
audit_only	Audit chain reads only
agent	The per-CN agent's key, used against tritond's /v1/agent/* surface
booter	Boot-config reads plus the discovered and booted reports, and nothing else
ur_enroll	The ur enrollment endpoints, deliberately unable to approve a node for setup or open the enrollment window

Two of those exist to make a leaked credential survivable. A stolen booter key cannot drive a lifecycle transition. A stolen ur_enroll key cannot authorize the disk wipe that compute-node setup performs, because that approval requires an operator action no scoped key can hold.

The dataplane

proteus is Triton Cloud's virtual network dataplane: an illumos kernel module providing per-port flow tables, a distributed firewall, NAT, a Geneve overlay, DHCP synthesis, an instance metadata service, and a cross-CN peer resolver. It is built on Oxide Computer's OPTE engine while providing its own driver interface.

It lives in its own repository, which ships the kernel module under kmod/proteus plus the crates proteus-api (wire types), proteus-host, proteus-ioctl, proteus-test-utils, proteusadm (a low-level operator CLI), and the triton-vpc plugin. This workspace consumes proteus-api, proteus-ioctl, and triton-vpc as git dependencies pinned to main.

The division of labor matches the rest of the system. tritond compiles stored network intent into a per-port PortBlueprint. The agent applies that blueprint through ioctls on /dev/proteus, and the driver stays authoritative for what is actually realized.

Identity

tritond issues and verifies its own operator credentials: HS256 JWTs signed with the cluster key, and tcadm_-prefixed API keys. That is sufficient on its own.

It can also accept access tokens from an external identity provider. Setting TRITOND_IDENTITYD_ISSUER_URL, or the identityd_issuer_url cluster setting, turns on a relying-party verifier. Absent both, the verify path is skipped and authentication behaves exactly as described above. That provider is not part of this workspace.

See also

• Compute node lifecycle explains the two state machines a compute node moves through and why the destructive transitions are operator-gated.
• Installation overview puts the install stages in order.
• Head node setup covers the bring-up sequence that creates the control-plane zones described here.
• Enroll a compute node takes a machine from network boot to an approved compute node.
• tritonadm command index is the complete operator CLI reference.