Skip to main content

How instance metadata works

How an instance learns its own identity, addresses, SSH keys, and user-data from inside the guest, and why the answer cannot be another instance's.

The instance metadata service (IMDS) answers HTTP on 169.254.169.254, TCP port 80, and on fd00:ec2::254 for an instance with an IPv6 address. Its paths follow the shape of the EC2 metadata service: /latest/meta-data/... and /latest/user-data. A second tree, /triton/..., carries Triton-specific facts and the metadata you set yourself.

No server sits at the address​

Nothing on your network owns 169.254.169.254. The compute node that hosts your instance answers for it, inside its own dataplane. The values it serves come from the control plane, and the compute node keeps a copy of each instance's values for 30 seconds. A read that needs a fresh copy while the control plane is unreachable gets 503, realized view unavailable.

When metadata is enabled for an instance, the platform adds a host route for each metadata address to that instance's network port. The port then hands every TCP packet the guest sends to the metadata address on port 80 to the compute node's agent, tritonagent, as the raw frame, tagged with the port it came from. The agent runs a small TCP stack for each port, answers the HTTP request, and sends the reply frames back into the same port.

Three properties follow from that path.

Identity comes from the port, not from the request. The dataplane hands over only frames whose source MAC and source IP are the pair assigned to that NIC. The agent looks up which instance owns the port and answers for that instance. Nothing in the HTTP request, no header and no path, can name a different instance.

Other traffic to the address goes nowhere. UDP, ICMP, and TCP to any other port addressed to a metadata address are dropped at the port. The drop applies whether or not metadata is enabled, so turning metadata off for an instance leaves the address unreachable from it, not routed elsewhere.

Nothing leaves the compute node. The request and the reply travel between the guest and the agent on the same host. They never cross the physical network.

Every request carries a session token​

The service accepts only token-authenticated reads, in the same shape as EC2's IMDSv2. A process first sends PUT /latest/api/token with the header X-aws-ec2-metadata-token-ttl-seconds, and receives a token. Every other request carries that token in X-aws-ec2-metadata-token. A request without a valid token gets 401.

The token is bound to the port and the instance it was issued to. A token copied to another instance fails there, because the agent checks the token against the port the request arrived on. Each compute node signs tokens with its own key, so a token issued on one compute node does not verify on another. An instance that moves to another compute node requests a new token.

The PUT itself is a defense against server-side request forgery: a flaw that only lets an attacker make your application fetch a URL with GET cannot obtain a token. The reply's IP time-to-live is a second defense. It is 1 by default, so a reply cannot be forwarded one hop further, for example into a container network inside the guest. The config/imds/hop-limit key raises it when you need that.

Where the values come from​

The service serves one merged set of keys for each instance, called the realized view. It has two sources.

The platform computes some keys from the instance's own records every time the view is built: the instance ID, image, hostname, CPU and memory, the primary NIC's MAC and addresses, the VPC and subnet, the SSH public keys you attached, and any shared filesystem mounts. These live under meta-data/ and triton/system/. Nobody can set them.

The rest is metadata that you, or the operator of your cloud, store at one of four scopes: silo, tenant, project, or instance. A key set at a wider scope reaches every instance under it, and a narrower scope overrides it. Instance scope wins over project, project over tenant, and tenant over silo. You manage tenant, project, and instance scope with tritonctl meta; silo scope belongs to the operator.

NamespaceHoldsScopes
config/Shared configurationAny
state/Shared runtime stateAny
instance/Per-instance settingsAny
guest/Values the guest itself writesAny
user-dataThe cloud-init user-data blobInstance only

Each stored key carries a guest_visible flag. A guest reads only the keys with that flag set. config/ and state/ keys are visible by default at every scope; other keys are visible by default at project and instance scope, and hidden at tenant and silo scope. GET /triton/dynamic/realized returns the whole visible view as JSON, with the scope each value came from, so a guest can see why it got the value it got.

The agent caches each instance's view for 30 seconds. A change you make through tritonctl meta reaches the guest within that window.

The guest can write back, narrowly​

A guest can change the value of an existing guest/ key with PUT /triton/guest/<KEY>, and only when the key carries the guest_writable flag. The guest cannot create keys, delete keys, or change either flag. The write lands at the scope where the key already lives. A guest/ key that is writable at project scope is shared by every instance in the project, which makes it a coordination point between instances, and every one of them can overwrite it. Silo-scope keys are never writable from a guest.

User-data​

User-data is the instance-scope key user-data, set at create time with tritonctl instance create --user-data. The guest reads it at /latest/user-data. On a bhyve instance whose image uses cloud-init's NoCloud datasource, the platform also writes it into the cloud-init seed disk the instance boots with. In that case your user-data replaces the default cloud-config the platform would otherwise write, with nothing merged in. That default allows root sign-in, installs the SSH keys you attached, sets the generated root password, and works around a cloud-init bug that leaves the static IP address unapplied on some images. A user-data file for such an instance carries its own versions of those settings.

What the service does not serve​

The service publishes no instance identity document and no credentials. /latest/dynamic/iam/... answers 404. The guest cannot read which compute node hosts it.

How the compute node runs it​

The agent on each compute node runs the service. Whether it runs, and the settings an operator controls, are covered in Instance metadata settings in the operator documentation. For where the agent and the dataplane sit in the rest of the system, see Architecture.

See also​