How sandboxes work
A sandbox is a small Linux virtual machine for an agent or for code you do not trust. This page explains how a sandbox is built, how your commands reach it, how it starts in under a second, what each lifecycle state means, and what isolates it from other sandboxes and from the cloud.
A microVM inside a zone
Each sandbox is a microVM: a virtual machine with its own Linux kernel and very few virtual devices. It boots a guest kernel and a small init image in well under a second. The sandbox's disk is a ZFS clone of its template image, so a new sandbox copies no data at create time. Snapshots and forks are ZFS snapshots and clones of that disk.
The virtual machine monitor for each sandbox runs inside its own illumos zone on the compute node. That gives two walls between a guest and the host. The first is the virtual machine boundary. The second is the zone around the monitor. A guest that breaks out of its virtual machine lands in a zone that holds nothing but its own monitor.
In a sandbox made from a built-in template, commands run as an ordinary user
named sandbox, and the working directory for your files is /work. In a
sandbox made from your own template, commands run as the user its build
recorded, or as root when it recorded none.
One channel to the guest
A guest agent is the first process in every sandbox. It mounts the disk and then waits for requests on a virtio-vsock socket, which is a host-to-guest channel that does not use the network.
Every interactive operation uses that channel: exec, files, file watches, port tunnels, code runs, the desktop, and metrics. Nothing listens on the guest's network for these. They work when the sandbox's network policy denies all traffic, and a sandbox exposes no SSH port.
One connection carries many streams, and each stream has its own flow-control window. A busy stream slows only itself. The host treats every frame from the guest as hostile: sizes, counts, rates, and paths are checked, and a guest that breaks the protocol is cut off.
For each call, the Cloud API gives the compute node a short-lived ticket that names the operations the call may do. A ticket for a file read cannot run a command.
Lifecycle operations do not use the channel. Stop, start, fork, pause, and delete run as durable operations in the control plane, which then sends a job to the compute node.
The booted pool
A cold create makes a zone, boots the guest, and waits for the guest agent. That takes about 1.5 seconds. To go faster, an operator can turn on a pool: each sandbox compute node then keeps a few sandboxes already booted, with their network port closed so they send and receive nothing. The pool is off until the operator sets a pool depth.
The control plane gives each pool entry its identity when it fills the entry: an instance ID, a sandbox ID, a network interface ID, and a MAC address. When your create takes the entry, the sandbox adopts that identity. No zone is renamed and no MAC address changes on a running virtual machine. The compute node then applies your network policy, opens the port, and sends your environment and start command to the guest.
Only a create from the built-in base template whose image and size match the
default package takes a pool entry. A create with a different shape, any other
template, or a fork boots cold. A pool entry is used
once. It is destroyed after a failed take, an image or runtime change, or an
agent restart, and never handed to a second tenant. If the entry's zone or
guest is down at take time, the create falls back to a cold boot.
Pool entries hold their guest memory while they wait. The pool depth is one cloud-wide setting: the number of entries each sandbox compute node keeps.
Sizes come from packages
A sandbox gets its vCPUs, memory, and disk from a sandbox package. The default
set is sandbox-2g, sandbox-4g, and sandbox-8g. A create that gives sizes
as numbers is refused, and the error lists the packages you can use. A
template sets the image and its defaults, but never the size.
A resize moves a stopped sandbox to another sandbox package. The disk never shrinks.
Lifecycle states
A sandbox reports one state at a time. A create starts at starting. Clients
must accept a state they do not know, because new states can be added without
a new API version. The API reports unknown for a stored state it has no name
for.
| State | Meaning |
|---|---|
starting | The sandbox is being placed and the guest is booting. |
running | The guest agent answers. |
stopping, stopped | The guest halts. The disk is kept, memory is not. |
pausing, paused | Memory and disk are saved. The zone halts and frees its memory. |
resuming | The saved memory is being restored. |
resizing | A stopped sandbox is moving to another package. |
deleting | The sandbox is being removed. Reads answer 404 soon after. |
failed | The error field says why. |
The API's state type also names pending, placing, forking,
snapshotting, and deleted, but no sandbox reports them. A sandbox that is
being forked or snapshotted stays running or stopped.
If the guest exits by itself, the sandbox goes to stopped. A resume that
fails before the guest runs keeps the saved memory, so the sandbox goes back
to paused and the resume can be asked for again. If the compute node reports
the saved memory gone, the sandbox goes to stopped with the reason
memory_lost. The sandbox record's error reads SandboxStateConflict, "the
paused memory could not be restored", and guest calls answer
409 SandboxNotRunning until it starts again. A start boots it cold with its disk.
Deadlines and idle time
Every sandbox has a deadline. You set it with a time to live (TTL) at create
time, and keepalive moves it later. The TTL defaults to 1 hour, and a create
may ask for at most 24 hours. A start or a resume resets the deadline to now
plus the TTL, or plus the TTL it names when that is longer. An update with --ttl, --idle-timeout, or
--on-deadline resets it to the creation time plus the TTL, which can move it
earlier. An idle timeout can also
apply: guest CPU and I/O count as activity.
When the deadline or the idle timeout arrives, the sandbox does one of three things, chosen at create time:
delete, the default.stop, which keeps the disk.pause, which keeps memory and disk. It needs thememoryfeature gate. Where memory cannot be kept, on a desktop sandbox or a compute node that cannot save memory, the deadline stops the sandbox instead. A pause that still fails 30 minutes after the deadline also becomes a stop.
A stopped or paused sandbox is deleted, with its disk and any saved memory, 7
days after it stops or pauses, or at its own deadline if that is later. An
update with --ttl, --idle-timeout, or --on-deadline on a stopped or paused
sandbox resets the deadline to the creation time plus the TTL, which can make it
due for deletion at once.
With auto-resume on, an exec, a file call, or a code run starts or resumes a
stopped or paused sandbox. Without it, those calls answer
409 SandboxNotRunning. Only the SDKs turn auto-resume on, with
auto_resume. tritonctl sandbox create always sends it off.
Memory: pause, memory snapshots, and memory forks
A pause freezes the guest's processes, stops the vCPUs, snapshots the disk at the same point, and writes memory and device state to files. The zone then halts. A paused sandbox holds no memory on the compute node. A resume restores that state, and the guest's processes go on from where they stopped.
A memory snapshot is the same save, but the sandbox keeps running. A memory fork starts children from a memory snapshot. Each child gets a new MAC and IP address, a new hostname, the correct clock, and a reseeded random number generator, so no two children share random numbers.
Everything in the parent's memory at snapshot time is in every memory fork, including secrets and open sessions. Take the snapshot before you load secrets.
A filesystem snapshot copies only the disk, with the guest filesystem frozen for a moment. A filesystem fork boots fresh children from it. Snapshots, forks, and memory restores stay on the parent's compute node. Snapshots belong to their sandbox and are deleted with it. To keep one, build a template from it.
Pause, memory snapshots, memory forks, and on_deadline pause are behind
the memory feature gate, which an operator turns on per tenant. A resume
needs no gate. A desktop sandbox never keeps its memory, and some compute
nodes cannot save memory. On either, a pause, a memory snapshot, or a memory
fork answers 400 PersistModeUnsupported. A resume on a compute node that
cannot save memory answers the same error.
The isolation model
Several layers keep a sandbox apart from other sandboxes and from the cloud:
- The virtual machine boundary, then the zone around the monitor.
- Its own network port in your virtual private cloud (VPC), with all traffic in and out denied by default. See How sandbox networking works.
- A fixed set of address ranges that the guest can never reach, whatever its
policy says. Private, loopback, and link-local ranges are among them. The
exceptions are DNS to the platform's resolvers, the metadata service at
169.254.169.254on TCP port 80, and blocks inside the sandbox's own subnet that the policy allows. - No listening service for exec, files, tunnels, or the desktop. All of these use the vsock channel, behind your API credential.
- Template builds from an OCI image or a Dockerfile run in a separate builder sandbox, so a hostile Dockerfile is one more untrusted guest.
Environment values you pass at create time are never returned by the API. Registry credentials for a template build are write-only.
Measured timings
These numbers were measured on a test cluster with the default package. They show the order of each step, not a guarantee.
| Step | Time |
|---|---|
| Create from the pool, client next to the API | about 360 ms (p50) |
| Cold create | about 1.5 s |
| Pause | 3 to 4 s |
| Resume | about 2 s |
| Memory fork, ready to first exec | about 0.3 s |
| Stop, then start | about 2 s, then 1 to 1.6 s |
Template build from python:3.12-slim | about 66 s |
| Create from a custom template | about 3.9 s |
Current limits
- Linux guests only, on amd64.
- The pool is off unless an operator sets a pool depth. When it is on, only
the
basetemplate at the default package's shape is pooled. Everything else boots cold. - A tenant holds at most 32 sandboxes and creates at most 60 a minute, unless
an operator changes these limits. Past them, a create answers
403 QuotaExceededor429 TooManyRequests. - Pause and resume time grows with the sandbox's memory size.
- Snapshots, forks, and memory restores stay on the parent's compute node.