Sandbox capacity
How many sandboxes fit on a compute node, and how many one tenant may hold, comes from four limits. This page explains each one, how the warm pool spends memory to buy fast creates, and how to estimate a node's capacity.
Memory: the reservoir share
Each sandbox guest takes its memory from the bhyve reservoir, which is memory set aside for guests. On a sandbox compute node (CN), the agent sizes the reservoir to the smallest of three bounds:
- the sandbox share of physical memory:
sandbox.reservoir_percent_default,0.25by default, or the per-nodetritonadm cn sandbox set --reservoir-percent; - physical memory minus the sum of
max_physical_memoryover the zones the CN runs; - the kernel's own reservoir limit.
The second bound keeps the reservoir from starving zones that already run beside the sandboxes. The agent applies the target when it starts and again every 300 seconds, because those zones come and go. The reservoir does not survive a reboot, so the startup pass sizes it again. A sandbox CN does not apply the general reservoir floor.
For most packages, the reservoir is the limit that refuses first. A rough estimate:
sandboxes per CN ≈ reservoir target ÷ package memory
A CN with 512 GiB of memory, few other zones, and the default share of 0.25 holds
128 GiB for sandboxes, which is about 64 sandbox-2g sandboxes. Pool entries hold
reservoir memory too, but the agent destroys an idle entry to make room for a cold
create, so they do not lower this number. The count cap below then applies.
If reservoir management is off on a CN, every sandbox create there is refused. On a
CN with sandboxes enabled and no reservoir override, the agent follows the cluster
setting reservoir.enabled_default, so setting it to false refuses sandbox creates
on every such CN.
The per-node count cap
sandbox.max_per_node_default, or the per-node --max, caps the sandboxes one CN
runs at once. Pool entries count against it, but the agent destroys an idle entry
to make room for a create. The range is 1 to 50, and the default is the cap,
50. A higher value is refused.
The cap exists because each sandbox is a zone with its own monitor process and host threads, so the cost per sandbox is not only memory. On a test cluster, 32 sandboxes on one node left the CPU 95 percent idle, with exec in all 32 at about 314 ms (p50).
The warm pool
sandbox.pool_depth sets how many booted sandboxes of the default package each
sandbox CN keeps waiting. The range is 0 to 8, and 0, the default, turns the pool
off. Only a CN whose traits take the default package keeps a pool.
A pool entry turns a create of about 1.5 seconds into one of about 0.4 seconds, measured on a test cluster. The cost is memory: each entry holds its guest memory in the reservoir while it waits. Two entries of a 2 GiB package hold 4 GiB on every sandbox CN, whether or not anyone creates a sandbox. A cold create that needs room destroys an idle entry.
The pool keeps the shape of sandbox.default_package as the default silo defines it,
and only on CNs whose traits take that package. A create takes an entry when it uses
the base image, its package has the same vCPUs, memory, and disk, and its placement
takes the entry's CN. Other packages, custom templates, and forks boot cold, so a
pool helps only when most creates use that shape.
Drain the pool, with sandbox.pool_depth 0, before you change the agent, the runtime
bundle, or an image pin, so no pooled zone holds the old pieces.
Per-tenant limits
Two settings bound one tenant, whatever the fleet's room:
| Setting | Default | Limit |
|---|---|---|
sandbox.max_per_tenant | 32 | Live sandboxes one tenant may hold. Read on every create. |
sandbox.creates_per_minute_per_tenant | 60 | Creates per tenant per minute, per control-plane node. |
Lifetimes bound how long capacity stays held. sandbox.default_ttl_secs (3600) is
the lifetime when a create names none, and sandbox.max_ttl_secs (86400) is the
longest one a create may ask for. A paused sandbox holds no memory on its CN, but a
stopped or paused sandbox keeps its disk.
When a create is refused
| Answer | Meaning |
|---|---|
503 SandboxNoCompatibleNode | No CN takes the sandbox now: none has room, none takes the package's traits, or none has sandboxes on. The cold-create message is no host has room for this sandbox now; retry later or ask for less. Nothing was created, and a retry is safe. |
403 QuotaExceeded | The tenant holds sandbox.max_per_tenant live sandboxes. |
429 TooManyRequests | The tenant reached sandbox.creates_per_minute_per_tenant. The message gives the seconds to wait. |