MantaBlock
MantaBlock is the replicated block storage behind the replicated_standard
and replicated_performance disks of bhyve instances. This page explains
what a volume is, which compute nodes hold its copies, what runs on each node,
and why a MantaBlock instance can migrate live without copying its disks.
Volumes, members, and coordinators
Each replicated disk is one MantaBlock volume. A volume has n copies,
called members, where n is 2 or 3, each on a different compute node (CN). A
write is acknowledged once two members have taken it. The volume is addressed
in 4096-byte blocks.
One CN is the volume's coordinator, also called its home. It is the node that runs the instance. Its engine serves the volume to the instance's zone on a Unix socket, and sends each write on to the members.
The registry of volumes, members, and node roles lives in FoundationDB, under
tritond. Engines never reach FoundationDB themselves. They call tritond
with their node's agent key, and tritond scopes every call to that node.
While tritond is down, a coordinator cannot start, but running volumes keep
their I/O.
Storage classes and journal tiers
The two replicated storage classes differ in where each member keeps its write journal:
| Storage class | Journal tier | Journal location |
|---|---|---|
replicated_performance | performance | Power-loss-protected NVMe drives |
replicated_standard | standard | Files on the node's pool |
performance is the default tier. A silo's provisioning policy lists the
classes its tenants can use and the copy count for every replicated disk. The
default policy offers local only, with 3 copies. Tenants cannot choose copies
or tier; they name a class. An operator request can name any class and copy
count. No tritonadm command sets the silo's classes; see
Silo storage classes.
Node roles
Every CN is a storage node, which holds members, or a compute node, which
holds none. The roles and how to change them are in
tritonadm cn mantablock.
A compute node needs no journal slice, journal drive, or capacity config; the
volumes of its instances live entirely on storage nodes. A node the
registry does not know is treated as storage. Because the engine reads the
role only when it starts, a node set back to storage can be given copies its
engine refuses until site/mantablock-engine restarts.
Which nodes hold the copies
When an instance with replicated disks is created, the platform first finds the storage nodes that can hold a member. A node is refused as a member when:
| Refusal | Meaning |
|---|---|
the node is not registered for MantaBlock | The engine has not registered the node. |
the node is compute-only: it holds no copy | Its role is compute. |
no usable journal slice of tier <T> | No healthy journal slice of the tier the disk needs. |
the node is unfit for the standard tier | The node is marked unfit for standard journals. |
the node has no capacity report | The agent has not reported pool space. |
its pools are 75% full or more | Pool fill limit. |
its volumes would pass twice its pool capacity | Overcommit limit: provisioned volume bytes stay under twice the pool size. |
Among the nodes that pass, those that already hold a base copy of the image
come first, then those with the fewest provisioned bytes. The create needs at
least as many eligible nodes as the largest copy count. With fewer, the tenant
sees Replicated disks are not available here right now. Use local disks, or ask your administrator.
Placement then allows two kinds of host:
- a storage node that can hold a member, when enough storage nodes can hold the copies;
- a
computenode, when every MantaBlock disk of the instance has 3 copies, none is a replicated boot disk, three storage nodes whose engines accept a coordinator that is not a member can hold them, and the compute node's agent reportsmantablock-compute-only-v1.
Every host must also report the mantablock-disks-v1 feature, plus
mantablock-boot-v1 for a replicated boot disk. The restriction holds even
when the request pins a node.
Replicated boot disks and image volumes
A replicated boot disk is a clone of an image volume. The first time an image
is used for a replicated boot disk, the platform imports it into its own
MantaBlock volume with 3 copies, or 2 when only two nodes can hold one. The
image must be a bhyve image with a source URL and a SHA-256 checksum, and its
file must be a gzip-compressed full ZFS volume stream of at most 16 TiB.
Image volumes and their clones use 512-byte sectors, because bhyve images
were built on ZFS volumes. The tritond sweeper seals imported image
volumes, retires those whose image left the catalog, and removes a retired
image volume once no clone references it.
Formats
New volumes take the cluster's active format. Format 2 adds 512-byte
sectors, which image volumes need, so replicated boot disks depend on it. The
format is raised by hand, after every engine can read it, because the registry
does not check engine versions. See
tritonadm mantablock format.
What runs on a compute node
The mantablock bundle unpacks under /opt/triton/mantablock and carries:
| Path | What it is | Source repository |
|---|---|---|
bin/mantablock-engine | The engine | manta-storage |
bin/mantablock-slice | Formats and registers journal slices | manta-storage |
bin/mantablock-import | The copy tool of image imports and disk moves | manta-storage |
vmm/rshyve | The VMM of MantaBlock instances | rust-bhyve |
smf/ | The site/mantablock-engine service |
site/mantablock-engine ships disabled. Its start method reads
/opt/triton/mantablock/etc/engine.env and takes the tritond endpoint from
the agent's agent.env and the node UUID from sysinfo. It logs to
/var/log/mantablock/engine.log. Replication between nodes uses mutual TLS
with /var/mantablock/tls/{cert,key,ca}.pem on every node, including compute
nodes, which dial their volumes' members.
site/mantablock-engine serves each volume at
/var/run/mantablock/sock/<INSTANCE_UUID>/<VOLUME>.sock. For a MantaBlock
instance, tritonagent mounts the socket directory into the zone at /mb and
the VMM directory at /rt, both read-only, and points the zone's /bhyve.zone
at /rt/rshyve. The brand's init then runs rshyve instead of the platform
bhyve. Each volume becomes an NVMe device, and the agent boots the zone only
after every socket accepts a connection. A node without the bundle fails the
boot with <PATH> is missing: install the mantablock bundle on this CN.
tritonadm installs no part of this bundle. Its install, setup, and update
targets do not name it, so the bundle, its service, and the journal slices are
put in place by hand on each node.
Moving a local disk onto MantaBlock
An operator can move a bhyve instance's local boot disk onto a new volume
with tritonadm instance move-to-mantablock. The copy runs while the guest
runs, followed by sync passes that shrink the difference. The guest then
stops, the last changes are copied, and the zone switches to the volume and
starts again. The instance's node holds one member, so it must be a storage
node. The old zvol stays until the operator finishes the move. See
Move an instance's boot disk to MantaBlock.
Live migration
A live migration of a MantaBlock instance moves no disk data. The members stay where they are; only each volume's coordinator moves. That is why the target must be a node that already holds a member of every disk, and why every disk must be on MantaBlock: a local disk would have to be copied, and an instance with one migrates cold only.
rshyve carries the guest. It copies RAM while the guest runs, then pauses the
guest, drains the disk rings, and waits at a cutover gate. The agents answer
the gate with tritond's go. After the go, the registry hands each volume's
coordinator to the target, and the target resumes the guest. The handoff state
moves forward only: prepared, then committing at the go, then target or
source. While the handoff is committing, the saga sends the guest back to
the source if the handoff has not finished 15 seconds after the go. Registry
read failures, and a source whose rshyve status is unclear, keep it waiting
longer. Once the handoff is target, there is no way back to the source.
A migration stuck after the go is decided by hand with
tritonadm migration recover.
tritonadm migration show prints the handoff state and each volume's fence.
A cold migration of a MantaBlock instance needs the instance stopped. It moves each volume's coordinator behind a fence and copies only the local disks.
Deleting volumes
When an instance is deleted, the delete saga fences each of its volumes as
deleting, so no engine serves it again, and waits up to 300 seconds for every
member to free its copy. If a member's node is down, the volume stays fenced
and tritond logs
alert: MantaBlock volume left deleting; a member did not free its copy.
tritonadm has no command to force that removal.
A node that holds or coordinates volumes cannot be factory reset or removed. See Reset or remove a compute node.
Current limits
- An instance has at most one data disk, and a replicated boot disk rules out a data disk.
- Replicated disks cannot be resized.
tritonadmhas no command to list or inspect volumes, mark a node lost, replace a member, or force-remove a fenced volume.- A volume's coordinator must also hold a member before it can migrate live; a remote coordinator cannot move yet.
- The bundle is installed by hand.