Move an instance's boot disk to MantaBlock
This guide moves a bhyve instance's boot disk from its local ZFS volume
(zvol) onto a new MantaBlock volume. The copy runs while the guest runs. A
running guest then stops for the last changes and starts again from the
volume; a stopped guest stays stopped. Once
every disk of an instance is on MantaBlock, the instance can migrate live.
For the volume model, see MantaBlock.
Before you start, you need:
tritonadmsigned in to the cluster as a root operator.- A
bhyveinstance that isrunningorstopped, placed on a compute node, with no migration and no other move in progress. - An agent on the instance's node that reports the
mantablock-move-v1andmantablock-disks-v1features, and the mantablock bundle installed there.
The commands are control-plane calls. The examples run on the founder. <INSTANCE>
is the instance's name, UUID, or a UUID prefix; add --tenant and --project
to narrow a name.
Step 1: Find the instance's node
On the founder, in the global zone, as root:
tritonadm instance show <INSTANCE>
The cn line names the compute node. Call it <CN>.
Step 2: Confirm the node can hold a copy
The instance's node holds one copy of the new volume, so it must be a
storage node that can take a member. On the founder, in the global zone, as
root:
tritonadm cn mantablock show <CN>
The command prints CN <SERVER_UUID> MantaBlock role: storage. The role is
not enough on its own. The saga also needs the node to be registered with
MantaBlock, to have a healthy journal slice of the tier you pick, to keep its
pools under 75% full, and to stay under twice its pool size in provisioned
volumes. It also needs --copies minus one other nodes that pass the same
checks. tritonadm has no command that reports these checks in advance; a
failed check ends the move with one of the messages in
If the move fails.
Step 3: Start the move
Pick the journal tier and the copy count. The defaults are performance and
3. If the instance already has MantaBlock disks, use their copy count.
A running guest is powered off near the end of the move, and forced off if it has not stopped after the grace time. Plan for that downtime.
On the founder, in the global zone, as root:
tritonadm instance move-to-mantablock <INSTANCE> --tier performance --copies 3
Output:
move <MOVE_ID> started
follow it with: tritonadm operation show <OPERATION_ID>
The guest gets 60 seconds to power off after its ACPI power-off before the stop
is forced. Set --stop-grace <SECONDS> to change that, up to 3600. Add
--full-verify to compare the whole volume with the disk while the guest is
stopped; the downtime then grows with the disk size.
Step 4: Follow the move
On the founder, in the global zone, as root:
tritonadm operation show <OPERATION_ID>
For a guest that was running when the move began, the operation ends after
the instance has switched to the volume and its start job has finished. A
guest that was stopped stays stopped. If the start job fails, the instance's
state becomes failed, and the disk stays on the volume: the move does not go
back to the zvol once the switch is done.
If the operation fails with a message about the cluster format, the guest sees
512-byte sectors and the cluster is still at format 1. Raise the format once
every node runs a MantaBlock engine that reads format 2, then go back to
step 3. On the founder, in the global zone, as root:
tritonadm mantablock format 2
The output reads active format 2, format floor <FLOOR>.
Step 5: Confirm the disk
On the founder, in the global zone, as root:
tritonadm disk list --instance <INSTANCE>
The boot disk's CLASS reads replicated_performance or
replicated_standard. Until the switch, it reads local.
Step 6: Remove the old zvol
The move keeps the old zvol on the node, tagged with the ZFS property
triton:mantablock-moved-to. While it is kept, the instance cannot migrate.
--finish destroys the old zvol. After it, the MantaBlock volume is the only
copy of the boot disk's old contents on that node, and the move cannot be
undone from the zvol.
On the founder, in the global zone, as root:
tritonadm instance move-to-mantablock <INSTANCE> --finish
Output:
finish <MOVE_ID> started
follow it with: tritonadm operation show <OPERATION_ID>
If the move fails
A failed move can leave snapshots, compare clones, and a raised zone quota on the node. Release them once no move step runs.
--release destroys the snapshots and compare clones that failed moves left.
They cannot be recovered. It does not touch a kept zvol.
On the founder, in the global zone, as root:
tritonadm instance move-to-mantablock <INSTANCE> --release
Output:
release queued as job <JOB_ID>
follow it with: tritonadm job show <JOB_ID>
Common failures:
| Message | Fix |
|---|---|
the instance's node agent does not report <FEATURE>; update it and the mantablock bundle | Update the agent and the mantablock bundle on the node. |
the instance's node agent does not report mantablock-move-v1 | --release refused: update the agent on the node. |
only bhyve instances move their disk to MantaBlock | --release refused: the instance is not bhyve. |
the instance's node cannot hold a copy of a MantaBlock volume (<REASON>); a move needs one there until remote copies are supported | Fix the cause named in <REASON>, as in the next table. |
<N> more compute nodes can hold a copy; the disks need <COUNT> | Too few other nodes can hold the remaining copies. Fix them as in the next table, or pass --copies 2. |
the node's pool has <N> bytes free; the move's snapshots can take up to <NEED> | Free space on the node's pool. |
the disk's block size is <N>; moving it needs whole 4 KiB blocks | This disk cannot be moved. |
the instance's other MantaBlock disks have <N> copies; move with the same | Pass --copies <N>. |
the instance is changing state; try the move again | Wait for the instance to settle, then repeat step 3. |
the boot disk is moving or keeps its old disk; wait for the move, or finish it | --release refused. The disk is still moving, keeps its zvol, or is already on MantaBlock. Wait for the move, or run step 6. A disk already on MantaBlock has nothing to release. |
<REASON> is the node's refusal:
<REASON> | Fix |
|---|---|
it is a compute-only CN | Set the node's role to storage with tritonadm cn mantablock set, then restart site/mantablock-engine on it. |
it is not an approved node | Approve the node, or run the instance on an approved one. |
the node is not registered for MantaBlock | Record the node and a formatted journal slice in the registry with mantablock-slice register. |
no usable journal slice of tier <T> | Format and register a slice of that tier with mantablock-slice, clear a faulty slice after its drive is replaced, or pick the other --tier. |
the node is unfit for the standard tier | Pick --tier performance. |
the node has no capacity report | Wait for the agent to report pool space. |
its pools are 75% full or more | Free pool space on the node. |
its volumes would pass twice its pool capacity | The node holds too many provisioned volume bytes for its pool. |
If the switch step fails, the saga tries to undo the switch 3 times, 10
seconds apart. If none of those undos is confirmed, it marks the instance
failed with MantaBlock move: the zone switch is unknown (move <MOVE_ID>); undo it on the node, then start the instance.
The zvol is intact at that point, and the saga keeps the volume and its
attachment because the zone can still name it.