tritonadm MantaBlock commands
The MantaBlock commands of tritonadm are spread over four families. This
page covers mantablock format, instance move-to-mantablock, the MantaBlock
rules of instance migrate, the handoff lines of migration show, and
migration recover. The per-node role command, cn mantablock, is in the
tritonadm cn reference.
Synopsis
On any host where tritonadm is signed in to the cluster, as the signed-in user:
tritonadm mantablock format <FORMAT>
tritonadm instance move-to-mantablock <INSTANCE> [--copies <N>] [--tier <TIER>] [--stop-grace <SECONDS>] [--full-verify] [SCOPE]
tritonadm instance move-to-mantablock <INSTANCE> --finish [SCOPE]
tritonadm instance move-to-mantablock <INSTANCE> --release [SCOPE]
tritonadm migration show <MIGRATION>
tritonadm migration recover <MIGRATION> (--to-source | --to-target)
Execution context
Every command on this page is an HTTP call against a running tritond. It runs
on any host where tritonadm resolves an endpoint and a credential. The
tritonadm command index documents that resolution and the global
options. Every command here is operator-only.
<INSTANCE> takes a name, a UUID, or a UUID prefix. The scope flags narrow a
name or prefix lookup:
| Scope flag | Value | Default | Description |
|---|---|---|---|
--tenant <TENANT> | Name, UUID, or prefix | configure --tenant | Tenant to search, with --project. |
--project <PROJECT> | Name, UUID, or prefix | configure --project | Project within --tenant. |
--cn <CN> | Hostname, UUID, or prefix | None | Search the instances on this compute node. |
--image <IMAGE> | Name, name@version, UUID, or prefix | None | Search the instances from this image. |
<MIGRATION> takes a migration UUID or a UUID prefix.
tritonadm mantablock format
Synopsis
On any host where tritonadm is signed in to the cluster, as the signed-in user:
tritonadm mantablock format <FORMAT>
Description
Raises the MantaBlock format that new volumes take. Calls
POST /v1/system/mantablock/format with {"format": <FORMAT>}.
The cluster starts at format 1. Format 2 is the highest this build reads,
and adds 512-byte sectors. Volumes with 512-byte sectors need format 2:
- image volumes, and the replicated boot disks cloned from them, which always use 512-byte sectors,
- a disk move from
instance move-to-mantablockof a guest that sees 512-byte sectors.
Data disks use 4096-byte sectors and need format 1 only.
The registry does not check which engines run on the compute nodes. Raise the format only after every node runs an engine that reads it. The call raises the active format only; it never changes the format floor.
A value at or below the active format changes nothing and succeeds.
Arguments
| Argument | Value | Required | Description |
|---|---|---|---|
<FORMAT> | Integer | Yes | The format to raise to. |
Output
active format <ACTIVE>, format floor <FLOOR>
With -j, the object {"active_format": <ACTIVE>, "format_floor": <FLOOR>}.
Errors
| Condition | Response |
|---|---|
<FORMAT> above 2 | 400, format <FORMAT>: this build reads up to 2 |
| The registry is fenced, for example during a restore hold | 409 |
tritonadm instance move-to-mantablock
Synopsis
On any host where tritonadm is signed in to the cluster, as the signed-in user:
tritonadm instance move-to-mantablock <INSTANCE> [--copies <N>] [--tier <TIER>] [--stop-grace <SECONDS>] [--full-verify] [SCOPE]
tritonadm instance move-to-mantablock <INSTANCE> --finish [SCOPE]
tritonadm instance move-to-mantablock <INSTANCE> --release [SCOPE]
Description
Without --finish or --release, moves the instance's boot disk from its ZFS
volume (zvol) onto a new MantaBlock volume. Calls
POST /v1/instances/{instance_id}/mantablock-move. The server starts the
instance-mantablock-move saga and answers with the move and operation IDs.
The saga runs these phases:
- Marks the disk as moving and probes it. The disk must be whole 4 KiB blocks, and the node's pool must have room for the move's snapshots.
- Creates the volume. The instance's node holds one copy, so it must pass
the member checks: an approved node with the
storagerole, registered with MantaBlock, with a usable journal slice of the tier, pools under 75% full, and provisioned volumes under twice its pool size.--copiesminus one other nodes must pass the same checks, unless the instance already has MantaBlock volumes, whose other members are reused. - Copies the whole disk while the guest runs, then up to 8 sync passes. The passes end early once one pass changes less than 256 MiB.
- Stops a running guest. The agent sends an ACPI power-off and forces the stop when
the guest has not powered off after
--stop-graceseconds. - Copies the last changes with the guest stopped.
- Switches the zone to the volume. A guest that was running gets a start
job, and the operation waits for that job to finish. A guest that was
stopped stays stopped. A failed start job sets the instance
failed; the disk stays on the volume.
The zvol stays on the node after the move, tagged with the ZFS property
triton:mantablock-moved-to. --finish removes it.
--finish calls POST /v1/instances/{instance_id}/mantablock-move/finish and
starts the instance-mantablock-move-finish saga. The agent destroys the
kept zvol and restores the zone quota. It refuses to remove a zvol that is
still a disk of the zone.
--release calls POST /v1/instances/{instance_id}/mantablock-move/release
and queues a job on the instance's node. The job drops the snapshots and
compare clones that failed moves left, and restores a raised zone quota. It
skips kept zvols. It runs only while no move step of the instance runs on the
node.
Options
| Flag | Value | Default | Description |
|---|---|---|---|
--copies <N> | 2 or 3 | 3 | Copies of the volume. When the instance already has MantaBlock disks, use their copy count. |
--tier <TIER> | performance or standard | performance | Journal tier. |
--stop-grace <SECONDS> | Integer, at most 3600 | 60 | Seconds the guest has to power off before the stop is forced. |
--full-verify | Flag | Off | Compare the whole volume with the disk while the guest is stopped. The downtime grows with the disk size. |
--finish | Flag | Off | Remove the zvol a finished move kept. Conflicts with --copies, --tier, --stop-grace, and --full-verify. |
--release | Flag | Off | Release what failed moves left on the node. Conflicts with every other option above. |
A --tier value other than the two names fails in the CLI with
tier "<VALUE>" is not performance or standard.
Output
A move or a finish prints:
move <MOVE_ID> started
follow it with: tritonadm operation show <OPERATION_ID>
--finish prints finish in place of move. --release prints:
release queued as job <JOB_ID>
follow it with: tritonadm job show <JOB_ID>
Errors at submit
| Condition | Response |
|---|---|
--copies other than 2 or 3 | 400, copies must be 2 or 3 |
--stop-grace above 3600 | 400, stop_grace_secs must be at most 3600 |
Instance brand is not bhyve | 409, only bhyve instances can move their disk to MantaBlock |
Instance not running or stopped | 409, the instance must be running or stopped |
| A move already runs | 409, a move of this instance's disk is in progress |
| No boot disk | 409, the instance has no boot disk |
| Boot disk already a MantaBlock volume | 409, the instance's boot disk is already on MantaBlock |
| A migration runs | 409, the instance is migrating |
| Not placed on a node | 409, the instance is not placed on a node |
| Node agent lacks the move or disk feature | 409, the instance's node agent does not report <FEATURE>; update it and the mantablock bundle |
--finish with no kept zvol | 409, the instance keeps no disk from a finished move |
--release, instance brand is not bhyve | 409, only bhyve instances move their disk to MantaBlock |
--release, boot disk not a plain zvol: a move runs, a zvol is kept, or the disk is already on MantaBlock | 409, the boot disk is moving or keeps its old disk; wait for the move, or finish it |
--release, node agent lacks the move feature | 409, the instance's node agent does not report mantablock-move-v1 |
<FEATURE> is mantablock-move-v1 or mantablock-disks-v1.
Errors inside the saga
These end the operation. Read them with tritonadm operation show.
| Message | Cause |
|---|---|
the disk's block size is <N>; moving it needs whole 4 KiB blocks | The zvol block size is not a multiple of 4 KiB. |
the disk is <N> bytes, not whole 4 KiB blocks | The disk size is not a multiple of 4 KiB. |
the node's pool has <N> bytes free; the move's snapshots can take up to <NEED> | Too little free space on the node. |
the instance's node cannot hold a copy of a MantaBlock volume (<REASON>); a move needs one there until remote copies are supported | The node fails a member check. <REASON> is the specific refusal, for example it is a compute-only CN, it is not an approved node, no usable journal slice of tier <T>, or its pools are 75% full or more. |
<N> more compute nodes can hold a copy; the disks need <COUNT> | Too few other nodes pass the member checks. |
the instance's other MantaBlock disks have <N> copies; move with the same | --copies differs from the instance's existing volumes. |
A registry refusal naming the format, followed by a hint to run tritonadm mantablock format 2 | The guest sees 512-byte sectors and the cluster is at format 1. |
the instance is changing state; try the move again | The saga could not hold the instance for the stop. |
If the switch fails, the saga tries to undo it 3 times, 10 seconds apart. If
no undo is confirmed, the zvol is intact and the saga marks the instance
failed with
MantaBlock move: the zone switch is unknown (move <MOVE_ID>); undo it on the node, then start the instance.
tritonadm instance migrate with MantaBlock disks
tritonadm instance migrate dispatches on the instance's disks. Only the
MantaBlock rules are listed here.
A live migration needs a running bhyve instance whose every disk is a
MantaBlock volume. It copies no disk data. The target must hold an active
member of every volume, and the source must hold a member of every volume.
--target-cn must name such a node.
While the handoff is committing, the saga sends the guest back to the source
if the handoff has not finished 15 seconds after the go. Registry read
failures, and a source whose rshyve status is in flight or unknown, keep the
saga waiting past that budget. Once the handoff is target, it cannot go back
to the source.
A cold migration (--cold) of an instance with MantaBlock disks needs the
instance stopped. It moves each volume's coordinator and copies only the
local disks. An instance with both local and MantaBlock disks migrates cold
only.
The server refuses these before it starts the migration:
| Condition | Response |
|---|---|
| Live, instance stopped | 409, the instance is stopped; migrate it cold |
| Live, a local disk | 409, disk <NAME> is on the node's local storage; live migration needs every disk on MantaBlock, so migrate the instance cold |
| Live, no MantaBlock disk | 409, live migration needs every disk on MantaBlock |
| A disk keeps its zvol from a move | 409, disk <NAME> still keeps its zvol from the move; finish the move first (tritonadm instance move-to-mantablock --finish) |
| Source agent or engine too old | 409, the source node's agent does not report <FEATURE>; update it, or the source node's MantaBlock engine does not report migrate-v1; update it |
| Cold, instance running | 409, stop the instance first: its MantaBlock disks move cold only while it is stopped |
The target is chosen inside the migrate-instance-mantablock saga, after the
server has answered with the migration and operation IDs. A target problem
fails that operation with the kind migration.refused and one of these
reasons. Read it with tritonadm operation show <OPERATION_ID>.
| Condition | Reason |
|---|---|
| A volume with no member on the source | volume <VOLUME> has no member on the source; a remote coordinator cannot move yet |
--target-cn without a member of every disk | node <CN> does not hold a member of every disk; targets: <LIST> |
| No other node holds a member of every disk | no other node holds a member of every disk |
| No member node passes the engine and CPU checks | no member node can take the instance: <REASONS> |
tritonadm migration show
Description
Prints the migration record from GET /v1/system/migrations/{migration_id}.
When the migration has a MantaBlock handoff record, the command then reads
GET /v1/system/migrations/{migration_id}/mantablock and prints the handoff:
MantaBlock handoff: <STATE>
volume <VOLUME_UUID> home <CN_UUID> zone <ZONE> fence <FLOOR>: <ACKED> acked, <LOST> lost, <NEED> needed
A volume with no fence prints the first three fields only. A missing home or
zone prints none. The handoff lines appear only in table output. With -j,
the command prints the migration record alone. Once the handoff record is
gone, no handoff lines print.
<STATE> | Meaning |
|---|---|
prepared | Before the go. Only the source can hold the guest. |
committing | The go was given. The source can be paused or can run again. |
target | The target owns the guest. The source never resumes it. |
source | The handoff went back to the source. |
aborted | The handoff was stopped before the go. |
unknown | A state this tritonadm build does not know. |
In committing and target, the command adds one line of guidance:
| State | Guidance printed |
|---|---|
target | The target owns the guest; the source never resumes. If the guest does not run on the target, power that node off and restart the instance from its volumes. |
committing | The go was given; the source may be paused or may run again. If this does not move, check rshyve on both nodes, then run `tritonadm migration recover`. |
tritonadm migration recover
Synopsis
On any host where tritonadm is signed in to the cluster, as the signed-in user:
tritonadm migration recover <MIGRATION> (--to-source | --to-target)
Description
Decides a MantaBlock live migration that is stuck after its go, or clears one
that stopped before it. Calls POST /v1/system/migrations/{migration_id}/recover
with {"to": "source"} or {"to": "target"}. One of the two flags is
required, and they conflict.
The CLI reads the handoff state first. --to-target is accepted only in
committing or target; in any other known state the CLI fails with
the handoff is <STATE>; --to-target applies only after the go.
| Flag | Handoff state | Effect | Output |
|---|---|---|---|
--to-target | target | The migration finishes on the target. | migration <ID>: the target owns the guest; the migration finishes |
--to-target | committing | The handoff goes on once the source rshyve shows its commit, and waits for every member fence. | migration <ID>: the handoff goes on to the target once the source rshyve shows its commit; it waits for every member fence |
--to-source | committing | The source resumes the guest once its rshyve answers, and the target is removed. | migration <ID>: the source resumes the guest once its rshyve answers; the target is removed |
--to-source | prepared, aborted, source | The migration stops and the guest stays on the source. With no migration running, the leftover record is cleared. | migration <ID>: stopped before the go; the guest stays on the source |
Choose --to-source after the go only when the target never started the
guest. A stored switch to the target does not force it: when the source
reports the guest running or stopped, the migration still rolls back to the
source.
Errors
| Condition | Response |
|---|---|
--to-source once the registry gave the guest to the target | 409, the registry gave the guest to the target; the source never resumes it |
| Migration ended on the target | 409, migration <ID> has ended |
| Not the instance's migration in progress | 409, migration <ID> is not the instance's migration in progress |
--to-target before the go, on the server | 409, the registry record is <STATE>; --to-target applies only after the go |
| Volumes of the migration unknown | 409, the migration's volumes are unknown |
Silo storage classes
No tritonadm command sets which storage classes a silo offers.
tritonadm silo provisioning-policy set takes only --instance-sizing, and
show does not print the classes. The field is storage_classes in the silo's
provisioning policy, written with PUT /v1/silos/{silo_id}/provisioning-policy
on the admin API:
"storage_classes": {
"enabled": ["local", "replicated_standard", "replicated_performance"],
"copies": 3
}
| Field | Values | Default |
|---|---|---|
enabled | One or more of local, replicated_standard, replicated_performance, each at most once | ["local"] |
copies | 2 or 3 | 3 |
A policy that breaks these rules is refused with
storage_classes.enabled must name at least one class,
storage_classes.enabled names a class twice, or
storage_classes.copies must be 2 or 3.
Endpoints and authorization
| Command | Method and path | Authorization |
|---|---|---|
mantablock format | POST /v1/system/mantablock/format | Action MantablockFormatRaise |
instance move-to-mantablock | POST /v1/instances/{instance_id}/mantablock-move | Action InstanceMantablockMove |
instance move-to-mantablock --finish | POST /v1/instances/{instance_id}/mantablock-move/finish | Action InstanceMantablockMove |
instance move-to-mantablock --release | POST /v1/instances/{instance_id}/mantablock-move/release | Action InstanceMantablockMove |
migration show (handoff lines) | GET /v1/system/migrations/{migration_id}/mantablock | Action MigrationList |
migration recover | POST /v1/system/migrations/{migration_id}/recover | Action MigrationRecover |
Read-only API keys cannot reach the three write actions. Each is allowed by the root-operator rule.