tritonadm storage
tritonadm storage deploy places mantad object storage clusters on compute
nodes from a plan file, and tritonadm storage cluster manages the storage
cluster registrations that tritond forwards to. storage filesystem and
storage gateway, which manage shared filesystems, are not covered here.
Synopsis
tritonadm storage deploy plan --file <PLAN>
tritonadm storage deploy apply --file <PLAN>
tritonadm storage deploy export <CLUSTER_NAME> [--file <PLAN>]
tritonadm storage deploy status <CLUSTER_NAME>
tritonadm storage deploy replace --file <PLAN> --node-id <NODE_ID>
tritonadm storage cluster list
tritonadm storage cluster show <CLUSTER>
tritonadm storage cluster add --name <NAME> --cluster-endpoint <URL> (--admin-token <TOKEN> | --admin-token-stdin) [--surface <SURFACE>] [--default-region <REGION>] [--display-name <LABEL>]
tritonadm storage cluster bootstrap --spec <FILE>
tritonadm storage cluster delete <CLUSTER> [-n] [-y] [--keep-going]
tritonadm storage cluster health <CLUSTER>
tritonadm storage cluster set-replication <CLUSTER> <FACTOR>
tritonadm storage cluster set-presigner <CLUSTER> --access-key-id <AKID> (--secret-access-key <SECRET> | --secret-access-key-stdin) [--s3-endpoint <URL>]
tritonadm storage cluster clear-presigner <CLUSTER>
cluster list also answers to ls, and cluster delete to rm. <CLUSTER>
is a cluster UUID or its name.
Execution context
Every subcommand except deploy plan calls the tritond admin API. The
endpoint and credential resolve as described in
Endpoint and credential resolution.
No subcommand has a --wait flag. Only cluster delete prompts.
Plan file
A TOML document. Unknown keys are rejected.
| Key | Type | Required | Rules |
|---|---|---|---|
version | integer | yes | Must be 2. 1 fails with Mantad deployment plan version 1 is obsolete; .... |
cluster.name | string | yes | Non-empty. Also the name of the storage cluster registration. |
cluster.region | string | yes | Non-empty. The region mantad reports to S3 clients. |
cluster.image_uuid | UUID | yes | The triton-platform image. |
cluster.replication | integer | yes | Odd, at least 3, and no more than the node count. 2 fails on its own message. |
cluster.s3_endpoint | URL | yes | The tenant-facing endpoint. https, unless allow_insecure_s3_endpoint is true and the host is a private IPv4 address or a declared peer_ip on port 7443. |
cluster.allow_insecure_s3_endpoint | boolean | no | Default false. |
cluster.require_fdb_ha | boolean | yes | When true, the FoundationDB cluster must have certified high-availability evidence. |
control_network.nic_tag | string | yes | A-Z, a-z, 0-9, _. |
control_network.vlan_id | integer | no | 0 to 4094. Default 0, untagged. |
control_network.netmask | string | yes | A contiguous IPv4 netmask. |
control_network.gateway | IPv4 | no | The zones' default route. |
control_network.resolvers | IPv4 array | yes | At least one. |
storage_network | table | no | Same keys as control_network. Its nic_tag differs from the control one. gateway and resolvers must be omitted. |
node | array of tables | yes | At least 3 entries. |
node.node_id | integer | yes | Non-zero, unique, strictly increasing. |
node.rack | string | yes | Unique per node. A-Z, a-z, 0-9, ., _, -. |
node.cn_uuid | UUID | yes | Unique per node. |
node.control_ip | IPv4 | yes | Unique per node. |
node.peer_ip | IPv4 | yes | Unique per node. Equals control_ip without a storage_network. Differs from it with one, and the two address sets do not overlap. |
The admin and peer port is always 7101. It is not a plan key.
tritonadm storage deploy plan
Validates the plan file locally and prints it with its revision. The revision
is the SHA-256 of the parsed plan serialized back to TOML, so comments and
formatting do not change it. It sends no request. Honors -o json.
tritonadm storage deploy apply
Checks that every cn_uuid is registered, approved, and has an agent
heartbeat in the last 2 minutes, then calls POST /v1/storage/deployments. It
returns as soon as the request is accepted, and prints one line by outcome:
| Outcome | Output |
|---|---|
| New deployment | Mantad deployment <NAME> creation started (id <ID>, state: <STATE>) |
| Same file as the stored plan | Mantad deployment <NAME> already matches the committed TOML plan (id <ID>, state: <STATE>) |
Only s3_endpoint changed | Mantad deployment <NAME> S3 endpoint updated to <URL> (id <ID>) |
require_fdb_ha changed to true | Mantad deployment <NAME> FDB HA policy upgraded to required (id <ID>) |
| Scale-up or scale-down | Mantad deployment <NAME> TOML scale-up started (operation <OP>, effective <N>/RF=<RF> -> target <N>/RF=<RF>, phase=<PHASE>), or ... is already in progress ..., or ... resumed ... |
A new deployment needs an available, data-healthy FoundationDB cluster with
coordinator quorum. Applying the same file to a failed deployment runs the
apply again.
Changing a deployment
The control plane accepts these differences from the stored plan. Any other
change returns 409, Mantad deployment `<NAME>` already exists with a different plan revision: ....
| Change | Rules |
|---|---|
| Scale up | Add nodes only, with contiguous node_id values. replication cannot decrease. Existing members are never removed if the operation unwinds. |
| Scale down | Remove exactly one node. The target replication is odd and at least 3. s3_endpoint cannot point at the retiring member. The member is drained and evacuated before it leaves. |
s3_endpoint only | An https endpoint cannot change to http. |
require_fdb_ha from false to true |
A topology change cannot also change the image, the networks, the region, or the FoundationDB policy.
tritonadm storage deploy export
Calls GET /v1/storage/deployments and writes the deployment as a version 2
plan, to --file or standard output. For a deployment at replication factor 1
it also prints, on standard error:
WARNING: exported RF1 Mantad deployment cannot be reapplied until edited to at least three members and RF3
tritonadm storage deploy status
Calls GET /v1/storage/deployments and prints one deployment. Text only.
<NAME> (id <ID>): plan-v=<N> desired=<STATE> effective-members=<N> effective-rf=<RF> observed-healthy=<H>/<N> S3 VIP=<URL>[; reason=<REASON>]
FoundationDB high-availability lines, the control and storage network, and any reconfiguration in progress follow. Then one line per node:
node=<ID> rack=<RACK> cn=<CN_UUID> control=<IP> peer=<IP>:7101 role=<ROLE> desired=<STATE> zone=<ZONE_UUID> zone-state=<STATE> provision-job=<JOB> provision-state=<STATE> membership=<STATE> health=<HEALTH>
tritond probes each member's admin port with a 3-second timeout.
| Field | Values |
|---|---|
desired | planned, provisioning, active, reconfiguring, reconfiguration_failed, failed, replacing, updating, update_failed, destroyed, unknown |
health | healthy, degraded, unreachable, stale, missing, planned, provisioning, retiring, destroyed, failed, unknown |
With an http endpoint it ends with WARNING: S3 endpoint is explicit insecure HTTP.
An unknown name fails with no Mantad deployment named `<NAME>` .
tritonadm storage deploy replace
Replaces one member of an active deployment with the entry for --node-id in
the plan file. The new entry's CN, control IP, and peer IP differ from every
live member. The rack can repeat. The control plane adds the new member before
it drains and removes the old one. Calls
POST /v1/storage/deployments/{id}/replace and prints
Mantad replacement accepted for ... (state: <STATE>).
tritonadm storage cluster list
Calls GET /v1/storage/clusters.
| Column set | Columns |
|---|---|
| Default | shortid, name, surface, status, region, endpoint, seen |
| Long | id, name, display_name, surface, status, region, endpoint, s3_endpoint, last_observed_at, created_at |
Sorted by name. endpoint is the admin URL, port 7101. status is the last
persisted probe result.
tritonadm storage cluster show
Calls GET /v1/storage/clusters/{id} and prints the long fields as
key value lines.
tritonadm storage cluster add
Registers an existing cluster. Calls POST /v1/storage/clusters and prints
Registered storage cluster ....
| Option | Default | Description |
|---|---|---|
--name | required | Unique name. |
--cluster-endpoint | required | Admin API base URL. |
--admin-token, --admin-token-stdin | one required | Bearer token tritond presents to the admin API. The two conflict. |
--surface | s3 | s3, fs, or block. |
--default-region | us-east-1 | Region reported to clients. |
--display-name | none | Label. |
tritonadm storage deploy registers its clusters itself, so add is for a
cluster deployed some other way.
tritonadm storage cluster bootstrap
Reads a JSON document with name, admin_endpoint, s3_endpoint,
admin_token, region, bucket, access_key_id, and secret_access_key,
plus optional anchors_pem and resolve_pin. It registers the cluster,
refuses to continue unless a health probe returns healthy, creates the
bucket, makes it publicly readable, and sets it as the cluster's image object
store. A different existing image object store is refused. The document holds
secrets.
tritonadm storage cluster delete
Deregisters a cluster. Calls DELETE /v1/storage/clusters/{id}. The command
looks the cluster up first, so it fails when the cluster does not exist, though
the route itself treats a missing cluster as success. Asks Would you like to continue? unless -y.
-n prints the plan and changes nothing. It removes the registration only,
not any zone.
tritonadm storage cluster health
Calls POST /v1/storage/clusters/{id}/health, stores the result, and prints:
Probe complete: <NAME> → <STATUS> (observed <TIMESTAMP>)
<STATUS> is healthy when every member is alive, unreachable when none is
or the probe fails, and degraded otherwise.
tritonadm storage cluster set-replication
Changes the replication factor on every member. Calls
PUT /v1/storage/clusters/{id}/replication. The factor is 1 or an odd number
no larger than the member count, and every member must be alive. The control
plane fences writes, sets and verifies the factor on each member, then reopens
writes. A failure while setting or verifying the factor leaves every member
fenced. A failure while reopening leaves the members it could not reopen
fenced and the rest open, and the error names the fenced members. It prints:
storage cluster <CLUSTER>: RF <OLD> -> <NEW> across <N> members; writes reopened
wait for mantas3_replication_underreplicated_gauge to reach 0 on a subsequent repair pass before testing member loss
It does not change the deployment record, so deploy status keeps showing the
previous effective-rf.
tritonadm storage cluster set-presigner, clear-presigner
set-presigner stores the access key tritond signs presigned URLs with, and
optionally the S3 endpoint. Calls POST /v1/storage/clusters/{id}/presigner.
--access-key-id is required, and one of --secret-access-key and
--secret-access-key-stdin. clear-presigner sends empty values to the same
route.
Clusters from setup converge
With datacenter.manta.enabled in the answers document, tritonadm setup converge creates a deployment named by datacenter.manta.cluster_name, with
one member at replication factor 1. It then grows the cluster to the size that
fault_tolerance asks for:
fault_tolerance | Members | Replication factor |
|---|---|---|
none | 1 | 1 |
one-node (default) | 3 | 3 |
two-node | 5 | 5 |
Growth adds one member per converge pass, then raises the replication factor,
and acts only while a standing approval is open:
tritonadm fdb approvals standing open --operation manta-grow. Without one, the
cluster stays below policy and converge reports the step as satisfied. Members
are placed on nodes with eligibility set to storage, or exactly where
datacenter.manta.topology.members names them.
The genesis endpoint is https://s3.<DOMAIN>:7443 when the answers configure
ACME on the DNS zone, and http://<CONTROL_IP>:7443 otherwise.
Related commands
| Command | Effect |
|---|---|
tritonadm update mantad | Rolls each member of a managed deployment to a new image, one at a time. Prompts unless --yes. Gives up after 60 minutes. See Updates and channels. |
tritonadm setup converge | See Clusters from setup converge and tritonadm setup. |
tritonadm tenant init-storage <SILO> <TENANT> | Binds a tenant to a workspace on the default cluster, storage.default_s3_cluster_id. |
tritonadm tenant drop-storage <SILO> <TENANT> | Removes the binding. Refused while the workspace has buckets. |
See also
- Deploy object storage
- Object storage in the architecture overview