Deploy object storage

This guide places a multi-node mantad object storage cluster on declared compute nodes (CNs) with tritonadm storage deploy, confirms it is healthy, and makes it the cluster that tenants are bound to. For where mantad fits in the control plane, see Object storage in the architecture overview.

This page uses a plan file that names every member, its CN, its rack, and its addresses. An answers document with datacenter.manta.enabled builds a cluster through tritonadm setup converge instead. See Clusters from setup converge.

Before you start, you need:

• At least three CNs in the approved state, each with an agent heartbeat in the last 2 minutes, each in a different rack.
• A FoundationDB cluster that is available, data-healthy, and has coordinator quorum. mantad stores its metadata there.
• One free address per member on the control network, and one more per member on a separate storage network if you use one.
• The UUID of the triton-platform image your control-plane zones run.
• For an https S3 endpoint, a DNS zone with ACME configured in the control plane. Each member gets its certificate only when such a zone exists. Without a certificate, an https member never opens its S3 listener.
• A DNS name and a network route by which tenants reach port 7443 on the members. The control plane publishes s3.<DOMAIN> only as an internal record that points at each member's control address.

Step 1: Write the plan file

On the founder, in the global zone, as root, create the plan file. This example places three members on one shared network:

mantad.toml

version = 2

[cluster]
name = "<CLUSTER_NAME>"
region = "us-east-1"
image_uuid = "<PLATFORM_IMAGE_UUID>"
replication = 3
s3_endpoint = "https://s3.<DOMAIN>:7443"
require_fdb_ha = false

[control_network]
nic_tag = "admin"
netmask = "<NETMASK>"
gateway = "<GATEWAY_IP>"
resolvers = ["<RESOLVER_IP>"]

[[node]]
node_id = 1
rack = "r1"
cn_uuid = "<CN_UUID_1>"
control_ip = "<CONTROL_IP_1>"
peer_ip = "<CONTROL_IP_1>"

[[node]]
node_id = 2
rack = "r2"
cn_uuid = "<CN_UUID_2>"
control_ip = "<CONTROL_IP_2>"
peer_ip = "<CONTROL_IP_2>"

[[node]]
node_id = 3
rack = "r3"
cn_uuid = "<CN_UUID_3>"
control_ip = "<CONTROL_IP_3>"
peer_ip = "<CONTROL_IP_3>"

With one network, each node's peer_ip equals its control_ip. To carry replication traffic on its own network, add a [storage_network] table with a different nic_tag and no gateway or resolvers, and give each node a peer_ip on it.

replication is odd, at least 3, and no more than the number of nodes. s3_endpoint is the URL tenants use. It must be https. Every field and rule is in the plan file reference.

Step 2: Validate the plan

On the founder, in the global zone, as root:

tritonadm storage deploy plan --file mantad.toml

This checks the file locally and changes nothing. It prints the plan and its revision, the SHA-256 of the parsed plan written back out as TOML, so comments and formatting do not change it:

Mantad deployment plan: <CLUSTER_NAME>
  version:  2
  revision: <REVISION>
  RF:       3
  image:    <PLATFORM_IMAGE_UUID>
  S3 VIP:   https://s3.<DOMAIN>:7443
  FDB HA:   false
  control:  nic-tag=admin vlan=0 netmask=<NETMASK> gateway=<GATEWAY_IP> resolvers=<RESOLVER_IP>
  storage:  shared with control network
  node 1 rack=r1 cn=<CN_UUID_1> control=<CONTROL_IP_1> peer=<CONTROL_IP_1>
...

A rule violation stops here with a message that names the field, for example node.rack values must be unique fault domains.

Step 3: Apply the plan

On the founder, in the global zone, as root:

tritonadm storage deploy apply --file mantad.toml

The command checks that every CN is registered, approved, and has a recent heartbeat, then submits the plan and returns:

Mantad deployment <CLUSTER_NAME> creation started (id <DEPLOYMENT_UUID>, state: planned)

The control plane then provisions one zone per node, named triton-mantad0, triton-mantad1, and so on, waits for every member to join, and registers the cluster under <CLUSTER_NAME>. Each wait gives up after 600 seconds, which leaves the deployment failed. Applying the same file again retries a failed deployment.

Step 4: Watch the deployment

On the founder, in the global zone, as root:

tritonadm storage deploy status <CLUSTER_NAME>

The first line reads desired=active and observed-healthy=3/3 when the cluster is up:

<CLUSTER_NAME> (id <DEPLOYMENT_UUID>): plan-v=2 desired=active effective-members=3 effective-rf=3 observed-healthy=3/3 S3 VIP=https://s3.<DOMAIN>:7443
...
  node=1 rack=r1 cn=<CN_UUID_1> control=<CONTROL_IP_1> peer=<CONTROL_IP_1>:7101 role=... desired=... zone=<ZONE_UUID> zone-state=... provision-job=... provision-state=... membership=... health=healthy
...

A node's health is healthy, degraded, unreachable, stale, missing, planned, provisioning, retiring, destroyed, failed, or unknown. A deployment that ends in failed carries a reason= at the end of the first line.

Step 5: Probe the registered cluster

On the founder, in the global zone, as root:

tritonadm storage cluster health <CLUSTER_NAME>

Probe complete: <CLUSTER_NAME> → healthy (observed <TIMESTAMP>)

The result is healthy when every member is alive, degraded when some are, and unreachable when none are.

Step 6: Make it the default cluster for tenants

An operator-created tenant is bound to a storage workspace only when a default cluster is set. On the founder, in the global zone, as root, find the cluster's UUID:

tritonadm storage cluster show <CLUSTER_NAME>

On the founder, in the global zone, as root, set it as the default:

tritonadm config set storage.default_s3_cluster_id <CLUSTER_UUID>

Tenants created before this have no workspace, and tenants created through self-signup never get one, even with a default set. Until a tenant is bound, its storage calls return 412 TenantStorageUnbound. On the founder, in the global zone, as root, bind each one:

tritonadm tenant init-storage <SILO> <TENANT>

A new workspace gets a quota of storage.default_workspace_quota_bytes, 100 GiB by default. Binding a tenant that already has a workspace returns 409.

Step 7: Issue the tenant a key

tritonadm has no command that creates a storage user or a plain access key. Use the operator console's storage cluster page, or call the tritond admin API with a root operator credential or one that holds the storage-admin capability. Tenant credentials are refused there.

Create a storage user in the tenant's workspace:

POST /v1/silos/<SILO_ID>/tenants/<TENANT_ID>/storage/users
{"name": "<STORAGE_USER>"}

Mint an access key for that user. The request has no body:

POST /v1/silos/<SILO_ID>/tenants/<TENANT_ID>/storage/users/<STORAGE_USER>/access-keys

The 201 response carries access_key_id and secret_access_key. The secret appears in this response only and cannot be read back. To revoke the key:

DELETE /v1/silos/<SILO_ID>/tenants/<TENANT_ID>/storage/users/<STORAGE_USER>/access-keys/<ACCESS_KEY_ID>

Give the tenant four values: the access key ID, the secret access key, the S3 endpoint from your plan's s3_endpoint, and the plan's region. tritonctl and the portal do not show the endpoint or region to tenants.

Step 8: Check S3 from the tenant network

From a host on the network your tenants use, with the AWS CLI configured for the key from step 7 as the tenant guide "Store objects with an S3 client" describes, list the buckets:

aws s3 ls --endpoint-url https://s3.<DOMAIN>:7443

No output and no error means the name resolves, the route works, the certificate is valid, and the key signs. On an https endpoint, a connection that is accepted but never answered means the member has no certificate yet: it binds the port and waits for its certificate before it serves.

If the deployment fails

Message	Cause
selected CN <CN_UUID> is not registered, is not approved, or has no active agent heartbeat	A CN in the plan is not ready. Check it with tritonadm cn list.
cluster.replication must be odd so read and write quorums cannot split evenly	replication is even.
Mantad deployment requires at least three [[node]] entries	Fewer than three nodes.
cluster.s3_endpoint must use HTTPS unless allow_insecure_s3_endpoint = true and ...	The endpoint is http.
A storage deployment requires an available, data-healthy FoundationDB cluster with coordinator quorum	FoundationDB is not healthy.
Mantad deployment `<CLUSTER_NAME>` already exists with a different plan revision: ...	The change in the file is not one the control plane can apply. See Changing a deployment.
desired=failed with a reason=	A zone provision or the membership wait failed. Fix the cause and run storage deploy apply again with the same file.

Next steps

• Every tritonadm storage command, the plan file, and how to scale or replace members: tritonadm storage.
• Where mantad keeps its metadata and how it fits the control plane: Architecture.