Garage S3¶

Garage is a self-hosted, S3-compatible object store. It runs outside the Kubernetes cluster, in its own LXC container on the proxmox-01 hypervisor, so it can serve as a backup target that does not depend on the cluster it backs up.

Architecture¶

flowchart LR
    subgraph PVE[proxmox-01 hypervisor]
        subgraph CT[LXC 210 garage-01]
            CADDY[Caddy\n:443 TLS] --> GARAGE[garage\nS3 API :3900]
            GARAGE --> META[/var/lib/garage/meta\nLMDB/]
            GARAGE --> DATA[/var/lib/garage/data\nblocks/]
        end
    end
    CLIENT[S3 clients\naws / rclone / volsync] --> CADDY
    META -.-> ZFS[(ZFS dataset\n500G mount point)]
    DATA -.-> ZFS

Why an LXC container rather than a VM: Garage is a single static binary with no kernel requirements, so a VM would only add a guest kernel and a virtual disk layer between it and the ZFS pool. The data volume is a plain ZFS dataset the host can snapshot and replicate directly.

Ownership split¶

Two layers own this service, and the boundary matters when changing it:

• terraform/proxmox/ct_garage.tf creates the container shell -- CPU/memory, the 16G rootfs, the 500G /var/lib/garage mount point, the NIC, and the injected SSH key. Nothing inside the guest.
• ansible/roles/garage owns everything inside the guest -- the binary, config, systemd unit, and the one-time cluster bootstrap.

Provisioning¶

Terraform creates the container:

AWS_PROFILE=wibrow-tf just tf::apply-proxmox

CI applies this automatically on merge to main (.github/workflows/terraform-proxmox.yaml), so in practice the container appears once the change lands.

Pin the DHCP lease with a UniFi reservation using the MAC:

terraform output garage_mac_address

pct set 210 -features nesting=1

Bootstrap¶

just ansible ping-garage     # SSH reachability
just ansible check-garage    # dry-run + diff
just ansible deploy-garage   # apply

The role is idempotent and does the following:

1. Installs the pinned, sha256-verified static musl binary to /usr/local/bin/garage.
2. Creates the garage system user and the meta/data directories under the mount point.
3. Templates /etc/garage.toml and the garage.service systemd unit, then enables and starts it.
4. On a fresh cluster only -- assigns this node a layout role and applies layout version 1.
5. Ensures the buckets in garage_buckets and keys in garage_keys exist, each key granted read/write/owner on its buckets.

Step 4 is guarded on the layout being empty, so re-runs never disturb an existing layout.

Secrets¶

rpc_secret and the admin API admin_token are SOPS-encrypted in ansible/roles/garage/vars/secrets.sops.yaml with the repo age key. Rotate them by editing that file with sops and re-deploying.

Using the bucket¶

Get the credentials from the host:

ssh root@garage-01.servers.local \
  'GARAGE_CONFIG_FILE=/etc/garage.toml garage key info --show-secret garage-admin'

Configure an aws CLI profile:

aws configure set aws_access_key_id     <ACCESS_KEY_ID>  --profile garage
aws configure set aws_secret_access_key <SECRET>         --profile garage
aws configure set region                garage           --profile garage
aws configure set s3.addressing_style   path             --profile garage

Read and write:

aws --profile garage --endpoint-url http://garage-01.servers.local:3900 \
  s3 cp ./file.txt s3://garage/

aws --profile garage --endpoint-url http://garage-01.servers.local:3900 \
  s3 ls s3://garage/

To drop the repeated flag, add an endpoint_url to the profile in ~/.aws/config (supported by AWS CLI v2):

[profile garage]
region = garage
endpoint_url = http://garage-01.servers.local:3900
s3 =
    addressing_style = path

Verify a write landed from the Garage side -- Objects should increment:

ssh root@garage-01.servers.local \
  'GARAGE_CONFIG_FILE=/etc/garage.toml garage bucket info garage'

TLS¶

Garage does not terminate TLS itself -- upstream expects a reverse proxy in front. The garage-tls role puts Caddy on :443 in front of the S3 API and serves https://s3.wibrow.dev.

just ansible check-garage-tls    # dry-run
just ansible deploy-garage-tls   # apply

Certificates come from certbot using the Let's Encrypt DNS-01 challenge, not Caddy's own ACME. Two reasons:

• s3.wibrow.dev resolves to an RFC1918 address, so HTTP-01 can never reach this host.
• Debian's packaged Caddy has no Cloudflare DNS module, and the custom-build download URL serves a rolling artifact -- pinning a checksum against it would break the role every time Caddy publishes a release.

certbot installs its own renewal timer; a deploy hook reloads Caddy after each renewal.

Plain HTTP on :3900 is intentionally left listening for in-VLAN clients that do not want TLS.

Prerequisites¶

Both the certificate challenge and the DNS record need a Cloudflare API token on the host, scoped to Zone → Read and Zone → DNS → Edit for wibrow.dev. Store it once, SOPS-encrypted, where the community.sops vars plugin picks it up for every host:

cat <<'EOF' > ansible/inventory/group_vars/all.sops.yaml
cloudflare_api_token: "<token>"
EOF
sops -e -i ansible/inventory/group_vars/all.sops.yaml

DNS record (automatic)¶

The cloudflare-ddns role (deployed by the same garage-tls playbook) keeps an unproxied s3.wibrow.dev A record pointed at the container's current VLAN-20 address. A systemd timer reconciles it every 5 minutes, so a changed DHCP lease self-heals -- no manual record and no UniFi reservation required. The record must be unproxied because it is an RFC1918 address: Cloudflare cannot proxy to it, and the endpoint is internal-only.

Records, interface, and interval are configurable in roles/cloudflare-ddns/defaults/main.yaml. The role is host-agnostic and can publish records for any DHCP host.

Using the TLS endpoint¶

aws --profile garage --endpoint-url https://s3.wibrow.dev s3 ls s3://garage/

Proxmox host certificate¶

The hypervisor's own web UI serves a self-signed certificate, which is why PROXMOX_VE_INSECURE = "true" exists in the root mise.toml. Proxmox VE has a built-in ACME client that can replace it, configured by the same proxmox role and the same Cloudflare token:

proxmox_acme_enabled: true
proxmox_acme_domain: proxmox-01.wibrow.dev

just ansible deploy-proxmox-01

It is opt-in (proxmox_acme_enabled: false by default) so it never fires unexpectedly on an unrelated run. Let's Encrypt cannot issue for .servers.local, so the host needs a name in a real zone with an A record pointing at its VLAN-20 address. PVE's daily timer renews it.

Operations¶

Every management command needs the config for its rpc_secret, so either export it or run as root where /etc/garage.toml is the default path:

export GARAGE_CONFIG_FILE=/etc/garage.toml

Health and layout¶

garage status         # nodes, capacity, version
garage layout show    # current layout and version
garage stats          # storage and request statistics

Managing buckets¶

garage bucket list
garage bucket create <name>
garage bucket info <name>
garage bucket delete <name>        # must be empty

Managing keys¶

garage key list
garage key create <name>
garage key info --show-secret <name>
garage bucket allow --read --write --owner <bucket> --key <name>
garage bucket deny  --read --write --owner <bucket> --key <name>

Prefer adding buckets and keys to garage_buckets / garage_keys in the role defaults rather than creating them by hand, so they are reproducible.

Upgrading¶

Bump garage_version and the matching entry in garage_checksums in ansible/roles/garage/defaults/main.yaml, then re-deploy. Get the checksum by downloading the release asset and hashing it:

curl -fsSL https://garagehq.deuxfleurs.fr/_releases/v<VERSION>/x86_64-unknown-linux-musl/garage \
  | shasum -a 256

Check the release notes for migration steps before crossing a major version.

Redundancy¶

A second replica on the Synology NAS is planned. Garage wants an odd number of nodes for metadata quorum, so a third node (or a lightweight tiebreaker) belongs in the plan before relying on it.

Raising replication_factor is not a config-only change -- the additional nodes must join and the layout must be recomputed first. When a second node joins, rpc_public_addr must also change from 127.0.0.1:3901 to the node's real VLAN-20 address so peers can dial it, and every node must share the same rpc_secret.

Troubleshooting¶

Service will not start

systemctl status garage
journalctl -u garage -n 100 --no-pager

Permission denied (publickey) when deploying

The container's root authorized_keys holds whichever key ran the Terraform apply. Because CI runs on the self-hosted runner, that is the runner's key, not yours. var.ssh_public_keys is supplied from the age-encrypted TF_VAR_ssh_public_keys in the root mise.toml, but ct_garage.tf ignores changes to initialization[0].user_account, so changing it does not re-key an existing container. Add the key directly instead:

pct exec 210 -- bash -c 'install -d -m700 /root/.ssh && \
  echo "<your-public-key>" >> /root/.ssh/authorized_keys'

Do not "fix" this by recreating the container -- a fresh create re-sends the full feature map and hits the same root@pam 403 described above.

Forbidden: Garage does not support anonymous access yet

This is the healthy response to an unauthenticated request; it confirms the API is up. Reaching it with curl is a quick liveness check:

curl -s http://garage-01.servers.local:3900/garage

Client errors mentioning a bucket-prefixed hostname

The client is using vhost-style addressing. Set addressing_style = path.

Reference¶

• Garage documentation
• terraform/proxmox/ct_garage.tf -- container definition
• ansible/roles/garage/ -- install and bootstrap
• ansible/README.md -- role usage