Private IPFS Cluster on Wasabi S3

The problem

A disk-only IPFS node holding millions of pins used about 6–7TB locally. Scaling that across peers meant:

• High persistent-disk cost on cloud VMs
• Slow, expensive growth every time the pinset grew
• Harder disaster recovery — everything tied to a single volume
• Operational risk when disks filled and GC was not automatic

The solution

Custom Kubo images with the go-ds-s3 plugin, coordinated by IPFS Cluster. S3 nodes stay around ~100GB local plus ~160GB in Wasabi instead of another 6–7TB volume. Cluster metadata still syncs so every node knows the same pinset; blocks are fetched from S3 or peers on demand.

Architecture

Private IPFS Cluster
├── Disk node(s)     → full local repo (~TB scale)
└── S3 nodes         → Wasabi blocks + small local cache
        ├── LevelDB  → CIDs / datastore metadata
        └── Wasabi   → actual IPFS blocks

What I implemented

• Custom Kubo + go-ds-s3 Docker images — the S3 plugin needs two different JSON shapes (config vs datastore_spec); getting that wrong means the daemon never starts
• Multi-node IPFS Cluster with unique peer names, isolated Docker networks, and a shared cluster secret. Wasabi credentials bound to the right regional endpoint, not a generic S3 URL
• Private swarm: no public bootstrap, no reprovide noise, no public DHT
• StorageMax plus cron GC so the local cache does not fill the VM
• Ops scripts: pin-count compare, sync/recover, disk monitor, emergency cleanup
• Pin-tracker tuning for millions of pins (queue size and concurrent pins) so recoveries do not stall

Results

• Most replica storage moved from TB-class disks to Wasabi object storage — flat S3-compatible pricing, no AWS egress tax
• Cluster-level pinset in the millions without giving every node a 7TB volume
• Replicas across three Wasabi regions, plus disk nodes for the hot path
• Automated GC and disk alerts keep S3 nodes in a ~100–200GB local envelope
• Closed cluster: pins and swarm traffic stay inside the private network

Stack