Storage for small clusters: Longhorn, Ceph and the honest trade-offs

The moment a cluster runs anything stateful — Postgres, a message queue, a filestore for Odoo — "just use the default StorageClass" stops being an answer. Every CSI driver makes different trade-offs on replication, failure domains and operational weight, and the difference only shows up the day a node dies. We've run both Longhorn and Ceph (via Rook) in production. Here's the honest version of when each one earns its keep.

What a StorageClass is actually promising you

A PersistentVolumeClaim is a request; the StorageClass behind it is the promise about what happens when the node holding your data disappears:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: longhorn-3replica
provisioner: driver.longhorn.io
parameters:
  numberOfReplicas: "3"
  staleReplicaTimeout: "30"
reclaimPolicy: Retain
allowVolumeExpansion: true

That's the whole interface applications see. What differs — wildly — is the engine underneath honoring it: how replicas are placed, how a rebuild happens after a node reboots, and what "degraded" costs you in latency while it heals.

Longhorn: simplicity that scales down, not just up

Longhorn replicates block volumes across nodes using a lightweight, mostly-userspace engine — no separate storage cluster to run, no dedicated OSD nodes, no admin fluent in Ceph's CRUSH maps. Install the Helm chart, label a few disks, and you have replicated block storage in under fifteen minutes.

apiVersion: longhorn.io/v1beta2
kind: Volume
metadata:
  name: erp-filestore
spec:
  numberOfReplicas: 3
  dataLocality: best-effort
  staleReplicaTimeout: 30

dataLocality: best-effort is the setting that matters for small clusters: Longhorn tries to keep one replica on the same node as the workload consuming it, so a healthy pod reads from local disk instead of hopping the network for every I/O. On a three-to-six node cluster that single knob is most of the latency win over a "real" distributed store, at a fraction of the operational cost.

Where it shows its size: Longhorn's replication is per-volume, not erasure-coded, so three replicas means three times the disk for every volume — fine at the scale we're describing, expensive once you're managing tens of terabytes. And its control plane (the longhorn-manager pods, engine images per node) is simple by comparison to Ceph, but it is still a piece of infrastructure with its own upgrade cadence and its own failure modes to page on. Attach/detach latency after a pod reschedule is the number we watch closest — it's the thing that decides how fast a CloudNativePG failover actually completes end to end, since the operator can only promote as fast as the disk underneath it reattaches.

Ceph: complexity that buys you scale and features

Rook-operated Ceph is a genuine distributed storage cluster living inside your Kubernetes cluster: OSDs (object storage daemons) per disk, a monitor quorum for cluster state, a manager for metrics and orchestration. In exchange for that weight, you get block (RBD), file (CephFS) and object (RGW, S3-compatible) storage from one system, erasure coding that beats 3x replication on raw capacity, and a cluster that keeps serving reads and writes through multiple simultaneous node failures without falling over.

apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
  name: rook-ceph
spec:
  cephVersion:
    image: quay.io/ceph/ceph:v18
  storage:
    useAllNodes: false
    nodes:
      - name: "worker-1"
        devices: [{ name: "sdb" }]
      - name: "worker-2"
        devices: [{ name: "sdb" }]
      - name: "worker-3"
        devices: [{ name: "sdb" }]
  mon:
    count: 3

That mon: count: 3 line is where the honesty comes in: Ceph wants a real quorum, which means a real minimum footprint — in practice we don't run Rook under six nodes. Below that, you're spending more compute on OSDs and monitors than on the workloads they're storing data for, and a single-digit-node outage can take out enough monitors to threaten quorum. Ceph also asks more of whoever's on call: PG (placement group) rebalancing after you add or remove a disk is a real operational event, not a background non-issue, and diagnosing "why is this OSD flapping" requires actually knowing Ceph, not just Kubernetes.

Where we land per project

• Under six nodes, one or two stateful workloads (Postgres via CNPG, an Odoo filestore, a Redis): Longhorn. The operational surface matches the team size, and dataLocality: best-effort gets you most of the latency benefit distributed storage would offer anyway.
• Six-plus nodes, multiple storage protocols needed (block for databases, object for backups/media, file for shared config), or capacity in the tens of terabytes: Rook/Ceph. Erasure coding and the multi-failure tolerance start paying for the extra monitors and OSD management.
• Anything smaller than three nodes total: neither. Use your cloud provider's managed block storage or a managed database, and spend the engineering time elsewhere. Storage replication inside Kubernetes is a tool for when you're already committed to running the platform yourself — not a reason to start.

The test that actually matters

Whichever engine you pick, the question that decides if it was the right call isn't a benchmark number — it's what happens when you kubectl drain a node with a stateful pod on it during business hours. Time the reschedule, time the volume reattach, watch whether the application noticed. We run that drill on every cluster before it goes anywhere near production, the same discipline we apply to restoring backups instead of just taking them — a replication model you haven't tested against a real node failure is still a guess, no matter how good the CRUSH map or the Helm chart looked on paper.

If you're weighing this trade-off for a cluster you're about to stand up, it's exactly the kind of call we make as part of every cloud hosting engagement — sized to the cluster you actually have, not the one a whitepaper assumes.