Technologies

DRBD: Replication That Holds When a Server Dies

DRBD audit, tuning, migration and operations by the team whose founder maintained its Debian packages from 2004 to 2007. Failovers measured, not assumed.

A DRBD cluster looks healthy for years. Then a node dies at night, the failover stalls or the two copies diverge, and nobody on the team has ever done this for real. Orders are lost without a single error message.

We have run DRBD in production for clients since at least 2012, across more than 80 client projects. Our founder, an official Debian developer, maintained its Debian packages from 2004 to 2007, brought them into their first stable Debian release, then successfully handed them over to a new maintainer. We hit the failures, then we crashed servers on purpose to measure them.

What We Measured

Crash tests on disposable nodes of our production cluster, with live MySQL, PostgreSQL and file writes, DRBD 8.4.11, asynchronous replication:

ScenarioDowntimeData lostCorruption
Kernel panic, server reboots86 sNoneNone
Kernel panic, server stays down, forced failover47 sNoneNone
Replication link cut, split-brain, forced failoverNot measured55 MySQL rows, 50 PostgreSQL rows, 52 filesNone

What this taught us:

  • Six kernel panics under load, zero corruption. The real risk is silent loss: in a split-brain, writes your application saw as saved disappear, and the database stays perfectly healthy.
  • Synchronous replication would not have prevented it. With protocol C, writes kept flowing during the cut: 10,855 acknowledged writes lost on the forced failover.
  • The zero-loss runs were at a low write rate. A failover under heavy writes is still on our test list, and we say so.

Why We Ran Those Tests

In 2024 a network failure at a hosting provider triggered a failover and a split-brain on a client’s platform. MySQL would not stay up, and we restored that morning’s backup: a day of orders lost. So we built a lab to reproduce it. The data loss now reproduces every time, and our procedures changed because of it.

A payment software vendor saw its batches run three times slower, and replication dropped out for hours each day. The obvious fix was a bigger network link. We traced the chain instead: more than a terabyte a day of temporary database files, all replicated for nothing. We moved them to RAM in one evening. Batches went from 5 minutes to 1, replication stayed on, and the link upgrade was cancelled.

What We Do

  • Audit of your DRBD cluster: configuration actually applied, failover tested, gaps stated in writing. Most audits complete in one week
  • Tuning under real load: replication that keeps up with your writes without slowing them down
  • Migration of DRBD and Pacemaker clusters off end-of-life systems. We are moving a MariaDB cluster from CentOS to Debian right now
  • Diagnosis of stalls, stuck resyncs and split-brains, then a plain written post-mortem
  • Full operation of replicated servers, 24/7, through our Managed Disaster Recovery

One-off work runs as a DevOps mission: scoped and priced before we start.

Who It Is For

CTOs whose platform depends on DRBD, and who want it proven rather than assumed. Not for you if you look for a tutorial or the cheapest host.