Distributed protocol testing: ditch containers, use state machines

Core insight. Distributed protocols are deceptively simple in pseudocode but notoriously hard to test in practice. Network failures, message reordering, timeouts, and node outages create a combinatorial explosion of states that integration tests struggle to cover. Standing up thousands of containers is slow, expensive, and non-reproducible. The speaker's team at Fly.io faced this while modifying the broadcast protocol for Corrosion, their SQLite replication system.

The solution. The team extracted the protocol's core logic into a pure state machine that never touches sockets, timers, or clocks. Instead of spawning threads or setting timers, the protocol receives events and returns an outbox of actions—send this message, schedule this timer, deliver this payload—that the caller must execute. The caller provides a trait or interface that implements all side effects. This pattern is known in the industry as "sans I/O."

Testing payoff. With the protocol isolated, the test harness simulates clusters of thousands of nodes in a single thread. Events (messages, timers, node failures) are queued in a binary heap and processed in virtual time—the loop jumps to the next smallest timestamp rather than waiting. Failures are injected as events: take a node down, drop its messages, bring it back. The test becomes deterministic, fast, and fully reproducible using a seeded random generator.

Real results. The team validated modifications to the Plumb Tree gossip protocol before production deployment. They tested eager peer selection strategies, measured message delivery latency across the cluster, and confirmed behavior under various failure scenarios. The test caught edge cases that would have been impossible to reproduce in integration tests. All of this ran locally on a developer machine without containers or network dependencies.

The trade-off. This approach does not eliminate integration risk. The protocol logic is verified, but the glue code that connects it to real I/O still needs testing. If the caller blocks on a send or mishandles a timer, the system breaks. The pattern also shifts complexity to the library user, who must implement the I/O trait. For many teams, this is a worthwhile trade-off against flaky, slow, and unrepeatable distributed system tests.

Bottom line for teams. Separating protocol logic from I/O is a well-known pattern that remains underused in distributed systems engineering. It delivers deterministic tests, rapid iteration, and production confidence without expensive infrastructure. Teams building or modifying consensus, gossip, or replication protocols should consider this architectural split before investing in complex integration test suites.