Resurgamus Horizon · Beta
HA, failover and chaos testing on a multi-node cluster
Load, failover and chaos tests on a multi-node cluster with PostgreSQL/Patroni. The capacity benchmark documents 1,348 secret reads/s, p99 latency of 290 ms and no errors at that load level. Separate tests measure median application failover of 27.47 s over 20 cycles and convergence in 2.1 s after key rotation.
- Built the lab on Proxmox with OpenTofu and Ansible. Machines are destroyed and recreated with the same configuration between test campaigns.
- Four failure scenarios, each with a pass criterion written in advance: 5, 10 then 20 nodes joining at once, an abrupt kill of the primary, a network partition then the return of the isolated node, and a rolling restart of the whole cluster.