The problem
An open broker platform was melting its message queue in every price spike. Orders backed up, the UI froze, and the trading desk resorted to shouting prices over the phone while engineers raced to drain a backlog that kept refilling.
Fundamentally it was a throughput problem hiding behind an incident problem: the order path was a single fragile chain, there was no idempotency, and every replayed message became a duplicate order.
What we did
We rebuilt the order path on Go + Kafka with an idempotent gateway at the edge — every order carries a client-generated ID, and dedupe happens on the wire, not in a retry loop. Market data, order validation and routing became separate lanes so a spike in one feed could never stall the others.
Deploys moved behind a canary pipeline: one pod at a time, smoke-tested against synthetic order flow, with automatic rollback on any p99 regression. Post-trade analytics landed in ClickHouse so the desk stopped polling the database for reports.
Impact
The system went from "urgent incident" to "boring Monday." Peak volumes during the next big market move were handled without a single dropped order, and the team that used to babysit the queue went back to building features.