High-frequency trading platform

IT.114 Capital markets

A multi-venue automated trading platform built for low-latency execution across markets, handling real-time data feeds, order routing and risk controls.

Ten milliseconds, end to end

Every trading strategy is a bet that you can see something in the market and act on it before it goes away. The client had strategies that worked on paper. What they needed was the machinery to run them live across several venues, and that machinery had to be quick. The target we agreed on was a P99en.wikipedia.orgThe value below which a given share of measurements fall. P99 is the slowest one in a hundred. of 10 milliseconds (99 out of every 100 orders leave for the venue within 10 ms of the market event that triggered them). Under it we set a P50 of 2 ms and a P95 of 5 ms, so a typical trade would sit well inside the budget and only real bursts would push towards the limit.

Two more things had to hold. The system had to be fully automated, and it had to show its work. The client wanted a dashboard covering everything from the overall health of the platform down to a single trade, the decision behind it and the order book changes that led up to it.

Drawing the funnel

We start every build with a definition day, sitting with the client until the system is clear enough to draw. The drawing that came out of this one was a funnel. Millions of order book events a second come in at the top. Signals boil them down to a handful of conditions. Strategies combine the signals into decisions. Risk checks throw out the decisions that should not be acted on, and what remains, a few trades, goes to execution. Each layer sees less data than the one before it, and each may add as little delay as possible.

The layer the client cared about most was risk. A strategy that is profitable on paper stays profitable only if it survives the days when things go wrong, and in live markets they go wrong often. Venues stop answering, orders fill halfway, connections drop. We built the handling of those cases into the core from the start.

Linux as the framework

For the first version we kept the stack as thin as we could. There is no trading framework underneath the platform. Each layer is a small program with a structured input and a structured output, and Linux does the rest. systemdsystemd.ioThe Linux service manager. It starts processes, restarts them when they die and orders them at boot. starts and supervises the processes, journaldman7.orgThe system log that comes with systemd. Every service writes to it without any logging code of its own. keeps the logs, and the kernel switches between processes in microseconds. Order book data goes into Valkeyvalkey.ioAn open source in-memory data store, forked from Redis and kept by the Linux Foundation. (an open source fork of Redisredis.ioThe in-memory key-value store that Valkey grew from.) on the same machine over a Unix socketen.wikipedia.orgA connection between two processes on the same machine that skips the network stack entirely., so a read or write costs microseconds instead of the milliseconds a network hop would add.

The live data lives in memory. A separate pipeline streams the order book history to disk continuously for storage and historical analysis, tuned so that it barely touches the latency of the trade path.

The payoff of this structure was how easy it made the client's part. A new signal or strategy is one more small program with a clear input and output, testable on its own and wired into a pipeline by configuration.

Where the first version fell short

The first version took in hundreds of thousands of events a second without trouble. It missed the latency target anyway. Every layer wrote its intermediate results into Valkey, and under a burst of market activity those writes stacked up until the CPU spiked and the tail went past 10 ms. P50 was fine. P99 was not, and P99 was the number that mattered.

So we kept the architecture and rewrote the hot path. Order book ingestion and the decision loop moved to Rustrust-lang.orgA compiled systems language: the speed of C with memory safety checked at compile time., still talking to Valkey, still supervised by systemd, still one small program per layer. Signals became gates. A strategy moves on to the risk and execution checks only when every one of its gates is open, which cut the work done per event down to what is actually needed. The second version hit the P99 target and has stayed there.

What the client sees

Every step in the pipeline is timed and the timings are stored per trade. The dashboard starts at the level of the whole system and the profit of each strategy, and drills down to a single trade, when each of its gates opened, how long each check took, and the order book in the seconds before it. When a trade looks odd, the answer is usually a few clicks away.

Months in, one machine

The whole platform runs on one dedicated server. A tuned kernel, critical processes pinned to their own cores, a few systemd services. Deploying is a script that packages the code, ships it and restarts the services in the right order, and a fresh server goes from empty to trading in a few minutes with one command.

It has been in production for months with very little intervention, handling hundreds of thousands of events a second and thousands of trades a day across several independently configured strategies. The running cost is essentially the server and where it sits. The client's time now goes into strategies.

The problems that were not in the spec

Most of the budget was spent outside our own code. Network latency was the biggest item. A venue's own data centre is the one place you can be certain of the round trip (colocationen.wikipedia.orgRenting rack space in someone else's data centre, in trading, the one next to the exchange. is a business of its own for that reason), and for the external data sources we ended up mapping which internet backboneen.wikipedia.orgThe long-haul links between the big networks. Which ones a packet takes decides how long it travels. route actually carried the packets fastest to the machine, which was rarely the shortest one on a map. TLSen.wikipedia.orgThe encryption under HTTPS and most secure connections. came next. A fresh handshakeen.wikipedia.orgThe round trips two sides need to agree keys before the first encrypted byte can be sent. costs more than the whole trade budget, so connections are kept warm and reconnects are handled well before the old link goes stale.

The kernel took its share too. Where it schedules a processdocs.kernel.orgHow the Linux kernel decides which process runs on which core, and when., how it hands a packet from the network card to userspace, when it decides to interrupt a core, all of it shows up on the latency graph once you are measuring in microseconds. We pinned the critical processes to isolated coresdocs.kernel.orgThe kernel's boot options, isolcpus among them, which keeps the scheduler off a set of cores., moved the interruptsdocs.kernel.orgChoosing which cores handle hardware interrupts, so the busy ones are left alone. away from them and tuned the scheduler so that the trade path is never waiting on something unrelated.

Then there was the market data itself. Venues publish order booksen.wikipedia.orgThe list of open buy and sell orders on a venue, by price. Every change to it is an event. at different depths, L1 (the best bid and ask), L2 (aggregated volume per price level) and L3 (every individual order), each with its own format, its own sequencing and its own way of telling you that you missed an update. Building one book that stays correct across venues meant getting snapshots and deltas, gap detection and resynchronisation right for each of them, because a book that is quietly wrong is worse than one that is late.

These are computing problems more than trading problems, and a project like this reaches almost every layer of the machine. A few of them deserve articles of their own, and they are on the list.