Diary of a SysAdmin

What would happen to low latency trading if exchanges moved to the cloud?

A thought experiment on the winners, losers and new players in a cloud-driven stock, fx and commodity-trading world

Cisco Nexus switch with empty switchports

A small UK stock exchange called Aquis hosts their matching engine in the cloud. Crypto exchanges are all cloud-based too. What if the NYSE, CME and others followed? The NASDAQ is already thinking about this since 2021. Let’s evaluate how HFTs would adapt, and how small-time traders could benefit.

A brief intro to ULL (ultra low latency) trading infra

ULL trading firms go to a lot of trouble to get their servers and switches within the same buildings as the exchanges they trade with to reduce latency. Some firms don’t even use layer 1 switches to be competitive.

Most major exchanges run out of public data centres, for example:

  • The NASDAQ is in an Equinix facility called NY11 in Carteret, NJ
  • The CME is in a site called Aurora
  • The LME sits in a Digital Realty facility at Brick Lane (London), and so on

The NYSE runs out of a facility in Mahwah, NJ, and whilst not a public colo, you can still rent a rack there.

You’ll pay heavily for a rack in one of these data centres. Individual quant traders cannot simply deploy ULL strategies here, due to the difficulty of sourcing the right hardware and getting set up in a site. You could go to a provider like Pico, but good infra is half of the competitive advantage when trading ULL (the other half is writing the trading strategy in low-level code and deploying it before other HFTs).

It’s not unthinkable that these facilities migrate to the cloud (which is where Coinbase’s trading happens).

Can I trade stocks at low latency myself?

  1. Reach out to Equinix, get a rack in NY11 with 5kVA power and about 4 cross connects
  2. Reach out to Arista and get a 7130 low latency switch
  3. Buy a Xilinx card that supports Onload, put it in a server, do this x2 for reliability
  4. Buy a time server to handle PTP so you know exactly when the market opens
  5. Become a member of the NASDAQ
  6. Pay for market data (hitting you in raw binary OUCH format)
  7. Get an Internet connection so you can connect your server to deploy strats

You’re looking at six to seven figures of set up costs and about that in yearly running costs. In other words, it’s not happening. Similar pains will exist for trading FX with some bank like Citi, or futures with the CME. You’ll also want to become a member on other exchanges and dark pools if you want to do arbitrage, so repeat this for all of their DCs too.

Low(ish) latency crypto

Smart quant crypto traders will spin up many thousands of EC2 instances, measure latency to the exchange, and only keep the best instances, continually. This process allows them to eventually get EC2 instances to trade from probably in the same rack, or maybe on the same bare metal box, as the exchange. This is pretty good going. Traffic will still be routed through some sort of overlay network, but everyone has that penalty. There are no large upfront costs to setting up a strategy and market data is free.

So, what happens if exchanges move?

The NASDAQ is already bringing AWS infra into their facilities (link). So let’s say the NYSE went full cloud tomorrow with AWS – what happens?

You’re already behind

ULL leaders like Jump Trading, XTX and PDT Partners probably already thought this through, so will already be ahead of you. They’d immediately deploy the abovementioned gameplan to continually find the lowest latency boxes. They’d probably try some EKS instances and bare metal instances too, in case the matching engines are now containerized, or bare-metal-ized.

Time-keeping

If you’re on a VM, you’re probably getting time via the hypervisor, but is it accurate enough? If you have a great strategy that needs to buy and sell two stocks 100 nanos (ns) after the open, and you know you’re 3 mics (µs) away from the matching engine, you may want to send those two trades at 09:29:59.9999971 at the hope your order arrives at 100ns past 9.30am. Since 2023, high-precision PTP has been available on some EC2 instances (link), so it’s likely you’d want to be on one of these boxes. Amazon says the accuracy is “within microseconds” so this could be problematic. A strategy could be to send multiple trades at 09:29:59.99999{…, 4,5,6} and see if they get filled, but this risks double-trading (and breaking the exchange’s rules) which may well itself affect the strategy’s alpha.

Market data

The NYSE might make it your problem to deal with recording data, or they may make an S3 bucket available to anyone who pays them. Anyways, you probably want to pipe that directly into the strategy at filesystem level from the virtual NIC, and secondarily into Kafka/RedPanda and into something like kdb+ or ClickHouse to look at later.

Strats/alphas

Next, the strats. All HDL-written code for FPGAs becomes irrelevant, and winning strategies would quickly be ported to C, compiled with some optimal compiler for wahtever CPU the VM/container/bare metal box chosen is running on. For example, Intel has Math Kernel Library, whilst AMD has their own called AOCL. Strategies may even need to be compiled with optimizations for both.

Optimisations
  1. Running some more lightweight OS like an ultra-thinned down Debian guest might yield some advantages to stock EC2 instances. This would be at the trade-off of increasing the time to build VMs, which will almost always land up somewhere useless for latency purposes.
  2. Going to bare metal even when the exchange is on EC2 might still be better if you could remove some crucial mics of hypervisor overhead. The create and kill game might need to be played with EC2 Bare Metal, and this may offend Amazon.
  3. Hogging instances might pay, if this stops competitors getting good hosts. This would eat into PnL and is also wasteful on energy.
Nasty stuff that might happen
  1. Bad players could do a ping test to many thousands of EC2 instances, find those which are also at very low latency to their good boxes (assuming these are competitors), and DDoS them during trading hours to hammer the hypervisor’s NIC. This would result in critical overhead occurring for competitors sending orders out.
  2. Bad players could also generate large bursts of traffic on good EC2 instances they control on hypervisors believed to be close to the exchange, but don’t plan to trade on, to achieve the same thing in a less blatantly offensive way (e.g. say 40 servers in a rack, you guesstimate which physical box your trading VM is on, but also try find VMs on the other 39 good servers, and just generate spam traffic to add overhead to the actual NICs).
Unsolved problems
  1. How would you short AMZN stock if there was an outage on AWS? This is super convenient for Amazon.

Winners and losers

Smaller quant shops and sophisticated home traders are the winners, whilst professional HFTs would be losers due to their sophisticated hardware advantages becoming worthless.

Equinix is also a loser, and obviously Amazon is the biggest winner overall.

Why did Aquis move?

  1. They probably can’t differentiate massively with LSEG or Euronext (who have way bigger tech spend), so being cloud-based is something different to do.
  2. AWS might have given them a good deal versus a colo, since Aquis doesn’t command the same sway someone like the NASDAQ does to bless Equinix with a cash cow dairy farm.
  3. The UK is an HFT-hostile country (due to share trading taxes) where most UK ULL trading is on swaps and CFDs anyways, so they have little to lose by angering the HFTs by levelling the playing field.