G3 Sequencer: How We Built a 60,000 TPS Sequencer with 10 ms Latency

Over the past few months, several high-performance blockchains have emerged with the goal of creating new categories of applications, enabled by high throughput and low latency. However, applications with the most demanding requirements are best served by a dedicated chain tuned to their workload.

In that effort, my team of core engineers has spent months building the most performant, production-grade sequencer available on the market, capable of sustaining ~60,000 payment-style transactions per second with ~10 ms receipt latency.

This post discusses some of the key innovations behind G3 and how they benefit teams seeking high throughput and low latency.

Building from Requirements

G3 Sequencer is built on the shoulders of Reth, leveraging its validation pipeline while rebuilding the ingestion and building flow.

But to build without an understanding of requirements and customers is to build the wrong thing. Fortunately, at Conduit, we work closely with dozens of customers, including Tempo, Plume, Polygon, Katana, and Ronin. This gives us a deep understanding of workloads ranging from thousands of small payment-style transactions to gas-guzzling order-book settlement transactions and calldata-heavy on-chain verification.

We’ve built a custom in-house testing harness that allows us to generate production-scale load across representative workloads and reveal bottlenecks that only appear at high throughput. Subsequently, every improvement that we’ve made has been tested on a production-like blockchain environment.

The Innovations

Each innovation is framed around the customer problem it solves.

Durable and Fast Transaction Receipts

The faster a transaction receipt is available, the faster an off-chain application can react and ultimately provide a better user experience to end customers. This allows applications to react to on-chain events multiple times within a block.

However, we’ve heard from our customers that the durability guarantees of these receipts are critical. Only a fraction of applications are willing to accept reduced durability for better latency. To that end, we’ve invested significantly into ensuring that observed confirmations can be recovered in disaster scenarios such as an unsafe shutdown or sequencer failover. We’ve also built custom monitoring software so we have observability if a preconfirmation is ever reorganized.

Much of how we achieved the performance improvements is a function of detailed per-step execution timing. For example, here is one of the views that breaks down the time we spend in assembly during a single 10 ms flashblock period.

Some of the notable techniques include:

• Reducing data copying — these are particularly bad as blocks get large at high throughput
• Deferring intermediate root calculations
• Leveraging execution and trie caches

Pipelined Block Building

You can’t improve what you don’t measure. We measure everything, and one of the metrics we quickly isolated was ‘builder idle time’. Quite simply, it refers to the time where transactions are not actively being executed.

We found that this idle time scaled as a function of load, i.e., the more load on the chain, the more time the builder spent idle. There were a few key culprits for this idle time:

• Engine API + serialization/deserialization: At large block sizes, this was especially bad, accounting for several hundred milliseconds.
• State/Receipt/Tx Root calculations: Naturally, these are a function of the transactions that are executed.
• Block Validation: i.e., the time taken to validate that the block was built according to protocol rules.

We approached this from two angles: reducing the time each takes and leveraging that time to do useful work.

This led us to a pipelined block-building architecture, where we begin building the next block ahead of time by providing attribute hints in the Engine API. We’ve done this in such a way that the attribute hints are always canonicalized. A positive side effect of pipelining is that it also improves tail latency. Below, you can see an example of how pipelining has allowed us to leverage most of the idle time, i.e., the ‘head start’.

Slipstream: The mempool for real-time apps

When most core dev teams quote their gas/second numbers, they are quoting the re-execution time, i.e., how quickly they can re-execute an existing chain. This is not the same as asking, “How fast can we sequence or create a new chain?”

Re-execution completely excludes a key pathway: the speed at which one can ingest and process transactions from the RPC into the mempool’s pending transactions list.

Our team quickly realized that the off-the-shelf Reth mempool is far too slow at refreshing the mempool. It was designed to be fetched every two seconds, but we were doing it every 10 ms, leading it to consume 80% of our build time.

We went back to the drawing board, based on a key realization: most transactions for high-throughput on-chain applications are submitted from backend relays, not directly from wallet users.

With this in mind, we built a new ingestion pathway for authorized backends called ‘Slipstream’ from the ground up. This pathway introduces a new native batch RPC method that entirely skips the vanilla mempool, where clients receive a synchronous inclusion guarantee or rejection and are responsible for managing their own backpressure.

This technique is one of those that only an application that owns its own chain can leverage. We’ve also optimized the normal transaction inclusion pathway, leveraging a stream-based approach for piping transactions to the builder.

Workload-Specific Prewarming

One of the superpowers of owning your own chain is the ability to tune it around the workloads it actually runs. This lets us address a problem that affects every chain at scale: as state grows (particularly within a single EVM account), the number of pages required to keep the working set hot grows beyond reasonable memory bounds.

In Reth, storage slots are distributed at ~random across pages, because their location in MDBX is derived from a hash. Such a distribution disregards the access patterns of the smart contract itself, which means that for most workloads this is a cache-locality problem rather than a state-size problem.

One of the techniques we've used to normalize the performance impact is 'workload-specific prewarming', through which we bring the storage slots relevant to a transaction into memory before it executes. We're also trialing new smart contract patterns and precompiles designed for better execution performance by optimizing for page locality.

Fixing Scaling Limits

This is not necessarily a performance technique, but is just as critical. With our ability to spin up and down production-like G3 environments in minutes, we encountered several limitations in Reth and contributed upstream fixes to improve the ecosystem. For example, we’re the first team to push block sizes large enough that they caused peering issues, forcing configurability in Reth.

We also hit the 16 MiB limit for block bodies in RLPx frames for workloads with large calldata. This would normally prevent nodes from syncing via the execution layer, but we created an RLPx sub-protocol that allows us to chunk block bodies and continue syncing.

Conclusion

The Core Team at Conduit has spent months building G3 to be the best-in-class sequencer, sustaining over 60,000 payment-style transactions per second with 10 ms receipt latency while solving problems no other team has encountered.

These results represent G3 today, and its performance continues to improve with every release. For applications that need this level of throughput and latency without spending months re-engineering from the ground up, there is no alternative.

If you are building applications where throughput, latency, and execution reliability matter and want to get a G3 demo, you can request one here.