Aptos Foundation:Clear standards, a clear future, and why verifiable performance benchmarks are critical for us at Web3.

The Aptos Foundation is proposing the first fully repeatable performance benchmark to start an industry-wide discussion around the definition of "performance" and hopefully reach a consensus.

• Due to the lack of transparency and accountability in Web3, there is an urgent need for verifiable real-world performance benchmarks. 
• The Aptos Foundation developed the first benchmark of its suite of repeatable benchmarks and came up with definitions of terms such as transaction and peak performance to properly measure performance.
• Aptos encourages and recognizes the need for continued community involvement to establish the most accurate standards.
• This technical analysis delves into how Block-STM and Quorum Store drive Aptos Network performance and why horizontal scaling is the way forward for Aptos.
  The industry needs clear standards.
  

With the rapid growth of the Web3 space, evaluating the performance of multiple blockchain and scaling solutions poses new challenges for both developers and users alike, namely, determining credibility, transparency, and accountability is key to building trust in the emerging industry. However, since no standardized evaluation methodology is currently established, this creates an urgent need for a verifiable, real-world benchmark.

The Aptos Foundation has proposed the first fully reproducible performance benchmarking to kick-start an industry-wide discussion on the definition of "performance" aimed at reaching a consensus. Over time, Aptos aims to collaborate with peers to develop this framework into the blockchain equivalent of the Aptos Foundation.TPC benchmarks.. Before diving into the details of testing, we need to first define the standard metric for blockchain performance: transactions.

What's the deal?
"Transactions per second" is a commonly used metric to assess the performance and scalability of blockchain networks. Since many networks have different definitions of their transactions, it is difficult to quantify this metric, making it difficult to directly assess performance.

At Aptos, we consider a transaction to be a sequence of one or more operations signed by a user as a single logical unit of work. For example, 100 operations signed by a user once is a transaction. Operations that are automatically added for the correct operation of the system (e.g., votes, metadata, or other artifacts) are system transactions, not user transactions. Transactions can be short and simple or long and complex, so benchmarks that capture common patterns are critical to understanding throughput for different use cases

By extension of this definition, a key measure of blockchain success is the "peak sustained performance" that a network can maintain without downtime. So how do you ensure that the network can credibly measure this sustained throughput? In order to assess throughput, we worked with a traditional Web2 company to conduct a benchmark test.

Introducing the first fully reproducible performance benchmark.

In order to achieve an end-to-end, real-world setup for this evaluation, we set up a complete network of nodes with a full stack of nodes running in it. To make it as close as possible to a real-world setup, we simulate it closely to the main network network.

The network consists of 100 `t2d-standard-48` nodes on the GCP, each with 48 vCPUs and 64 GB of RAM, distributed across three regions (USA, Asia and Europe). The maximum egress bandwidth of these nodes is 10 Gbps, and the round-trip latency between nodes varies between 120 ms and 250 ms depending on the region. This setup closely replicates the network latency and reliability conditions of the main network. We test whether the consensus mechanism can scale to a meaningful number of nodes. We write a transaction load generator that generates coin transfer transactions between randomized pairs of accounts, using large working sets and reasonable conflicts.

To ensure that the benchmarks are reproducible and can be run independently, we have released anrepository. This includes everything you need to set up machines and nodes, initialize the network, run load tests, and view results. This will inform and guide us as we explore Web3 solutions for gaming, social, and other industries.Aptos aims to extend this generic benchmark and build a suite of benchmarks for common industry use cases. With this setup we were able to validate a sustained TPS of 20k over 30 minutes. let's take a look at how a well-performing blockchain is designed and how these initial results pave the way for significant performance gains through horizontal scaling.

How to build a high-performing blockchain.

A typical blockchain stack consists of three main components: consensus, execution, and storage.

• The consensus layer receives incoming transactions and is responsible for ensuring that all nodes in the network agree on a particular order of transactions.
• The execution layer takes the current state and incoming transactions reached by consensus and is responsible for processing smart contracts and executing transactions.
• The storage layer is responsible for persisting all data associated with the blockchain, including the state of the ledger and any associated smart contract data. It provides the current state to the execution layer and updates the state based on the results of the execution.

In order to design a high-performance blockchain that supports a large number of transactions while maintaining low latency, it is critical to ensure that all components can scale to support high throughput and low latency.

As the Aptos Foundation explores how best to handle performance issues, we work with the community to use unique and innovative strategies for consensus, execution, and storage:

• Consensus: the Quorum Store is Aptos' "Narwhal"Implementations that decouple data from metadata. This allows data propagation to occur outside of the consensus critical path, making it very efficient and scalable. Developers are in the final stages of deploying the Quorum Store to the main network.
• Implementation:Block-STM is Aptos' parallel execution engine that uses a novel approach to integrate theSoftware Transactional Memory(STM) combined with concurrency control to execute transactions in a parallel fashion and validate them after execution and re-execute them if needed.
• Storage: Aptos' storage approach uses a combination of persistence and in-memory storage schemes, a lock-free sparse Merkle tree implementation, caching and parallelization specifically for Block-STM.
  Looking ahead, as Aptos Network's throughput continues to expand, our team identified horizontal scaling as the most effective approach. By scaling resources in a cloud-like fashion, orders of magnitude more resources can be brought into the system, which is critical for Web3 to be adopted for mainstream use cases.
  Scaling the storage tier is the most urgent priority because it is both the bottleneck and the easiest component to scale to multiple disks or machines. There is also an opportunity to extend the Quorum Store-based consensus by scaling it horizontally. AsTheNarwhalTheAs demonstrated in the paper, when data dissemination is decoupled from metadata ordering, consensus can exceed 600k tps on multiple working machines.Finally, while the Block-STM implementations have demonstrated that it is possible to have the execution component on a single 32-core machine 160k TPS, but it will eventually become a bottleneck as storage and consensus scale. To prepare for this, developers are exploring sharding as a way to scale the execution horizontally across multiple machines.
  Building the future of Web3 together.
  At the Aptos Foundation, we are committed to the core concept of Web3 - transparency: openness.Blockchain Designand all aspects of network operations. Driving this commitment was the impetus for developing this replicable performance benchmark. This setup is a first attempt, and we hope it serves as a lens through which a dialog on performance issues can take place and consensus can be reached.
  Web3's success relies on collaboration and shared standards. We encourage the community to independently validate the results and look to our peers to challenge us to improve these methods and collectively move the industry forward.