Benchmarks

• 1: Storage Memory Usage Benchmark
• 2: Watch Memory Usage Benchmark
• 3: Benchmarking etcd v3
• 4: Benchmarking etcd v2.2.0-rc-memory
• 5: Benchmarking etcd v2.2.0-rc
• 6: Benchmarking etcd v2.2.0
• 7: Benchmarking etcd v2.1.0

1 - Storage Memory Usage Benchmark

Two components of etcd storage consume physical memory. The etcd process allocates an in-memory index to speed key lookup. The process’s page cache, managed by the operating system, stores recently-accessed data from disk for quick re-use.

The in-memory index holds all the keys in a B-tree data structure, along with pointers to the on-disk data (the values). Each key in the B-tree may contain multiple pointers, pointing to different versions of its values. The theoretical memory consumption of the in-memory index can hence be approximated with the formula:

N * (c1 + avg_key_size) + N * (avg_versions_of_key) * (c2 + size_of_pointer)

where c1 is the key metadata overhead and c2 is the version metadata overhead.

The graph shows the detailed structure of the in-memory index B-tree.

                                In mem index

                               +------------+
                               | key || ... |
  +--------------+             |     ||     |
  |              |             +------------+
  |              |             | v1  || ... |
  |   disk    <----------------|     ||     | Tree Node
  |              |             +------------+
  |              |             | v2  || ... |
  |           <----------------+     ||     |
  |              |             +------------+
  +--------------+       +-----+    |   |   |
                         |     |    |   |   |
                         |     +------------+
                         |
                         |
                         ^
                      ------+
                      | ... |
                      |     |
                      +-----+
                      | ... | Tree Node
                      |     |
                      +-----+
                      | ... |
                      |     |
                      ------+

Page cache memory is managed by the operating system and is not covered in detail in this document.

Testing Environment

etcd version

• git head https://github.com/etcd-io/etcd/commit/776e9fb7be7eee5e6b58ab977c8887b4fe4d48db

GCE n1-standard-2 machine type

• 7.5 GB memory
• 2x CPUs

In-memory index memory usage

In this test, we only benchmark the memory usage of the in-memory index. The goal is to find c1 and c2 mentioned above and to understand the hard limit of memory consumption of the storage.

We calculate the memory usage consumption via the Go runtime.ReadMemStats. We calculate the total allocated bytes difference before creating the index and after creating the index. It cannot perfectly reflect the memory usage of the in-memory index itself but can show the rough consumption pattern.

Based on the result, we can calculate c1=120bytes, c2=30bytes. We only need two sets of data to calculate c1 and c2, since they are the only unknown variable in the formula. The c1=120bytes and c2=30bytes are the average value of the 4 sets of c1 and c2 we calculated. The key metadata overhead is still relatively nontrivial (50%) for small key-value pairs. However, this is a significant improvement over the old store, which had at least 1000% overhead.

Overall memory usage

The overall memory usage captures how much RSS etcd consumes with the storage. The value size should have very little impact on the overall memory usage of etcd, since we keep values on disk and only retain hot values in memory, managed by the OS page cache.

Based on the result, we know the value size does not significantly impact the memory consumption. There is some minor increase due to more data held in the OS page cache.

2 - Watch Memory Usage Benchmark

A primary goal of etcd is supporting a very large number of watchers doing a massively large amount of watching. etcd aims to support O(10k) clients, O(100K) watch streams (O(10) streams per client) and O(10M) total watchings (O(100) watching per stream). The memory consumed by each individual watching accounts for the largest portion of etcd’s overall usage, and is therefore the focus of current and future optimizations.

Three related components of etcd watch consume physical memory: each grpc.Conn, each watch stream, and each instance of the watching activity. grpc.Conn maintains the actual TCP connection and other gRPC connection state. Each grpc.Conn consumes O(10kb) of memory, and might have multiple watch streams attached.

Each watch stream is an independent HTTP2 connection which consumes another O(10kb) of memory. Multiple watchings might share one watch stream.

Watching is the actual struct that tracks the changes on the key-value store. Each watching should only consume < O(1kb).

                                          +-------+
                                          | watch |
                              +---------> | foo   |
                              |           +-------+
                       +------+-----+
                       |   stream   |
      +--------------> |            |
      |                +------+-----+     +-------+
      |                       |           | watch |
      |                       +---------> | bar   |
+-----+------+                            +-------+
|            |         +------------+
|   conn     +-------> |   stream   |
|            |         |            |
+-----+------+         +------------+
      |
      |
      |
      |                +------------+
      +--------------> |   stream   |
                       |            |
                       +------------+

The theoretical memory consumption of watch can be approximated with the formula: memory = c1 * number_of_conn + c2 * avg_number_of_stream_per_conn + c3 * avg_number_of_watch_stream

Testing Environment

etcd version

• git head https://github.com/etcd-io/etcd/commit/185097ffaa627b909007e772c175e8fefac17af3

GCE n1-standard-2 machine type

• 7.5 GB memory
• 2x CPUs

Overall memory usage

The overall memory usage captures how much RSS etcd consumes with the client watchers. While the result may vary by as much as 10%, it is still meaningful, since the goal is to learn about the rough memory usage and the pattern of allocations.

With the benchmark result, we can calculate roughly that c1 = 17kb, c2 = 18kb and c3 = 350bytes. So each additional client connection consumes 17kb of memory and each additional stream consumes 18kb of memory, and each additional watching only cause 350bytes. A single etcd server can maintain millions of watchings with a few GB of memory in normal case.

3 - Benchmarking etcd v3

Physical machines

GCE n1-highcpu-2 machine type

• 1x dedicated local SSD mounted under /var/lib/etcd
• 1x dedicated slow disk for the OS
• 1.8 GB memory
• 2x CPUs
• etcd version 2.2.0

etcd Cluster

1 etcd member running in v3 demo mode

Testing

Use etcd v3 benchmark tool .

Performance

reading one single key

The performance is nearly the same as the one with empty server handler.

reading one single key after putting

The performance with empty server handler is not affected by one put. So the performance downgrade should be caused by storage package.

4 - Benchmarking etcd v2.2.0-rc-memory

Physical machine

GCE n1-standard-2 machine type

• 1x dedicated local SSD mounted under /var/lib/etcd
• 1x dedicated slow disk for the OS
• 7.5 GB memory
• 2x CPUs

etcd

etcd Version: 2.2.0-rc.0+git
Git SHA: 103cb5c
Go Version: go1.5
Go OS/Arch: linux/amd64

Testing

Start 3-member etcd cluster, each of which uses 2 cores.

The length of key name is always 64 bytes, which is a reasonable length of average key bytes.

Memory Maximal Usage

• etcd may use maximal memory if one follower is dead and the leader keeps sending snapshots.
• max RSS is the maximal memory usage recorded in 3 runs.

Data Size Threshold

• When etcd reaches data size threshold, it may trigger leader election easily and drop part of proposals.
• For most cases, the etcd cluster should work smoothly if it doesn’t hit the threshold. If it doesn’t work well due to insufficient resources, decrease its data size.

5 - Benchmarking etcd v2.2.0-rc

Physical machine

GCE n1-highcpu-2 machine type

• 1x dedicated local SSD mounted under /var/lib/etcd
• 1x dedicated slow disk for the OS
• 1.8 GB memory
• 2x CPUs

etcd Cluster

3 etcd 2.2.0-rc members, each runs on a single machine.

Detailed versions:

etcd Version: 2.2.0-alpha.1+git
Git SHA: 59a5a7e
Go Version: go1.4.2
Go OS/Arch: linux/amd64

Also, we use 3 etcd 2.1.0 alpha-stage members to form cluster to get base performance. etcd’s commit head is at c7146bd5 , which is the same as the one that we use in etcd 2.1 benchmark .

Testing

Bootstrap another machine and use the hey HTTP benchmark tool to send requests to each etcd member. Check the benchmark hacking guide for detailed instructions.

Performance

reading one single key

writing one single key

performance changes explanation

• read QPS in most scenarios is decreased by 5~8%. The reason is that etcd records store metrics for each store operation. The metrics is important for monitoring and debugging, so this is acceptable.

• write QPS to leader is increased by 20~30%. This is because we decouple raft main loop and entry apply loop, which avoids them blocking each other.

• write QPS to all servers is increased by 30~80% because follower could receive latest commit index earlier and commit proposals faster.

6 - Benchmarking etcd v2.2.0

Physical Machines

GCE n1-highcpu-2 machine type

• 1x dedicated local SSD mounted as etcd data directory
• 1x dedicated slow disk for the OS
• 1.8 GB memory
• 2x CPUs

etcd Cluster

3 etcd 2.2.0 members, each runs on a single machine.

Detailed versions:

etcd Version: 2.2.0
Git SHA: e4561dd
Go Version: go1.5
Go OS/Arch: linux/amd64

Testing

Bootstrap another machine, outside of the etcd cluster, and run the hey HTTP benchmark tool with a connection reuse patch to send requests to each etcd cluster member. See the benchmark instructions for the patch and the steps to reproduce our procedures.

The performance is calculated through results of 100 benchmark rounds.

Performance

Single Key Read Performance

Single Key Write Performance

Performance Changes

• Because etcd now records metrics for each API call, read QPS performance seems to see a minor decrease in most scenarios. This minimal performance impact was judged a reasonable investment for the breadth of monitoring and debugging information returned.

• Write QPS to cluster leaders seems to be increased by a small margin. This is because the main loop and entry apply loops were decoupled in the etcd raft logic, eliminating several blocks between them.

• Write QPS to all members seems to be increased by a significant margin, because followers now receive the latest commit index sooner, and commit proposals more quickly.

7 - Benchmarking etcd v2.1.0

Physical machines

GCE n1-highcpu-2 machine type

• 1x dedicated local SSD mounted under /var/lib/etcd
• 1x dedicated slow disk for the OS
• 1.8 GB memory
• 2x CPUs
• etcd version 2.1.0 alpha

etcd Cluster

3 etcd members, each runs on a single machine

Testing

Bootstrap another machine and use the hey HTTP benchmark tool to send requests to each etcd member. Check the benchmark hacking guide for detailed instructions.

Performance

reading one single key

writing one single key