Developer guide

• 1: Discovery service protocol
• 2: Set up a local cluster
• 3: Interacting with etcd
• 4: Why gRPC gateway
• 5: gRPC naming and discovery
• 6: Embedding etcd in a Go Application
• 7: System limits
• 8: etcd features
• 9: API reference
• 10: API reference: concurrency

1 - Discovery service protocol

Discovery service protocol helps new etcd member to discover all other members in cluster bootstrap phase using a shared discovery URL.

Discovery service protocol is only used in cluster bootstrap phase, and cannot be used for runtime reconfiguration or cluster monitoring.

The protocol uses a new discovery token to bootstrap one unique etcd cluster. Remember that one discovery token can represent only one etcd cluster. As long as discovery protocol on this token starts, even if it fails halfway, it must not be used to bootstrap another etcd cluster.

The rest of this article will walk through the discovery process with examples that correspond to a self-hosted discovery cluster. The public discovery service, discovery.etcd.io, functions the same way, but with a layer of polish to abstract away ugly URLs, generate UUIDs automatically, and provide some protections against excessive requests. At its core, the public discovery service still uses an etcd cluster as the data store as described in this document.

Protocol workflow

The idea of discovery protocol is to use an internal etcd cluster to coordinate bootstrap of a new cluster. First, all new members interact with discovery service and help to generate the expected member list. Then each new member bootstraps its server using this list, which performs the same functionality as -initial-cluster flag.

In the following example workflow, we will list each step of protocol in curl format for ease of understanding.

By convention the etcd discovery protocol uses the key prefix _etcd/registry. If http://example.com hosts an etcd cluster for discovery service, a full URL to discovery keyspace will be http://example.com/v2/keys/_etcd/registry. We will use this as the URL prefix in the example.

Creating a new discovery token

Generate a unique token that will identify the new cluster. This will be used as a unique prefix in discovery keyspace in the following steps. An easy way to do this is to use uuidgen:

UUID=$(uuidgen)

Specifying the expected cluster size

The discovery token expects a cluster size that must be specified. The size is used by the discovery service to know when it has found all members that will initially form the cluster.

curl -X PUT http://example.com/v2/keys/_etcd/registry/${UUID}/_config/size -d value=${cluster_size}

Usually the cluster size is 3, 5 or 7. Check optimal cluster size for more details.

Bringing up etcd processes

Given the discovery URL, use it as -discovery flag and bring up etcd processes. Every etcd process will follow this next few steps internally if given a -discovery flag.

Registering itself

The first thing for etcd process is to register itself into the discovery URL as a member. This is done by creating member ID as a key in the discovery URL.

curl -X PUT http://example.com/v2/keys/_etcd/registry/${UUID}/${member_id}?prevExist=false -d value="${member_name}=${member_peer_url_1}&${member_name}=${member_peer_url_2}"

Checking the status

It checks the expected cluster size and registration status in discovery URL, and decides what the next action is.

curl -X GET http://example.com/v2/keys/_etcd/registry/${UUID}/_config/size
curl -X GET http://example.com/v2/keys/_etcd/registry/${UUID}

If registered members are still not enough, it will wait for left members to appear.

If the number of registered members is bigger than the expected size N, it treats the first N registered members as the member list for the cluster. If the member itself is in the member list, the discovery procedure succeeds and it fetches all peers through the member list. If it is not in the member list, the discovery procedure finishes with the failure that the cluster has been full.

In etcd implementation, the member may check the cluster status even before registering itself. So it could fail quickly if the cluster has been full.

Waiting for all members

The wait process is described in detail in the etcd API documentation .

curl -X GET http://example.com/v2/keys/_etcd/registry/${UUID}?wait=true&waitIndex=${current_etcd_index}

It keeps waiting until finding all members.

Public discovery service

CoreOS Inc. hosts a public discovery service at https://discovery.etcd.io/ , which provides some nice features for ease of use.

Mask key prefix

Public discovery service will redirect https://discovery.etcd.io/${UUID} to etcd cluster behind for the key at /v2/keys/_etcd/registry. It masks register key prefix for short and readable discovery url.

Get new token

GET /new

Sent query:
	size=${cluster_size}
Possible status codes:
	200 OK
	400 Bad Request
200 Body:
	generated discovery url

The generation process in the service follows the steps from Creating a New Discovery Token to Specifying the Expected Cluster Size .

Check discovery status

GET /${UUID}

The status for this discovery token, including the machines that have been registered, can be checked by requesting the value of the UUID.

Open-source repository

The repository is located at https://github.com/coreos/discovery.etcd.io . It could be used to build a custom discovery service.

2 - Set up a local cluster

For testing and development deployments, the quickest and easiest way is to configure a local cluster. For a production deployment, refer to the clustering section.

Local standalone cluster

Starting a cluster

Run the following to deploy an etcd cluster as a standalone cluster:

$ ./etcd
...

If the etcd binary is not present in the current working directory, it might be located either at $GOPATH/bin/etcd or at /usr/local/bin/etcd. Run the command appropriately.

The running etcd member listens on localhost:2379 for client requests.

Interacting with the cluster

Use etcdctl to interact with the running cluster:

1. Store an example key-value pair in the cluster:

     $ ./etcdctl put foo bar
     OK

   If OK is printed, storing key-value pair is successful.

2. Retrieve the value of foo:

   $ ./etcdctl get foo
   bar

   If bar is returned, interaction with the etcd cluster is working as expected.

Local multi-member cluster

Starting a cluster

A Procfile at the base of the etcd git repository is provided to easily configure a local multi-member cluster. To start a multi-member cluster, navigate to the root of the etcd source tree and perform the following:

1. Install goreman to control Procfile-based applications:

   $ go install github.com/mattn/goreman@latest

2. Start a cluster with goreman using etcd’s stock Procfile:

   $ goreman -f Procfile start

   The members start running. They listen on localhost:2379, localhost:22379, and localhost:32379 respectively for client requests.

Interacting with the cluster

Use etcdctl to interact with the running cluster:

1. Print the list of members:

   $ etcdctl --write-out=table --endpoints=localhost:2379 member list

   The list of etcd members are displayed as follows:

   +------------------+---------+--------+------------------------+------------------------+
   |        ID        | STATUS  |  NAME  |       PEER ADDRS       |      CLIENT ADDRS      |
   +------------------+---------+--------+------------------------+------------------------+
   | 8211f1d0f64f3269 | started | infra1 | http://127.0.0.1:2380  | http://127.0.0.1:2379  |
   | 91bc3c398fb3c146 | started | infra2 | http://127.0.0.1:22380 | http://127.0.0.1:22379 |
   | fd422379fda50e48 | started | infra3 | http://127.0.0.1:32380 | http://127.0.0.1:32379 |
   +------------------+---------+--------+------------------------+------------------------+

2. Store an example key-value pair in the cluster:

   $ etcdctl put foo bar
   OK

   If OK is printed, storing key-value pair is successful.

Testing fault tolerance

To exercise etcd’s fault tolerance, kill a member and attempt to retrieve the key.

1. Identify the process name of the member to be stopped.

   The Procfile lists the properties of the multi-member cluster. For example, consider the member with the process name, etcd2.

2. Stop the member:

   # kill etcd2
   $ goreman run stop etcd2

3. Store a key:

   $ etcdctl put key hello
   OK

4. Retrieve the key that is stored in the previous step:

   $ etcdctl get key
   hello

5. Retrieve a key from the stopped member:

   $ etcdctl --endpoints=localhost:22379 get key

   The command should display an error caused by connection failure:

   2017/06/18 23:07:35 grpc: Conn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 127.0.0.1:22379: getsockopt: connection refused"; Reconnecting to "localhost:22379"
   Error:  grpc: timed out trying to connect

6. Restart the stopped member:

   $ goreman run restart etcd2

7. Get the key from the restarted member:

   $ etcdctl --endpoints=localhost:22379 get key
   hello

   Restarting the member re-establish the connection. etcdctl will now be able to retrieve the key successfully. To learn more about interacting with etcd, read interacting with etcd section .

3 - Interacting with etcd

Users mostly interact with etcd by putting or getting the value of a key. This section describes how to do that by using etcdctl, a command line tool for interacting with etcd server. The concepts described here should apply to the gRPC APIs or client library APIs.

The API version used by etcdctl to speak to etcd may be set to version 2 or 3 via the ETCDCTL_API environment variable. By default, etcdctl on master (3.4) uses the v3 API and earlier versions (3.3 and earlier) default to the v2 API.

Note that any key that was created using the v2 API will not be able to be queried via the v3 API. A v3 API etcdctl get of a v2 key will exit with 0 and no key data, this is the expected behaviour.

export ETCDCTL_API=3

Find versions

etcdctl version and Server API version can be useful in finding the appropriate commands to be used for performing various operations on etcd.

Here is the command to find the versions:

$ etcdctl version
etcdctl version: 3.1.0-alpha.0+git
API version: 3.1

Write a key

Applications store keys into the etcd cluster by writing to keys. Every stored key is replicated to all etcd cluster members through the Raft protocol to achieve consistency and reliability.

Here is the command to set the value of key foo to bar:

$ etcdctl put foo bar
OK

Also a key can be set for a specified interval of time by attaching lease to it.

Here is the command to set the value of key foo1 to bar1 for 10s.

$ etcdctl put foo1 bar1 --lease=1234abcd
OK

Read keys

Applications can read values of keys from an etcd cluster. Queries may read a single key, or a range of keys.

Suppose the etcd cluster has stored the following keys:

foo = bar
foo1 = bar1
foo2 = bar2
foo3 = bar3

Here is the command to read the value of key foo:

$ etcdctl get foo
foo
bar

Here is the command to read the value of key foo in hex format:

$ etcdctl get foo --hex
\x66\x6f\x6f          # Key
\x62\x61\x72          # Value

Here is the command to read only the value of key foo:

$ etcdctl get foo --print-value-only
bar

Here is the command to range over the keys from foo to foo3:

$ etcdctl get foo foo3
foo
bar
foo1
bar1
foo2
bar2

Here is the command to range over all keys prefixed with foo:

$ etcdctl get --prefix foo
foo
bar
foo1
bar1
foo2
bar2
foo3
bar3

Here is the command to range over all keys prefixed with foo, limiting the number of results to 2:

$ etcdctl get --prefix --limit=2 foo
foo
bar
foo1
bar1

Here is the command to range over all keys prefixed with foo using the RangeStream RPC. The result is identical to a unary Range:

$ etcdctl get --stream --prefix foo
foo
bar
foo1
bar1
foo2
bar2
foo3
bar3

--stream does not support --order, --sort-by, or revision filters.

Read past version of keys

Applications may want to read superseded versions of a key. For example, an application may wish to roll back to an old configuration by accessing an earlier version of a key. Alternatively, an application may want a consistent view over multiple keys through multiple requests by accessing key history. Since every modification to the etcd cluster key-value store increments the global revision of an etcd cluster, an application can read superseded keys by providing an older etcd revision.

Suppose an etcd cluster already has the following keys:

foo = bar         # revision = 2
foo1 = bar1       # revision = 3
foo = bar_new     # revision = 4
foo1 = bar1_new   # revision = 5

Here are an example to access the past versions of keys:

$ etcdctl get --prefix foo # access the most recent versions of keys
foo
bar_new
foo1
bar1_new

$ etcdctl get --prefix --rev=4 foo # access the versions of keys at revision 4
foo
bar_new
foo1
bar1

$ etcdctl get --prefix --rev=3 foo # access the versions of keys at revision 3
foo
bar
foo1
bar1

$ etcdctl get --prefix --rev=2 foo # access the versions of keys at revision 2
foo
bar

$ etcdctl get --prefix --rev=1 foo # access the versions of keys at revision 1

Read keys which are greater than or equal to the byte value of the specified key

Applications may want to read keys which are greater than or equal to the byte value of the specified key.

Suppose an etcd cluster already has the following keys:

a = 123
b = 456
z = 789

Here is the command to read keys which are greater than or equal to the byte value of key b :

$ etcdctl get --from-key b
b
456
z
789

Delete keys

Applications can delete a key or a range of keys from an etcd cluster.

Suppose an etcd cluster already has the following keys:

foo = bar
foo1 = bar1
foo3 = bar3
zoo = val
zoo1 = val1
zoo2 = val2
a = 123
b = 456
z = 789

Here is the command to delete key foo:

$ etcdctl del foo
1 # one key is deleted

Here is the command to delete keys ranging from foo to foo9:

$ etcdctl del foo foo9
2 # two keys are deleted

Here is the command to delete key zoo with the deleted key value pair returned:

$ etcdctl del --prev-kv zoo
1   # one key is deleted
zoo # deleted key
val # the value of the deleted key

Here is the command to delete keys having prefix as zoo:

$ etcdctl del --prefix zoo
2 # two keys are deleted

Here is the command to delete keys which are greater than or equal to the byte value of key b :

$ etcdctl del --from-key b
2 # two keys are deleted

Watch key changes

Applications can watch on a key or a range of keys to monitor for any updates.

Here is the command to watch on key foo:

$ etcdctl watch foo
# in another terminal: etcdctl put foo bar
PUT
foo
bar

Here is the command to watch on key foo in hex format:

$ etcdctl watch foo --hex
# in another terminal: etcdctl put foo bar
PUT
\x66\x6f\x6f          # Key
\x62\x61\x72          # Value

Here is the command to watch on a range key from foo to foo9:

$ etcdctl watch foo foo9
# in another terminal: etcdctl put foo bar
PUT
foo
bar
# in another terminal: etcdctl put foo1 bar1
PUT
foo1
bar1

Here is the command to watch on keys having prefix foo:

$ etcdctl watch --prefix foo
# in another terminal: etcdctl put foo bar
PUT
foo
bar
# in another terminal: etcdctl put fooz1 barz1
PUT
fooz1
barz1

Here is the command to watch on multiple keys foo and zoo:

$ etcdctl watch -i
$ watch foo
$ watch zoo
# in another terminal: etcdctl put foo bar
PUT
foo
bar
# in another terminal: etcdctl put zoo val
PUT
zoo
val

Watch historical changes of keys

Applications may want to watch for historical changes of keys in etcd. For example, an application may wish to receive all the modifications of a key; if the application stays connected to etcd, then watch is good enough. However, if the application or etcd fails, a change may happen during the failure, and the application will not receive the update in real time. To guarantee the update is delivered, the application must be able to watch for historical changes to keys. To do this, an application can specify a historical revision on a watch, just like reading past version of keys.

Suppose we finished the following sequence of operations:

$ etcdctl put foo bar         # revision = 2
OK
$ etcdctl put foo1 bar1       # revision = 3
OK
$ etcdctl put foo bar_new     # revision = 4
OK
$ etcdctl put foo1 bar1_new   # revision = 5
OK

Here is an example to watch the historical changes:

# watch for changes on key `foo` since revision 2
$ etcdctl watch --rev=2 foo
PUT
foo
bar
PUT
foo
bar_new

# watch for changes on key `foo` since revision 3
$ etcdctl watch --rev=3 foo
PUT
foo
bar_new

Here is an example to watch only from the last historical change:

# watch for changes on key `foo` and return last revision value along with modified value
$ etcdctl watch --prev-kv foo
# in another terminal: etcdctl put foo bar_latest
PUT
foo         # key
bar_new     # last value of foo key before modification
foo         # key
bar_latest  # value of foo key after modification

Watch progress

Applications may want to check the progress of a watch to determine how up-to-date the watch stream is. For example, if a watch is used to update a cache, it can be useful to know if the cache is stale compared to the revision from a quorum read.

Progress requests can be issued using the “progress” command in interactive watch session to ask the etcd server to send a progress notify update in the watch stream:

$ etcdctl watch -i
$ watch a
$ progress
progress notify: 1
# in another terminal: etcdctl put x 0
# in another terminal: etcdctl put y 1
$ progress
progress notify: 3

Compacted revisions

As we mentioned, etcd keeps revisions so that applications can read past versions of keys. However, to avoid accumulating an unbounded amount of history, it is important to compact past revisions. After compacting, etcd removes historical revisions, releasing resources for future use. All superseded data with revisions before the compacted revision will be unavailable.

Here is the command to compact the revisions:

$ etcdctl compact 5
compacted revision 5

# any revisions before the compacted one are not accessible
$ etcdctl get --rev=4 foo
Error:  rpc error: code = 11 desc = etcdserver: mvcc: required revision has been compacted

$ etcdctl get mykey -w=json
{"header":{"cluster_id":14841639068965178418,"member_id":10276657743932975437,"revision":15,"raft_term":4}}

Grant leases

Applications can grant leases for keys from an etcd cluster. When a key is attached to a lease, its lifetime is bound to the lease’s lifetime which in turn is governed by a time-to-live (TTL). Each lease has a minimum time-to-live (TTL) value specified by the application at grant time. The lease’s actual TTL value is at least the minimum TTL and is chosen by the etcd cluster. Once a lease’s TTL elapses, the lease expires and all attached keys are deleted.

Here is the command to grant a lease:

# grant a lease with 60 second TTL
$ etcdctl lease grant 60
lease 32695410dcc0ca06 granted with TTL(60s)

# attach key foo to lease 32695410dcc0ca06
$ etcdctl put --lease=32695410dcc0ca06 foo bar
OK

Revoke leases

Applications revoke leases by lease ID. Revoking a lease deletes all of its attached keys.

Suppose we finished the following sequence of operations:

$ etcdctl lease grant 60
lease 32695410dcc0ca06 granted with TTL(60s)
$ etcdctl put --lease=32695410dcc0ca06 foo bar
OK

Here is the command to revoke the same lease:

$ etcdctl lease revoke 32695410dcc0ca06
lease 32695410dcc0ca06 revoked

$ etcdctl get foo
# empty response since foo is deleted due to lease revocation

Keep leases alive

Applications can keep a lease alive by refreshing its TTL so it does not expire.

Suppose we finished the following sequence of operations:

$ etcdctl lease grant 60
lease 32695410dcc0ca06 granted with TTL(60s)

Here is the command to keep the same lease alive:

$ etcdctl lease keep-alive 32695410dcc0ca06
lease 32695410dcc0ca06 keepalived with TTL(60)
lease 32695410dcc0ca06 keepalived with TTL(60)
lease 32695410dcc0ca06 keepalived with TTL(60)
...

Get lease information

Applications may want to know about lease information, so that they can be renewed or to check if the lease still exists or it has expired. Applications may also want to know the keys to which a particular lease is attached.

Suppose we finished the following sequence of operations:

# grant a lease with 500 second TTL
$ etcdctl lease grant 500
lease 694d5765fc71500b granted with TTL(500s)

# attach key zoo1 to lease 694d5765fc71500b
$ etcdctl put zoo1 val1 --lease=694d5765fc71500b
OK

# attach key zoo2 to lease 694d5765fc71500b
$ etcdctl put zoo2 val2 --lease=694d5765fc71500b
OK

Here is the command to get information about the lease:

$ etcdctl lease timetolive 694d5765fc71500b
lease 694d5765fc71500b granted with TTL(500s), remaining(258s)

Here is the command to get information about the lease along with the keys attached with the lease:

$ etcdctl lease timetolive --keys 694d5765fc71500b
lease 694d5765fc71500b granted with TTL(500s), remaining(132s), attached keys([zoo2 zoo1])

# if the lease has expired or does not exist it will give the below response:
Error:  etcdserver: requested lease not found

4 - Why gRPC gateway

etcd v3 uses gRPC for its messaging protocol. The etcd project includes a gRPC-based Go client and a command line utility, etcdctl , for communicating with an etcd cluster through gRPC. For languages with no gRPC support, etcd provides a JSON gRPC gateway . This gateway serves a RESTful proxy that translates HTTP/JSON requests into gRPC messages.

Using gRPC gateway

The gateway accepts a JSON mapping for etcd’s protocol buffer message definitions. Note that key and value fields are defined as byte arrays and therefore must be base64 encoded in JSON. The following examples use curl, but any HTTP/JSON client should work all the same.

Notes

gRPC gateway endpoint has changed since etcd v3.3:

• etcd v3.2 or before uses only [CLIENT-URL]/v3alpha/*.
• etcd v3.3 uses [CLIENT-URL]/v3beta/* while keeping [CLIENT-URL]/v3alpha/*.
• etcd v3.4 uses [CLIENT-URL]/v3/* while keeping [CLIENT-URL]/v3beta/*.
  • [CLIENT-URL]/v3alpha/* is deprecated.
• etcd v3.5 or later uses only [CLIENT-URL]/v3/*.
  • [CLIENT-URL]/v3beta/* is deprecated.

gRPC-gateway does not support authentication using TLS Common Name.

Put and get keys

Use the /v3/kv/range and /v3/kv/put services to read and write keys:

<<COMMENT
https://www.base64encode.org/
foo is 'Zm9v' in Base64
bar is 'YmFy'
COMMENT

curl -L http://localhost:2379/v3/kv/put \
  -X POST -d '{"key": "Zm9v", "value": "YmFy"}'
# {"header":{"cluster_id":"12585971608760269493","member_id":"13847567121247652255","revision":"2","raft_term":"3"}}

curl -L http://localhost:2379/v3/kv/range \
  -X POST -d '{"key": "Zm9v"}'
# {"header":{"cluster_id":"12585971608760269493","member_id":"13847567121247652255","revision":"2","raft_term":"3"},"kvs":[{"key":"Zm9v","create_revision":"2","mod_revision":"2","version":"1","value":"YmFy"}],"count":"1"}

# get all keys prefixed with "foo"
curl -L http://localhost:2379/v3/kv/range \
  -X POST -d '{"key": "Zm9v", "range_end": "Zm9w"}'
# {"header":{"cluster_id":"12585971608760269493","member_id":"13847567121247652255","revision":"2","raft_term":"3"},"kvs":[{"key":"Zm9v","create_revision":"2","mod_revision":"2","version":"1","value":"YmFy"}],"count":"1"}

Watch keys

Use the /v3/watch service to watch keys:

curl -N http://localhost:2379/v3/watch \
  -X POST -d '{"create_request": {"key":"Zm9v"} }' &
# {"result":{"header":{"cluster_id":"12585971608760269493","member_id":"13847567121247652255","revision":"1","raft_term":"2"},"created":true}}

curl -L http://localhost:2379/v3/kv/put \
  -X POST -d '{"key": "Zm9v", "value": "YmFy"}' >/dev/null 2>&1
# {"result":{"header":{"cluster_id":"12585971608760269493","member_id":"13847567121247652255","revision":"2","raft_term":"2"},"events":[{"kv":{"key":"Zm9v","create_revision":"2","mod_revision":"2","version":"1","value":"YmFy"}}]}}

Transactions

Issue a transaction with /v3/kv/txn:

# target CREATE
curl -L http://localhost:2379/v3/kv/txn \
  -X POST \
  -d '{"compare":[{"target":"CREATE","key":"Zm9v","createRevision":"2"}],"success":[{"requestPut":{"key":"Zm9v","value":"YmFy"}}]}'
# {"header":{"cluster_id":"12585971608760269493","member_id":"13847567121247652255","revision":"3","raft_term":"2"},"succeeded":true,"responses":[{"response_put":{"header":{"revision":"3"}}}]}

# target VERSION
curl -L http://localhost:2379/v3/kv/txn \
  -X POST \
  -d '{"compare":[{"version":"4","result":"EQUAL","target":"VERSION","key":"Zm9v"}],"success":[{"requestRange":{"key":"Zm9v"}}]}'
# {"header":{"cluster_id":"14841639068965178418","member_id":"10276657743932975437","revision":"6","raft_term":"3"},"succeeded":true,"responses":[{"response_range":{"header":{"revision":"6"},"kvs":[{"key":"Zm9v","create_revision":"2","mod_revision":"6","version":"4","value":"YmF6"}],"count":"1"}}]}

Authentication

Set up authentication with the /v3/auth service:

# create root user
curl -L http://localhost:2379/v3/auth/user/add \
  -X POST -d '{"name": "root", "password": "pass"}'
# {"header":{"cluster_id":"14841639068965178418","member_id":"10276657743932975437","revision":"1","raft_term":"2"}}

# create root role
curl -L http://localhost:2379/v3/auth/role/add \
  -X POST -d '{"name": "root"}'
# {"header":{"cluster_id":"14841639068965178418","member_id":"10276657743932975437","revision":"1","raft_term":"2"}}

# grant root role
curl -L http://localhost:2379/v3/auth/user/grant \
  -X POST -d '{"user": "root", "role": "root"}'
# {"header":{"cluster_id":"14841639068965178418","member_id":"10276657743932975437","revision":"1","raft_term":"2"}}

# enable auth
curl -L http://localhost:2379/v3/auth/enable -X POST -d '{}'
# {"header":{"cluster_id":"14841639068965178418","member_id":"10276657743932975437","revision":"1","raft_term":"2"}}

Authenticate with etcd for an authentication token using /v3/auth/authenticate:

# get the auth token for the root user
curl -L http://localhost:2379/v3/auth/authenticate \
  -X POST -d '{"name": "root", "password": "pass"}'
# {"header":{"cluster_id":"14841639068965178418","member_id":"10276657743932975437","revision":"1","raft_term":"2"},"token":"sssvIpwfnLAcWAQH.9"}

Set the Authorization header to the authentication token to fetch a key using authentication credentials:

curl -L http://localhost:2379/v3/kv/put \
  -H 'Authorization: sssvIpwfnLAcWAQH.9' \
  -X POST -d '{"key": "Zm9v", "value": "YmFy"}'
# {"header":{"cluster_id":"14841639068965178418","member_id":"10276657743932975437","revision":"2","raft_term":"2"}}

Error responses

The gRPC gateway translates gRPC status into HTTP status codes and a JSON error body. Starting in etcd v3.6, the upgrade to grpc-gateway v2 changed error handling (see the v2 migration guide’s error-handling note ), and the gateway behavior now aligns with google.rpc.Status (code, message, details) as described in Google’s API error model . Historically, older grpc-gateway versions also included a top-level error field, but this field is not supported in etcd v3.6 and higher versions.

Clients should treat the HTTP status code as the primary indicator of success or failure. If a request fails, clients should rely on the message field as the primary source of error information and use any additional details for further context.

Swagger

Generated Swagger API definitions can be found at rpc.swagger.json .

5 - gRPC naming and discovery

etcd provides a gRPC resolver to support an alternative name system that fetches endpoints from etcd for discovering gRPC services. The underlying mechanism is based on watching updates to keys prefixed with the service name.

Note that this feature is experimental because it depends on the google.golang.org/grpc/resolver package, which is still experimental in grpc-go.

Using etcd discovery with go-grpc

The etcd client provides a gRPC resolver for resolving gRPC endpoints with an etcd backend. The resolver is initialized with an etcd client:

import (
	clientv3 "go.etcd.io/etcd/client/v3"
	etcdnaming "go.etcd.io/etcd/client/v3/naming/resolver"

	"google.golang.org/grpc"
)

...

cli, err := clientv3.NewFromURL("http://localhost:2379")
if err != nil {
    // ...
}
r, err := etcdnaming.NewBuilder(cli)
if err != nil {
    // ...
}
conn, gerr := grpc.NewClient("my-service", grpc.WithResolvers(r), ...)

Managing service endpoints

The etcd resolver treats all keys under the prefix of the resolution target following a “/” (e.g., “foo/bar/my-service/”) with JSON-encoded (historically go-grpc naming.Update) values as potential service endpoints. Endpoints are added to the service by creating new keys and removed from the service by deleting keys.

Adding an endpoint

New endpoints can be added to the service through etcdctl:

ETCDCTL_API=3 etcdctl put foo/bar/my-service/1.2.3.4 '{"Addr":"1.2.3.4"}'

The etcd client’s endpoints.Manager method can also register new endpoints with a key matching the Addr:

em := endpoints.NewManager(client, "foo/bar/my-service")
err := em.AddEndpoint(context.TODO(),"foo/bar/my-service/e1", endpoints.Endpoint{Addr:"1.2.3.4"})

To enable round-robin load balancing when dialing service with multiple endpoints, you can set up you connection with grpc internal round-robin load balancer:

conn, gerr := grpc.NewClient("etcd:///foo", grpc.WithResolvers(etcdResolver),
grpc.WithDefaultServiceConfig(`{"loadBalancingPolicy":"round_robin"}`))

Deleting an endpoint

Hosts can be deleted from the service through etcdctl:

ETCDCTL_API=3 etcdctl del foo/bar/my-service/1.2.3.4

The etcd client’s endpoints.Manager method also supports deleting endpoints:

em := endpoints.NewManager(client, "foo/bar/my-service")
err := em.DeleteEndpoint(context.TODO(), "foo/bar/my-service/e1")

Registering an endpoint with a lease

Registering an endpoint with a lease ensures that if the host can’t maintain a keepalive heartbeat (e.g., its machine fails), it will be removed from the service:

lease=`ETCDCTL_API=3 etcdctl lease grant 5 | cut -f2 -d' '`
ETCDCTL_API=3 etcdctl put --lease=$lease my-service/1.2.3.4 '{"Addr":"1.2.3.4"}'
ETCDCTL_API=3 etcdctl lease keep-alive $lease

In the golang:

em := endpoints.NewManager(client, "foo/bar/my-service")
err := em.AddEndpoint(context.TODO(), "foo/bar/my-service/e1", endpoints.Endpoint{Addr:"1.2.3.4"})

Atomically updating endpoints

If it’s desired to modify multiple endpoints in a single transaction, endpoints.Manager can be used directly:

em := endpoints.NewManager(c, "foo")

err := em.Update(context.TODO(), []*endpoints.UpdateWithOpts{
    endpoints.NewDeleteUpdateOpts("foo/bar/my-service/e1", endpoints.Endpoint{Addr: "1.2.3.4"}),
	endpoints.NewAddUpdateOpts("foo/bar/my-service/e1", endpoints.Endpoint{Addr: "1.2.3.14"})})

6 - Embedding etcd in a Go Application

The etcd embed go package provides a simple way to embed an etcd server directly into your application.

For more details, see the embed package documentation .

7 - System limits

Request size limit

etcd is designed to handle small key value pairs typical for metadata. Larger requests will work, but may increase the latency of other requests. By default, the maximum size of any request is 1.5 MiB. This limit is configurable through --max-request-bytes flag for etcd server.

Storage size limit

The default storage size limit is 2 GiB, configurable with --quota-backend-bytes flag. 8 GiB is a suggested maximum size for normal environments and etcd warns at startup if the configured value exceeds it.

8 - etcd features

This document provides an overview of etcd features to help users better understand the features and related deprecation process. If you are interested in knowing about how features are developed in the etcd, please see these development guidelines .

The etcd features fall into three stages, experimental, stable, and unsafe. You can get the list of features by running etcd --help.

Experimental

In order to get early feedback, any new feature is usually added as an experimental feature. The experimental feature can be identified by looking at the flag name, which should have --experimental as a prefix. Please consider the following points while using an experimental feature:

• It might be buggy due to a lack of user testing. Enabling the feature may not work as expected.
• It is disabled by default.
• Support for such a feature may be dropped at any time without notice
  • It can be removed in the next minor or major release without following the feature deprecation policy unless it graduates to a stable future.
  • The project team would appreciate users reporting any issues related to experimental features. However, such issues may be given lower priorities compared to the issues related to stable featuers.
• An experimental feature flag deprecates when it graduates to the stable stage. Users should start using a stable feature flag as soon as possible.

Stable

This is the most common stage of features in the etcd. A stable feature is characterized as below:

• Supported as part of the supported releases of etcd.
• May be enabled by default.
• Discontinuation of support must follow the feature deprecation policy.

Unsafe

Unsafe features are rare and listed under the Unsafe feature: section in the etcd usage documentation. By default, they are disabled. They should be used with caution following documentation. An unsafe feature can be removed in the next minor or major release without following the feature deprecation policy.

Feature Deprecation

Experimental

An experimental feature deprecates when it graduates to the stable stage.

• The experimental feature documentation will show a deprecation message with a recommendation to use a related stable feature flag. e.g. DEPRECATED. Use <feature-name> instead.
• A deprecated feature will be removed in the following release.

Stable

As the project evolves, a stable feature may sometimes need to be deprecated and removed. When that happens,

• The feature documentation will show a warning message before a planned release for deprecation. e.g. To be deprecated in <release>.. If a new feature is already planned to replace the To be deprecated feature, then the documentation will also provide a message saying so. e.g. Use <feature-name> instead..
• The feature will be deprecated in the planned release. At that time, the feature documentation will show a deprecation message with a recommendation to use a related stable feature. e.g. DEPRECATED. Use <feature-name> instead.
• A deprecated feature will be removed in the following release.

9 - API reference

This API reference is autogenerated from the named .proto files.

service Auth (api/etcdserverpb/rpc.proto)

service Cluster (api/etcdserverpb/rpc.proto)

service KV (api/etcdserverpb/rpc.proto)

service Lease (api/etcdserverpb/rpc.proto)

service Maintenance (api/etcdserverpb/rpc.proto)

service Watch (api/etcdserverpb/rpc.proto)

message AlarmMember (api/etcdserverpb/rpc.proto)

message AlarmRequest (api/etcdserverpb/rpc.proto)

message AlarmResponse (api/etcdserverpb/rpc.proto)

message AuthDisableRequest (api/etcdserverpb/rpc.proto)

message AuthDisableResponse (api/etcdserverpb/rpc.proto)

message AuthEnableRequest (api/etcdserverpb/rpc.proto)

message AuthEnableResponse (api/etcdserverpb/rpc.proto)

message AuthRoleAddRequest (api/etcdserverpb/rpc.proto)

message AuthRoleAddResponse (api/etcdserverpb/rpc.proto)

message AuthRoleDeleteRequest (api/etcdserverpb/rpc.proto)

message AuthRoleDeleteResponse (api/etcdserverpb/rpc.proto)

message AuthRoleGetRequest (api/etcdserverpb/rpc.proto)

message AuthRoleGetResponse (api/etcdserverpb/rpc.proto)

message AuthRoleGrantPermissionRequest (api/etcdserverpb/rpc.proto)

message AuthRoleGrantPermissionResponse (api/etcdserverpb/rpc.proto)

message AuthRoleListRequest (api/etcdserverpb/rpc.proto)

message AuthRoleListResponse (api/etcdserverpb/rpc.proto)

message AuthRoleRevokePermissionRequest (api/etcdserverpb/rpc.proto)

message AuthRoleRevokePermissionResponse (api/etcdserverpb/rpc.proto)

message AuthStatusRequest (api/etcdserverpb/rpc.proto)

message AuthStatusResponse (api/etcdserverpb/rpc.proto)

message AuthUserAddRequest (api/etcdserverpb/rpc.proto)

message AuthUserAddResponse (api/etcdserverpb/rpc.proto)

message AuthUserChangePasswordRequest (api/etcdserverpb/rpc.proto)

message AuthUserChangePasswordResponse (api/etcdserverpb/rpc.proto)

message AuthUserDeleteRequest (api/etcdserverpb/rpc.proto)

message AuthUserDeleteResponse (api/etcdserverpb/rpc.proto)

message AuthUserGetRequest (api/etcdserverpb/rpc.proto)

message AuthUserGetResponse (api/etcdserverpb/rpc.proto)

message AuthUserGrantRoleRequest (api/etcdserverpb/rpc.proto)

message AuthUserGrantRoleResponse (api/etcdserverpb/rpc.proto)

message AuthUserListRequest (api/etcdserverpb/rpc.proto)

message AuthUserListResponse (api/etcdserverpb/rpc.proto)

message AuthUserRevokeRoleRequest (api/etcdserverpb/rpc.proto)

message AuthUserRevokeRoleResponse (api/etcdserverpb/rpc.proto)

message AuthenticateRequest (api/etcdserverpb/rpc.proto)

message AuthenticateResponse (api/etcdserverpb/rpc.proto)

message CompactionRequest (api/etcdserverpb/rpc.proto)

CompactionRequest compacts the key-value store up to a given revision. All superseded keys with a revision less than the compaction revision will be removed.

message CompactionResponse (api/etcdserverpb/rpc.proto)

message Compare (api/etcdserverpb/rpc.proto)

message DefragmentRequest (api/etcdserverpb/rpc.proto)

message DefragmentResponse (api/etcdserverpb/rpc.proto)

message DeleteRangeRequest (api/etcdserverpb/rpc.proto)

message DeleteRangeResponse (api/etcdserverpb/rpc.proto)

message DowngradeInfo (api/etcdserverpb/rpc.proto)

message DowngradeRequest (api/etcdserverpb/rpc.proto)

message DowngradeResponse (api/etcdserverpb/rpc.proto)

message DowngradeVersionTestRequest (api/etcdserverpb/rpc.proto)

DowngradeVersionTestRequest is used for test only. The version in this request will be read as the WAL record version.If the downgrade target version is less than this version, then the downgrade(online) or migration(offline) isn’t safe, so shouldn’t be allowed.

message HashKVRequest (api/etcdserverpb/rpc.proto)

message HashKVResponse (api/etcdserverpb/rpc.proto)

message HashRequest (api/etcdserverpb/rpc.proto)

message HashResponse (api/etcdserverpb/rpc.proto)

message LeaseCheckpoint (api/etcdserverpb/rpc.proto)

message LeaseCheckpointRequest (api/etcdserverpb/rpc.proto)

message LeaseCheckpointResponse (api/etcdserverpb/rpc.proto)

message LeaseGrantRequest (api/etcdserverpb/rpc.proto)

message LeaseGrantResponse (api/etcdserverpb/rpc.proto)

message LeaseKeepAliveRequest (api/etcdserverpb/rpc.proto)

message LeaseKeepAliveResponse (api/etcdserverpb/rpc.proto)

message LeaseLeasesRequest (api/etcdserverpb/rpc.proto)

message LeaseLeasesResponse (api/etcdserverpb/rpc.proto)

message LeaseRevokeRequest (api/etcdserverpb/rpc.proto)

message LeaseRevokeResponse (api/etcdserverpb/rpc.proto)

message LeaseStatus (api/etcdserverpb/rpc.proto)

message LeaseTimeToLiveRequest (api/etcdserverpb/rpc.proto)

message LeaseTimeToLiveResponse (api/etcdserverpb/rpc.proto)

message Member (api/etcdserverpb/rpc.proto)

message MemberAddRequest (api/etcdserverpb/rpc.proto)

message MemberAddResponse (api/etcdserverpb/rpc.proto)

message MemberListRequest (api/etcdserverpb/rpc.proto)

message MemberListResponse (api/etcdserverpb/rpc.proto)

message MemberPromoteRequest (api/etcdserverpb/rpc.proto)

message MemberPromoteResponse (api/etcdserverpb/rpc.proto)

message MemberRemoveRequest (api/etcdserverpb/rpc.proto)

message MemberRemoveResponse (api/etcdserverpb/rpc.proto)

message MemberUpdateRequest (api/etcdserverpb/rpc.proto)

message MemberUpdateResponse (api/etcdserverpb/rpc.proto)

message MoveLeaderRequest (api/etcdserverpb/rpc.proto)

message MoveLeaderResponse (api/etcdserverpb/rpc.proto)

message PutRequest (api/etcdserverpb/rpc.proto)

message PutResponse (api/etcdserverpb/rpc.proto)

message RangeRequest (api/etcdserverpb/rpc.proto)

message RangeResponse (api/etcdserverpb/rpc.proto)

message RequestOp (api/etcdserverpb/rpc.proto)

message ResponseHeader (api/etcdserverpb/rpc.proto)

message ResponseOp (api/etcdserverpb/rpc.proto)

message SnapshotRequest (api/etcdserverpb/rpc.proto)

message SnapshotResponse (api/etcdserverpb/rpc.proto)

message StatusRequest (api/etcdserverpb/rpc.proto)

message StatusResponse (api/etcdserverpb/rpc.proto)

message TxnRequest (api/etcdserverpb/rpc.proto)

From google paxosdb paper: Our implementation hinges around a powerful primitive which we call MultiOp. All other database operations except for iteration are implemented as a single call to MultiOp. A MultiOp is applied atomically and consists of three components: 1. A list of tests called guard. Each test in guard checks a single entry in the database. It may check for the absence or presence of a value, or compare with a given value. Two different tests in the guard may apply to the same or different entries in the database. All tests in the guard are applied and MultiOp returns the results. If all tests are true, MultiOp executes t op (see item 2 below), otherwise it executes f op (see item 3 below). 2. A list of database operations called t op. Each operation in the list is either an insert, delete, or lookup operation, and applies to a single database entry. Two different operations in the list may apply to the same or different entries in the database. These operations are executed if guard evaluates to true. 3. A list of database operations called f op. Like t op, but executed if guard evaluates to false.

message TxnResponse (api/etcdserverpb/rpc.proto)

message WatchCancelRequest (api/etcdserverpb/rpc.proto)

message WatchCreateRequest (api/etcdserverpb/rpc.proto)

message WatchProgressRequest (api/etcdserverpb/rpc.proto)

Requests the a watch stream progress status be sent in the watch response stream as soon as possible.

message WatchRequest (api/etcdserverpb/rpc.proto)

message WatchResponse (api/etcdserverpb/rpc.proto)

message Event (api/mvccpb/kv.proto)

message KeyValue (api/mvccpb/kv.proto)

message Lease (server/lease/leasepb/lease.proto)

message LeaseInternalRequest (server/lease/leasepb/lease.proto)

message LeaseInternalResponse (server/lease/leasepb/lease.proto)

message Permission (api/authpb/auth.proto)

Permission is a single entity

message Role (api/authpb/auth.proto)

Role is a single entry in the bucket authRoles

message User (api/authpb/auth.proto)

User is a single entry in the bucket authUsers

message UserAddOptions (api/authpb/auth.proto)

10 - API reference: concurrency

This API reference is autogenerated from the named .proto files.

service Lock (server/etcdserver/api/v3lock/v3lockpb/v3lock.proto)

The lock service exposes client-side locking facilities as a gRPC interface.

message LockRequest (server/etcdserver/api/v3lock/v3lockpb/v3lock.proto)

message LockResponse (server/etcdserver/api/v3lock/v3lockpb/v3lock.proto)

message UnlockRequest (server/etcdserver/api/v3lock/v3lockpb/v3lock.proto)

message UnlockResponse (server/etcdserver/api/v3lock/v3lockpb/v3lock.proto)

service Election (server/etcdserver/api/v3election/v3electionpb/v3election.proto)

The election service exposes client-side election facilities as a gRPC interface.

message CampaignRequest (server/etcdserver/api/v3election/v3electionpb/v3election.proto)

message CampaignResponse (server/etcdserver/api/v3election/v3electionpb/v3election.proto)

message LeaderKey (server/etcdserver/api/v3election/v3electionpb/v3election.proto)

message LeaderRequest (server/etcdserver/api/v3election/v3electionpb/v3election.proto)

message LeaderResponse (server/etcdserver/api/v3election/v3electionpb/v3election.proto)

message ProclaimRequest (server/etcdserver/api/v3election/v3electionpb/v3election.proto)

message ProclaimResponse (server/etcdserver/api/v3election/v3electionpb/v3election.proto)

message ResignRequest (server/etcdserver/api/v3election/v3electionpb/v3election.proto)

message ResignResponse (server/etcdserver/api/v3election/v3electionpb/v3election.proto)

message Event (api/mvccpb/kv.proto)

message KeyValue (api/mvccpb/kv.proto)