Task on PIG.CENTER

Usage: pgbouncer command

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

pgbouncer [-d][-R][-v][-u user] <pgbouncer.ini>
pgbouncer -V|-h

On Windows, the options are:

pgbouncer.exe [-v][-u user] <pgbouncer.ini>
pgbouncer.exe -V|-h

Additional options for setting up a Windows service:

pgbouncer.exe --regservice <pgbouncer.ini>
pgbouncer.exe --unregservice <pgbouncer.ini>

Description

pgbouncer is a PostgreSQL connection pooler. Any target application can be connected to pgbouncer as if it were a PostgreSQL server, and pgbouncer will create a connection to the actual server, or it will reuse one of its existing connections.

PostgreSQL Logging Configuration

Mon, 01 Jan 0001 00:00:00 +0000

pgBadger can only report information that PostgreSQL writes to the log. Start with a parseable prefix and a deliberate statement-logging policy, then add the operational events you want to analyze.

Minimum query logging

To include query text and duration, enable duration-based statement logging:

log_min_duration_statement = 0

0 logs every completed statement. On a busy server, choose a higher threshold in milliseconds to control log volume. Measure the overhead and storage growth before enabling a low threshold in production.

Parallel Processing

Mon, 01 Jan 0001 00:00:00 +0000

pgBadger has two complementary multiprocessing modes. Choose according to the shape of the input, not simply the number of CPUs.

Option	Parallel unit	Best fit	Main constraint
`-j N` / `--jobs N`	chunks of one log file	one large, seekable log	chunk boundaries can duplicate or omit a small number of queries
`-J N` / `--Jobs N`	whole log files	many independent logs	useful only when enough files are available to keep workers busy

Split one large file with `-j`

$ pgbadger -j 8 /var/log/postgresql/postgresql.log

The upstream algorithm divides each file into N byte ranges, forks one parser per range, writes temporary binary statistics, then merges those statistics into the final report.

Incremental Reports

Mon, 01 Jan 0001 00:00:00 +0000

Incremental mode stores parsed statistics in binary form, then builds one HTML report per day, a cumulative report per week, and a calendar-style index linking them together. It is intended for repeated processing of rotated logs without counting the same entries again.

Build daily and weekly reports

Run pgBadger after the daily log rotation and provide a persistent output directory:

0 4 * * * /usr/bin/pgbadger -I -q /var/log/postgresql/postgresql.log.1 -O /var/www/pg_reports/

-I enables incremental mode and -O selects the directory that holds the binary state, calendar index, and generated reports. pgBadger maintains its own incremental state in that directory, so --last-parsed is unnecessary unless you deliberately want the state file elsewhere.

Replica imaging and bootstrap

Mon, 01 Jan 0001 00:00:00 +0000

Patroni allows customizing creation of a new replica. It also supports defining what happens when the new empty cluster is being bootstrapped. The distinction between two is well defined: Patroni creates replicas only if the initialize key is present in DCS for the cluster. If there is no initialize key - Patroni calls bootstrap exclusively on the first node that takes the initialize key lock.

Bootstrap

PostgreSQL provides initdb command to initialize a new cluster and Patroni calls it by default. In certain cases, particularly when creating a new cluster as a copy of an existing one, it is necessary to replace a built-in method with custom actions. Patroni supports executing user-defined scripts to bootstrap new clusters, supplying some required arguments to them, i.e. the name of the cluster and the path to the data directory. This is configured in the bootstrap section of the Patroni configuration. For example:

Releases, Packages, and Upgrades

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy is an open source project covered by the GPLv2 license, meaning that everyone is allowed to redistribute it provided that access to the sources is also provided upon request, especially if any modifications were made.

HAProxy evolves as a main development branch called “master” or “mainline”, from which new branches are derived once the code is considered stable. A lot of web sites run some development branches in production on a voluntarily basis, either to participate to the project or because they need a bleeding edge feature, and their feedback is highly valuable to fix bugs and judge the overall quality and stability of the version being developed.

Standby cluster

Mon, 01 Jan 0001 00:00:00 +0000

Patroni also support running cascading replication to a remote datacenter (region) using a feature that is called “standby cluster”. This type of clusters has:

“standby leader”, that behaves pretty much like a regular cluster leader, except it replicates from a remote node.
cascade replicas, that are replicating from standby leader.

Standby leader holds and updates a leader lock in DCS. If the leader lock expires, cascade replicas will perform an election to choose another leader from the standbys.

Tasks

Mon, 01 Jan 0001 00:00:00 +0000

Operator Tasks

Mon, 01 Jan 0001 00:00:00 +0000

How to Set Up a Demo etcd Cluster

Mon, 01 Jan 0001 00:00:00 +0000

On each etcd node, specify the cluster members:

TOKEN=token-01
CLUSTER_STATE=new
NAME_1=machine-1
NAME_2=machine-2
NAME_3=machine-3
HOST_1=10.240.0.17
HOST_2=10.240.0.18
HOST_3=10.240.0.19
CLUSTER=${NAME_1}=http://${HOST_1}:2380,${NAME_2}=http://${HOST_2}:2380,${NAME_3}=http://${HOST_3}:2380

Run this on each machine:

# For machine 1
THIS_NAME=${NAME_1}
THIS_IP=${HOST_1}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
	--initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://${THIS_IP}:2380 \
	--advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://${THIS_IP}:2379 \
	--initial-cluster ${CLUSTER} \
	--initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}

# For machine 2
THIS_NAME=${NAME_2}
THIS_IP=${HOST_2}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
	--initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://${THIS_IP}:2380 \
	--advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://${THIS_IP}:2379 \
	--initial-cluster ${CLUSTER} \
	--initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}

# For machine 3
THIS_NAME=${NAME_3}
THIS_IP=${HOST_3}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
	--initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://${THIS_IP}:2380 \
	--advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://${THIS_IP}:2379 \
	--initial-cluster ${CLUSTER} \
	--initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}

Or use our public discovery service:

Pause/Resume mode for the cluster

Mon, 01 Jan 0001 00:00:00 +0000

The goal

Under certain circumstances Patroni needs to temporarily step down from managing the cluster, while still retaining the cluster state in DCS. Possible use cases are uncommon activities on the cluster, such as major version upgrades or corruption recovery. During those activities nodes are often started and stopped for reasons unknown to Patroni, some nodes can be even temporarily promoted, violating the assumption of running only one primary. Therefore, Patroni needs to be able to “detach” from the running cluster, implementing an equivalent of the maintenance mode in Pacemaker.

Support and Contributing

Mon, 01 Jan 0001 00:00:00 +0000

pgBadger is maintained as an open project. Bug reports, feature proposals, documentation fixes, and patches are handled through the upstream GitHub repository.

Bugs and feature requests

Upgrade to the newest released version and confirm the behavior still occurs.
Search open issues and closed issues for an existing answer.
Reduce the problem to the smallest safe log sample and command line that still reproduces it.
Remove credentials, sensitive SQL, bind values, host names, addresses, and business data.
Open a new issue with the pgBadger version, operating system, input format, exact options, observed result, and expected result.

For crashes or parser mistakes, include only the minimum sanitized log lines needed to reproduce the boundary. A complete production log or generated report is rarely appropriate for a public issue.

Using Patroni with Kubernetes

Mon, 01 Jan 0001 00:00:00 +0000

Patroni can use Kubernetes objects in order to store the state of the cluster and manage the leader key. That makes it capable of operating Postgres in Kubernetes environment without any consistency store, namely, one doesn’t need to run an extra Etcd deployment. There are two different type of Kubernetes objects Patroni can use to store the leader and the configuration keys, they are configured with the kubernetes.use_endpoints or PATRONI_KUBERNETES_USE_ENDPOINTS environment variable.

Convert a Standalone to a Patroni Cluster

Mon, 01 Jan 0001 00:00:00 +0000

This section describes the process for converting a standalone PostgreSQL instance into a Patroni cluster.

To deploy a Patroni cluster without using a pre-existing PostgreSQL instance, see Running and Configuring instead.

Procedure

You can find below an overview of steps for converting an existing Postgres cluster to a Patroni managed cluster. In the steps we assume all nodes that are part of the existing cluster are currently up and running, and that you do not intend to change Postgres configuration while the migration is ongoing. The steps:

Developer Tasks

Mon, 01 Jan 0001 00:00:00 +0000

8. Logging

Mon, 01 Jan 0001 00:00:00 +0000

One of HAProxy’s strong points certainly lies is its precise logs. It probably provides the finest level of information available for such a product, which is very important for troubleshooting complex environments. Standard information provided in logs include client ports, TCP/HTTP state timers, precise stream state at termination and precise termination cause, information about decisions to direct traffic to a server, and of course the ability to capture arbitrary headers.

Contributing guidelines

Mon, 01 Jan 0001 00:00:00 +0000

Chatting

If you have a question, looking for an interactive troubleshooting help or want to chat with other Patroni users, join us on channel #patroni in the PostgreSQL Slack .

Reporting bugs

Before reporting a bug please make sure to reproduce it with the latest Patroni version! Also please double check if the issue already exists in our Issues Tracker .

Running tests

Requirements for running behave tests:

Reading from etcd

Mon, 01 Jan 0001 00:00:00 +0000

Prerequisites

Install etcdctl

Procedure

Use the get subcommand to read from etcd:

$ etcdctl --endpoints=$ENDPOINTS get foo
foo
Hello World!
$

where:

foo is the requested key
Hello World! is the retrieved value

Or, for formatted output:

$ etcdctl --endpoints=$ENDPOINTS --write-out="json" get foo
{"header":{"cluster_id":289318470931837780,"member_id":14947050114012957595,"revision":3,"raft_term":4,
"kvs":[{"key":"Zm9v","create_revision":2,"mod_revision":3,"version":2,"value":"SGVsbG8gV29ybGQh"}]}}
$

where write-out="json" causes the value to be output in JSON format (note that the key is not returned).

1. Prerequisites

Mon, 01 Jan 0001 00:00:00 +0000

This document describes how to start, stop, manage, and troubleshoot HAProxy, as well as some known limitations and traps to avoid. It does not describe how to configure it (for this please read configuration.txt ).

In this document it is assumed that the reader has sufficient administration skills on a UNIX-like operating system, uses the shell on a daily basis and is familiar with troubleshooting utilities such as strace and tcpdump.

Writing to etcd

Mon, 01 Jan 0001 00:00:00 +0000

Prerequisites

Install etcdctl

Procedure

Use the put subcommand to write a key-value pair:

etcdctl --endpoints=$ENDPOINTS put foo "Hello World!"

where:

foo is the key name
"Hello World!" is the quote-delimited value

3. Starting HAProxy

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy is started by invoking the “haproxy” program with a number of arguments passed on the command line. The actual syntax is:

$ haproxy [<options>]*

where [<options>]* is any number of options. An option always starts with ‘-’ followed by one of more letters, and possibly followed by one or multiple extra arguments. Without any option, HAProxy displays the help page with a reminder about supported options. Available options may vary slightly based on the operating system. A fair number of these options overlap with an equivalent one in the “global” section. In this case, the command line always has precedence over the configuration file, so that the command line can be used to quickly enforce some settings without touching the configuration files. The current list of options is:

4. Stopping and Restarting HAProxy

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy supports a graceful and a hard stop. The hard stop is simple, when the SIGTERM signal is sent to the haproxy process, it immediately quits and all established connections are closed. The graceful stop is triggered when the SIGUSR1 signal is sent to the haproxy process. It consists in only unbinding from listening ports, but continue to process existing connections until they close. Once the last connection is closed, the process leaves.

5. File-Descriptor Limitations

Mon, 01 Jan 0001 00:00:00 +0000

In order to ensure that all incoming connections will successfully be served, HAProxy computes at load time the total number of file descriptors that will be needed during the process’s life. A regular Unix process is generally granted 1024 file descriptors by default, and a privileged process can raise this limit itself. This is one reason for starting HAProxy as root and letting it adjust the limit. The default limit of 1024 file descriptors roughly allow about 500 concurrent connections to be processed. The computation is based on the global maxconn parameter which limits the total number of connections per process, the number of listeners, the number of servers which have a health check enabled, the agent checks, the peers, the loggers and possibly a few other technical requirements. A simple rough estimate of this number consists in simply doubling the maxconn value and adding a few tens to get the approximate number of file descriptors needed.

6. Memory Management

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy uses a simple and fast pool-based memory management. Since it relies on a small number of different object types, it’s much more efficient to pick new objects from a pool which already contains objects of the appropriate size than to call malloc() for each different size. The pools are organized as a stack or LIFO, so that newly allocated objects are taken from recently released objects still hot in the CPU caches. Pools of similar sizes are merged together, in order to limit memory fragmentation.

How to get keys by prefix

Mon, 01 Jan 0001 00:00:00 +0000

Pre-requisites

Get keys by prefix

$ etcdctl --endpoints=$ENDPOINTS get PREFIX --prefix

Global Options

--endpoints=[127.0.0.1:2379], gRPC endpoints

Options

--prefix, get a range of keys with matching prefix

Example

etcdctl --endpoints=$ENDPOINTS put web1 value1
etcdctl --endpoints=$ENDPOINTS put web2 value2
etcdctl --endpoints=$ENDPOINTS put web3 value3

etcdctl --endpoints=$ENDPOINTS get web --prefix

7. CPU Usage

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy normally spends most of its time in the system and a smaller part in userland. A finely tuned 3.5 GHz CPU can sustain a rate about 80000 end-to-end connection setups and closes per second at 100% CPU on a single core. When one core is saturated, typical figures are:

95% system, 5% user for long TCP connections or large HTTP objects
85% system and 15% user for short TCP connections or small HTTP objects in close mode
70% system and 30% user for small HTTP objects in keep-alive mode

The amount of rules processing and regular expressions will increase the user land part. The presence of firewall rules, connection tracking, complex routing tables in the system will instead increase the system part.

8. Logging

Mon, 01 Jan 0001 00:00:00 +0000

For logging, HAProxy always relies on a syslog server since it does not perform any file-system access. The standard way of using it is to send logs over UDP to the log server (by default on port 514). Very commonly this is configured to 127.0.0.1 where the local syslog daemon is running, but it’s also used over the network to log to a central server. The central server provides additional benefits especially in active-active scenarios where it is desirable to keep the logs merged in arrival order. HAProxy may also make use of a UNIX socket to send its logs to the local syslog daemon, but it is not recommended at all, because if the syslog server is restarted while haproxy runs, the socket will be replaced and new logs will be lost. Since HAProxy will be isolated inside a chroot jail, it will not have the ability to reconnect to the new socket. It has also been observed in field that the log buffers in use on UNIX sockets are very small and lead to lost messages even at very light loads. But this can be fine for testing however.

9. Statistics and Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

It is possible to query HAProxy about its status. The most commonly used mechanism is the HTTP statistics page. This page also exposes an alternative CSV output format for monitoring tools. The same format is provided on the Unix socket.

Statistics are regroup in categories labelled as domains, corresponding to the multiple components of HAProxy. There are two domains available: proxy and resolvers. If not specified, the proxy domain is selected. Note that only the proxy statistics are printed on the HTTP page.

10. Easier Configuration Management

Mon, 01 Jan 0001 00:00:00 +0000

It is very common that two HAProxy nodes constituting a cluster share exactly the same configuration modulo a few addresses. Instead of having to maintain a duplicate configuration for each node, which will inevitably diverge, it is possible to include environment variables in the configuration. Thus multiple configuration may share the exact same file with only a few different system wide environment variables. This started in version 1.5 where only addresses were allowed to include environment variables, and 1.6 goes further by supporting environment variables everywhere. The syntax is the same as in the UNIX shell, a variable starts with a dollar sign (’$’), followed by an opening curly brace (’{’), then the variable name followed by the closing brace (’}’). Except for addresses, environment variables are only interpreted in arguments surrounded with double quotes (this was necessary not to break existing setups using regular expressions involving the dollar symbol).

11. Well-Known Traps to Avoid

Mon, 01 Jan 0001 00:00:00 +0000

Once in a while, someone reports that after a system reboot, the haproxy service wasn’t started, and that once they start it by hand it works. Most often, these people are running a clustered IP address mechanism such as keepalived, to assign the service IP address to the master node only, and while it used to work when they used to bind haproxy to address 0.0.0.0, it stopped working after they bound it to the virtual IP address. What happens here is that when the service starts, the virtual IP address is not yet owned by the local node, so when HAProxy wants to bind to it, the system rejects this because it is not a local IP address. The fix doesn’t consist in delaying the haproxy service startup (since it wouldn’t stand a restart), but instead to properly configure the system to allow binding to non-local addresses. This is easily done on Linux by setting the net.ipv4.ip_nonlocal_bind sysctl to 1. This is also needed in order to transparently intercept the IP traffic that passes through HAProxy for a specific target address.

12. Debugging and Performance Issues

Mon, 01 Jan 0001 00:00:00 +0000

When HAProxy is started with the “-d” option, it will stay in the foreground and will print one line per event, such as an incoming connection, the end of a connection, and for each request or response header line seen. This debug output is emitted before the contents are processed, so they don’t consider the local modifications. The main use is to show the request and response without having to run a network sniffer. The output is less readable when multiple connections are handled in parallel, though the “debug2ansi” and “debug2html” scripts found in the examples/ directory definitely help here by coloring the output.

13. Security Considerations

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy is designed to run with very limited privileges. The standard way to use it is to isolate it into a chroot jail and to drop its privileges to a non-root user without any permissions inside this jail so that if any future vulnerability were to be discovered, its compromise would not affect the rest of the system.

In order to perform a chroot, it first needs to be started as a root user. It is pointless to build hand-made chroots to start the process there, these ones are painful to build, are never properly maintained and always contain way more bugs than the main file-system. And in case of compromise, the intruder can use the purposely built file-system. Unfortunately many administrators confuse “start as root” and “run as root”, resulting in the uid change to be done prior to starting haproxy, and reducing the effective security restrictions.

How to delete keys

Mon, 01 Jan 0001 00:00:00 +0000

Prerequisites

Install etcd and etcdctl

Add or delete keys

del to remove the specified key or range of keys:

etcdctl del $KEY [$END_KEY]

Options

--prefix[=false]: delete keys with matching prefix
--prev-kv[=false]: return deleted key-value pairs
--from-key[=false]: delete keys that are greater than or equal to the given key using byte compare
--range[=false]: delete range of keys without delay

Options inherited from parent commands

--endpoints="127.0.0.1:2379": gRPC endpoints

Examples

etcdctl --endpoints=$ENDPOINTS put key myvalue
etcdctl --endpoints=$ENDPOINTS del key

etcdctl --endpoints=$ENDPOINTS put k1 value1
etcdctl --endpoints=$ENDPOINTS put k2 value2
etcdctl --endpoints=$ENDPOINTS del k --prefix

How to make multiple writes in a transaction

Mon, 01 Jan 0001 00:00:00 +0000

Prerequisites

Install etcd and etcdctl .
A running etcd cluster.

Terminology

Here are definitions of some key terms used in the Example below.

Terms	Definition
etcdctl	The command line tool for interacting with the etcd server.
`txn` command	`txn` command is an abbreviation for “transaction”. It reads multiple etcd requests from standard input and applies them as a single atomic transaction. A transaction consists of list of conditions, a list of requests to apply if all the conditions are true, and a list of requests to apply if any condition is false. View etcdctl key-value commands for more information.
`compare`	The `compare` clause within a transaction (`txn`) serves as a conditional check that determines whether the transaction’s operations should proceed. It ensures changes are only applied if the current state of the key-value store matches expected conditions, thereby maintaining data consistency and preventing conflicts in concurrent environments. To see how the command is structured, view Perform a transaction section below.

Transactions

txn to process all the requests in one transaction:

How to watch keys

Mon, 01 Jan 0001 00:00:00 +0000

Prerequisites

Install etcd and etcdctl

Watching keys

watch to get notified of future changes:

etcdctl watch $KEY [$END_KEY]

Options

-i, --interactive[=false]: interactive mode
--prefix[=false]: watch on a prefix if prefix is set
--rev=0: Revision to start watching
--prev-kv[=false]: get the previous key-value pair before the event happens
--progress-notify[=false]: get periodic watch progress notification from server

Options inherited from parent commands

--endpoints="127.0.0.1:2379": gRPC endpoints

Examples

etcdctl --endpoints=$ENDPOINTS watch stock1
etcdctl --endpoints=$ENDPOINTS put stock1 1000

etcdctl --endpoints=$ENDPOINTS watch stock --prefix
etcdctl --endpoints=$ENDPOINTS put stock1 10
etcdctl --endpoints=$ENDPOINTS put stock2 20

How to create lease

Mon, 01 Jan 0001 00:00:00 +0000

lease to write with TTL:

etcdctl --endpoints=$ENDPOINTS lease grant 300
# lease 2be7547fbc6a5afa granted with TTL(300s)

etcdctl --endpoints=$ENDPOINTS put sample value --lease=2be7547fbc6a5afa
etcdctl --endpoints=$ENDPOINTS get sample

etcdctl --endpoints=$ENDPOINTS lease keep-alive 2be7547fbc6a5afa
etcdctl --endpoints=$ENDPOINTS lease revoke 2be7547fbc6a5afa
# or after 300 seconds
etcdctl --endpoints=$ENDPOINTS get sample

How to create locks

Mon, 01 Jan 0001 00:00:00 +0000

LOCK acquires a distributed mutex with a given name. Once the lock is acquired, it will be held until etcdctl is terminated.

Prerequisites

Install etcd and etcdctl

Creating a lock

lock for distributed lock:

etcdctl --endpoints=$ENDPOINTS lock mutex1

Options

endpoints - defines a comma-delimited list of machine addresses in the cluster.
ttl - time out in seconds of lock session.

How to conduct leader election in etcd cluster

Mon, 01 Jan 0001 00:00:00 +0000

Prerequisites

Ensure etcd and etcdctl is installed.
Check for active etcd cluster.

Conduct Leader election

The etcdctl command is used to conduct leader elections in an etcd cluster. It makes sure that only one client can become leader at a time.

etcdctl --endpoints=$ENDPOINTS elect <election-name> [proposal]

etcdctl --endpoints=$ENDPOINTS elect election-name p1

Options

--endpoints : $ENDPOINTS

Address of each etcd cluster members.

How to check Cluster status

Mon, 01 Jan 0001 00:00:00 +0000

Prerequisites

Install etcd and etcdctl

Check Overall Status

endpoint status to check the overall status of each endpoint specified in --endpoints flag:

etcdctl endpoint status (--endpoints=$ENDPOINTS|--cluster)

Options

--cluster[=false]: use all endpoints from the cluster member list

Check Health

endpoint health to check the healthiness of each endpoint specified in --endpoints flag:

How to save the database

Mon, 01 Jan 0001 00:00:00 +0000

Pre-requisites

Snapshot a database

snapshot to save point-in-time snapshot of etcd database:

etcdctl --endpoints=$ENDPOINT snapshot save DB_NAME

Global Options

etcdctl

--endpoints=[127.0.0.1:2379], gRPC endpoints

Snapshot can only be requested from one etcd node, so --endpoints flag should contain only one endpoint.

Install

Mon, 01 Jan 0001 00:00:00 +0000

Requirements

Before installing etcd, see the following pages:

Install pre-built binaries

The easiest way to install etcd is from pre-built binaries:

Download the compressed archive file for your platform from Releases , choosing release v3.7.0 or later.
Unpack the archive file. This results in a directory containing the binaries.
Add the executable binaries to your path. For example, rename and/or move the binaries to a directory in your path (like /usr/local/bin), or add the directory created by the previous step to your path.

How to Add and Remove Members

Mon, 01 Jan 0001 00:00:00 +0000

member to add,remove,update membership:

# For each machine
TOKEN=my-etcd-token-1
CLUSTER_STATE=new
NAME_1=etcd-node-1
NAME_2=etcd-node-2
NAME_3=etcd-node-3
HOST_1=10.240.0.13
HOST_2=10.240.0.14
HOST_3=10.240.0.15
CLUSTER=${NAME_1}=http://${HOST_1}:2380,${NAME_2}=http://${HOST_2}:2380,${NAME_3}=http://${HOST_3}:2380

# For node 1
THIS_NAME=${NAME_1}
THIS_IP=${HOST_1}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
	--initial-advertise-peer-urls http://${THIS_IP}:2380 \
	--listen-peer-urls http://${THIS_IP}:2380 \
	--advertise-client-urls http://${THIS_IP}:2379 \
	--listen-client-urls http://${THIS_IP}:2379 \
	--initial-cluster ${CLUSTER} \
	--initial-cluster-state ${CLUSTER_STATE} \
	--initial-cluster-token ${TOKEN}

# For node 2
THIS_NAME=${NAME_2}
THIS_IP=${HOST_2}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
	--initial-advertise-peer-urls http://${THIS_IP}:2380 \
	--listen-peer-urls http://${THIS_IP}:2380 \
	--advertise-client-urls http://${THIS_IP}:2379 \
	--listen-client-urls http://${THIS_IP}:2379 \
	--initial-cluster ${CLUSTER} \
	--initial-cluster-state ${CLUSTER_STATE} \
	--initial-cluster-token ${TOKEN}

# For node 3
THIS_NAME=${NAME_3}
THIS_IP=${HOST_3}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
	--initial-advertise-peer-urls http://${THIS_IP}:2380 \
	--listen-peer-urls http://${THIS_IP}:2380 \
	--advertise-client-urls http://${THIS_IP}:2379 \
	--listen-client-urls http://${THIS_IP}:2379 \
	--initial-cluster ${CLUSTER} \
	--initial-cluster-state ${CLUSTER_STATE} \
	--initial-cluster-token ${TOKEN}

Then replace a member with member remove and member add commands:

Reporting bugs

Mon, 01 Jan 0001 00:00:00 +0000

If any part of the etcd project has bugs or documentation mistakes, please let us know by opening an issue . We treat bugs and mistakes very seriously and believe no issue is too small. Before creating a bug report, please check that an issue reporting the same problem does not already exist.

To make the bug report accurate and easy to understand, please try to create bug reports that are:

Tuning

Mon, 01 Jan 0001 00:00:00 +0000

The default settings in etcd should work well for installations on a local network where the average network latency is low. However, when using etcd across multiple data centers or over networks with high latency, the heartbeat interval and election timeout settings may need tuning.

The network isn’t the only source of latency. Each request and response may be impacted by slow disks on both the leader and follower. Each of these timeouts represents the total time from request to successful response from the other machine.

Developer guide

Mon, 01 Jan 0001 00:00:00 +0000

Interacting with etcd

Mon, 01 Jan 0001 00:00:00 +0000

Users mostly interact with etcd by putting or getting the value of a key. This section describes how to do that by using etcdctl, a command line tool for interacting with etcd server. The concepts described here should apply to the gRPC APIs or client library APIs.

The API version used by etcdctl to speak to etcd may be set to version 2 or 3 via the ETCDCTL_API environment variable. By default, etcdctl on master (3.4) uses the v3 API and earlier versions (3.3 and earlier) default to the v2 API.

gRPC naming and discovery

Mon, 01 Jan 0001 00:00:00 +0000

etcd provides a gRPC resolver to support an alternative name system that fetches endpoints from etcd for discovering gRPC services. The underlying mechanism is based on watching updates to keys prefixed with the service name.

Note that this feature is experimental because it depends on the google.golang.org/grpc/resolver package, which is still experimental in grpc-go.

Using etcd discovery with go-grpc

The etcd client provides a gRPC resolver for resolving gRPC endpoints with an etcd backend. The resolver is initialized with an etcd client:

Embedding etcd in a Go Application

Mon, 01 Jan 0001 00:00:00 +0000

The etcd embed go package provides a simple way to embed an etcd server directly into your application.

For more details, see the embed package documentation .

Authentication Guides

Mon, 01 Jan 0001 00:00:00 +0000

Operations guide

Mon, 01 Jan 0001 00:00:00 +0000

Authentication

Mon, 01 Jan 0001 00:00:00 +0000

auth,user,role for authentication:

export ETCDCTL_API=3
ENDPOINTS=localhost:2379

etcdctl --endpoints=${ENDPOINTS} role add root
etcdctl --endpoints=${ENDPOINTS} role get root

etcdctl --endpoints=${ENDPOINTS} user add root
etcdctl --endpoints=${ENDPOINTS} user grant-role root root
etcdctl --endpoints=${ENDPOINTS} user get root

etcdctl --endpoints=${ENDPOINTS} role add role0
etcdctl --endpoints=${ENDPOINTS} role grant-permission role0 readwrite foo
etcdctl --endpoints=${ENDPOINTS} user add user0
etcdctl --endpoints=${ENDPOINTS} user grant-role user0 role0

etcdctl --endpoints=${ENDPOINTS} auth enable
# now all client requests go through auth

etcdctl --endpoints=${ENDPOINTS} --user=user0:123 put foo bar
etcdctl --endpoints=${ENDPOINTS} get foo
# permission denied, user name is empty because the request does not issue an authentication request
etcdctl --endpoints=${ENDPOINTS} --user=user0:123 get foo
# user0 can read the key foo
etcdctl --endpoints=${ENDPOINTS} --user=user0:123 get foo1

Note:

This is just a stub which needs to be filled and updated with more information on authentication. The text above is just a code example.

Role-based access control

Mon, 01 Jan 0001 00:00:00 +0000

Overview

Authentication was added in etcd 2.1. The etcd v3 API slightly modified the authentication feature’s API and user interface to better fit the new data model. This guide is intended to help users set up basic authentication and role-based access control in etcd v3.

Special users and roles

There is one special user, root, and one special role, root.

Transport security model

Mon, 01 Jan 0001 00:00:00 +0000

etcd supports automatic TLS as well as authentication through client certificates for both clients to server as well as peer (server to server / cluster) communication. Note that etcd doesn’t enable RBAC based authentication or the authentication feature in the transport layer by default to reduce friction for users getting started with the database. Further, changing this default would be a breaking change for the project which was established since 2013. An etcd cluster which doesn’t enable security features can expose its data to any clients.

Clustering Guide

Mon, 01 Jan 0001 00:00:00 +0000

Overview

Starting an etcd cluster statically requires that each member knows another in the cluster. In a number of cases, the IPs of the cluster members may be unknown ahead of time. In these cases, the etcd cluster can be bootstrapped with the help of a discovery service.

Once an etcd cluster is up and running, adding or removing members is done via runtime reconfiguration . To better understand the design behind runtime reconfiguration, we suggest reading the runtime configuration design document .

Run etcd clusters as a Kubernetes StatefulSet

Mon, 01 Jan 0001 00:00:00 +0000

Below demonstrates how to perform the static bootstrap process as a Kubernetes StatefulSet.

Example Manifest

This manifest contains a service and statefulset for deploying a static etcd cluster in kubernetes.

If you copy the contents of the manifest into a file named etcd.yaml, it can be applied to a cluster with this command.

$ kubectl apply --filename etcd.yaml

Upon being applied, wait for the pods to become ready.

Run etcd clusters inside containers

Mon, 01 Jan 0001 00:00:00 +0000

The following guide shows how to run etcd with Docker using the static bootstrap process .

Docker

In order to expose the etcd API to clients outside of Docker host, use the host IP address of the container. Please see docker inspect for more detail on how to get the IP address. Alternatively, specify --net=host flag to docker run command to skip placing the container inside of a separate network stack.

Disaster recovery

Mon, 01 Jan 0001 00:00:00 +0000

etcd is designed to withstand machine failures. An etcd cluster automatically recovers from temporary failures (e.g., machine reboots) and tolerates up to (N-1)/2 permanent failures for a cluster of N members. When a member permanently fails, whether due to hardware failure or disk corruption, it loses access to the cluster. If the cluster permanently loses more than (N-1)/2 members then it disastrously fails, irrevocably losing quorum. Once quorum is lost, the cluster cannot reach consensus and therefore cannot continue accepting updates.

etcd gateway

Mon, 01 Jan 0001 00:00:00 +0000

What is etcd gateway

etcd gateway is a simple TCP proxy that forwards network data to the etcd cluster. The gateway is stateless and transparent; it neither inspects client requests nor interferes with cluster responses. It does not terminate TLS connections, do TLS handshakes on behalf of its clients, or verify if the connection is secured.

The gateway supports multiple etcd server endpoints and works on a simple round-robin policy. It only routes to available endpoints and hides failures from its clients. Other retry policies, such as weighted round-robin, may be supported in the future.

gRPC proxy

Mon, 01 Jan 0001 00:00:00 +0000

The gRPC proxy is a stateless etcd reverse proxy operating at the gRPC layer (L7). The proxy is designed to reduce the total processing load on the core etcd cluster. For horizontal scalability, it coalesces watch and lease API requests. To protect the cluster against abusive clients, it caches key range requests.

The gRPC proxy supports multiple etcd server endpoints. When the proxy starts, it randomly picks one etcd server endpoint to use. This endpoint serves all requests until the proxy detects an endpoint failure. If the gRPC proxy detects an endpoint failure, it switches to a different endpoint, if available, to hide failures from its clients. Other retry policies, such as weighted round-robin, may be supported in the future.

Maintenance

Mon, 01 Jan 0001 00:00:00 +0000

Overview

An etcd cluster needs periodic maintenance to remain reliable. Depending on an etcd application’s needs, this maintenance can usually be automated and performed without downtime or significantly degraded performance.

All etcd maintenance manages storage resources consumed by the etcd keyspace. Failure to adequately control the keyspace size is guarded by storage space quotas; if an etcd member runs low on space, a quota will trigger cluster-wide alarms which will put the system into a limited-operation maintenance mode. To avoid running out of space for writes to the keyspace, the etcd keyspace history must be compacted. Storage space itself may be reclaimed by defragmenting etcd members. Finally, periodic snapshot backups of etcd member state makes it possible to recover any unintended logical data loss or corruption caused by operational error.

Monitoring etcd

Mon, 01 Jan 0001 00:00:00 +0000

Each etcd server provides local monitoring information on its client port through http endpoints. The monitoring data is useful for both system health checking and cluster debugging.

Debug endpoint

If --log-level=debug is set, the etcd server exports debugging information on its client port under the /debug path. Take care when setting --log-level=debug, since there will be degraded performance and verbose logging.

The /debug/pprof endpoint is the standard go runtime profiling endpoint. This can be used to profile CPU, heap, mutex, and goroutine utilization. For example, here go tool pprof gets the top 10 functions where etcd spends its time:

Runtime reconfiguration

Mon, 01 Jan 0001 00:00:00 +0000

etcd comes with support for incremental runtime reconfiguration, which allows users to update the membership of the cluster at run time.

Reconfiguration requests can only be processed when a majority of cluster members are functioning. It is highly recommended to always have a cluster size greater than two in production. It is unsafe to remove a member from a two member cluster. The majority of a two member cluster is also two. If there is a failure during the removal process, the cluster might not be able to make progress and need to restart from majority failure .

Data Corruption

Mon, 01 Jan 0001 00:00:00 +0000

etcd has built in automated data corruption detection to prevent member state from diverging.

Enabling data corruption detection

Data corruption detection can be done using:

Initial check, enabled with --experimental-initial-corrupt-check flag.
Periodic check of:
- Compacted revision hash, enabled with --experimental-compact-hash-check-enabled flag.
- Latest revision hash, enabled with --experimental-corrupt-check-time flag.

Initial check will be executed during bootstrap of etcd member. Member will compare its persistent state vs other members and exit if there is a mismatch.

Downgrading

Mon, 01 Jan 0001 00:00:00 +0000

Upgrading

Mon, 01 Jan 0001 00:00:00 +0000

Downgrading etcd clusters and applications

Mon, 01 Jan 0001 00:00:00 +0000

This section contains documents specific to downgrading etcd clusters and applications.

Downgrading an etcd v3.x cluster

Upgrading etcd clusters and applications

Mon, 01 Jan 0001 00:00:00 +0000

This section contains documents specific to upgrading etcd clusters and applications.

Upgrade policy

Before upgrading, note that etcd only supports the following two upgrade cases:

Patch upgrade: Upgrading between patch releases within the same minor version (e.g. 3.7.0 - 3.7.1).
Minor upgrade: Upgrading one minor version at a time (e.g. 3.6 - 3.7). Upgrades that skip a minor version are not supported and will likely fail. Update to the most recent patch version before upgrading to the next minor version.

Upgrading an etcd v3.x cluster

Upgrading from etcd v2.3

Upgrade a v2.3 cluster to v3.0

Downgrade etcd from v3.7 to v3.6

Mon, 01 Jan 0001 00:00:00 +0000

In the general case, downgrading from etcd v3.7 to v3.6 can be a zero-downtime, rolling downgrade:

one by one, stop the etcd v3.7 processes and replace them with etcd v3.6 processes
after enabling the downgrade, new features in v3.7 are no longer available to the cluster

Before starting a downgrade , read through the rest of this guide to prepare.

Downgrade checklists

Highlighted differences between v3.7 and v3.6:

Upgrade etcd from v3.5 to v3.6

Mon, 01 Jan 0001 00:00:00 +0000

In the general case, upgrading from etcd v3.5 to v3.6 can be a zero-downtime, rolling upgrade:

one by one, stop the etcd v3.5 processes and replace them with etcd v3.6 processes
after running all v3.6 processes, new features in v3.6 are available to the cluster

Before starting an upgrade , read through the rest of this guide to prepare.

Upgrade checklists

Update 3.5

Important

Before upgrading to 3.6, make sure that all of your 3.5 members are updated to 3.5.32 or later . Patch releases 3.5.24 through 3.5.26 fix several potential upgrade blockers; 3.5.32 adds --v2-deprecation=write-only-skip-check and extends etcdutl check v2store to inspect WAL records as well as the v2 snapshot.

Downgrade etcd from 3.5 to 3.4

Mon, 01 Jan 0001 00:00:00 +0000

In the general case, downgrading from etcd 3.5 to 3.4 can be a zero-downtime, rolling downgrade:

one by one, stop the etcd 3.5 processes and replace them with etcd 3.4 processes
after starting any 3.4 processes, new features in 3.5 are not longer available to the cluster

Before starting a downgrade , read through the rest of this guide to prepare.

Downgrade checklists

content/enhttps://etcd.io/docs/v3.5/op-guide/authentication/rbac.md

Warning

If your cluster enables auth, rolling downgrade from 3.5 isn’t supported because 3.5 changes a format of WAL entries related to auth . You can follow the authentification instructions to disable auth, and delete all users first.

Downgrade etcd from v3.6 to v3.5

Mon, 01 Jan 0001 00:00:00 +0000

In the general case, downgrading from etcd v3.6 to v3.5 can be a zero-downtime, rolling downgrade:

one by one, stop the etcd v3.6 processes and replace them with etcd v3.5 processes
after enabling the downgrade, new features in v3.6 are no longer available to the cluster

Before starting a downgrade , read through the rest of this guide to prepare.

Downgrade checklists

Highlighted breaking changes from v3.6 to v3.5:

Upgrade etcd from 3.4 to 3.5

Mon, 01 Jan 0001 00:00:00 +0000

In the general case, upgrading from etcd 3.4 to 3.5 can be a zero-downtime, rolling upgrade:

one by one, stop the etcd v3.4 processes and replace them with etcd v3.5 processes
after running all v3.5 processes, new features in v3.5 are available to the cluster

Before starting an upgrade , read through the rest of this guide to prepare.

Upgrade checklists

Warning

When migrating from v2 with no v3 data , etcd server v3.2+ panics when etcd restores from existing snapshots but no v3 ETCD_DATA_DIR/member/snap/db file. This happens when the server had migrated from v2 with no previous v3 data. This also prevents accidental v3 data loss (e.g. db file might have been moved). etcd requires that post v3 migration can only happen with v3 data. Do not upgrade to newer v3 versions until v3.0 server contains v3 data.

Upgrade etcd from 3.3 to 3.4

Mon, 01 Jan 0001 00:00:00 +0000

In the general case, upgrading from etcd 3.3 to 3.4 can be a zero-downtime, rolling upgrade:

one by one, stop the etcd v3.3 processes and replace them with etcd v3.4 processes
after running all v3.4 processes, new features in v3.4 are available to the cluster

Before starting an upgrade , read through the rest of this guide to prepare.

Upgrade checklists

Warning

Upgrade etcd from v3.6 to v3.7

Mon, 01 Jan 0001 00:00:00 +0000

In the general case, upgrading from etcd v3.6 to v3.7 can be a zero-downtime, rolling upgrade:

one by one, stop the etcd v3.6 processes and replace them with etcd v3.7 processes
after running all v3.7 processes, new features in v3.7 are available to the cluster

Before starting an upgrade , read through the rest of this guide to prepare.

Upgrade checklists

Update 3.6

Important

Before upgrading to 3.7, make sure that all of your 3.6 members are updated to 3.6.11 or later. Earlier 3.6 patch releases may not be compatible with a rolling upgrade to 3.7.

Upgrade etcd from 3.2 to 3.3

Mon, 01 Jan 0001 00:00:00 +0000

In the general case, upgrading from etcd 3.2 to 3.3 can be a zero-downtime, rolling upgrade:

one by one, stop the etcd v3.2 processes and replace them with etcd v3.3 processes
after running all v3.3 processes, new features in v3.3 are available to the cluster

Before starting an upgrade , read through the rest of this guide to prepare.

Upgrade checklists

Warning

Upgrade etcd from 3.1 to 3.2

Mon, 01 Jan 0001 00:00:00 +0000

In the general case, upgrading from etcd 3.1 to 3.2 can be a zero-downtime, rolling upgrade:

one by one, stop the etcd v3.1 processes and replace them with etcd v3.2 processes
after running all v3.2 processes, new features in v3.2 are available to the cluster

Before starting an upgrade , read through the rest of this guide to prepare.

Upgrade checklists

Warning

Upgrade etcd from 3.0 to 3.1

Mon, 01 Jan 0001 00:00:00 +0000

In the general case, upgrading from etcd 3.0 to 3.1 can be a zero-downtime, rolling upgrade:

one by one, stop the etcd v3.0 processes and replace them with etcd v3.1 processes
after running all v3.1 processes, new features in v3.1 are available to the cluster

Before starting an upgrade , read through the rest of this guide to prepare.

Upgrade checklists

Warning

Upgrade etcd from 2.3 to 3.0

Mon, 01 Jan 0001 00:00:00 +0000

In the general case, upgrading from etcd 2.3 to 3.0 can be a zero-downtime, rolling upgrade:

one by one, stop the etcd v2.3 processes and replace them with etcd v3.0 processes
after running all v3.0 processes, new features in v3.0 are available to the cluster

Before starting an upgrade , read through the rest of this guide to prepare.

Upgrade checklists

Warning

Triage

Mon, 01 Jan 0001 00:00:00 +0000

Issue triage guidelines

Mon, 01 Jan 0001 00:00:00 +0000

Purpose

Speed up issue management.

The etcd issues are listed at https://github.com/etcd-io/etcd/issues and are identified with labels. For example, an issue that is identified as a bug will eventually be set to label area/bug . New issues will start out without any labels, but typically etcd maintainers and active contributors add labels based on their findings. The detailed list of labels can be found at https://github.com/kubernetes/kubernetes/labels

Following are few predetermined searches on issues for convenience:

PR management

Mon, 01 Jan 0001 00:00:00 +0000

Purpose

Speed up PR management.

The etcd PRs are listed at https://github.com/etcd-io/etcd/pulls A PR can have various labels, milestone, reviewer etc. The detailed list of labels can be found at https://github.com/kubernetes/kubernetes/labels

Following are few example searches on PR for convenience:

Scope

These guidelines serves as a primary document for managing PRs in etcd. Everyone is welcome to help manage PRs but the work and responsibilities discussed in this document is created with etcd maintainers and active contributors in mind.

Task on PIG.CENTER

Usage: pgbouncer command

Synopsis

Description

PostgreSQL Logging Configuration

Minimum query logging

Parallel Processing

Split one large file with -j

Incremental Reports

Build daily and weekly reports

Replica imaging and bootstrap

Bootstrap

Releases, Packages, and Upgrades

Standby cluster

Tasks

Operator Tasks

How to Set Up a Demo etcd Cluster

Pause/Resume mode for the cluster

The goal

Support and Contributing

Bugs and feature requests

Using Patroni with Kubernetes

Convert a Standalone to a Patroni Cluster

Procedure

Developer Tasks

8. Logging

Contributing guidelines

Chatting

Reporting bugs

Running tests

Reading from etcd

Prerequisites

Procedure

1. Prerequisites

Writing to etcd

Prerequisites

Procedure

3. Starting HAProxy

4. Stopping and Restarting HAProxy

5. File-Descriptor Limitations

6. Memory Management

How to get keys by prefix

Pre-requisites

Get keys by prefix

Global Options

Options

Example

7. CPU Usage

8. Logging

9. Statistics and Monitoring

10. Easier Configuration Management

11. Well-Known Traps to Avoid

12. Debugging and Performance Issues

13. Security Considerations

How to delete keys

Prerequisites

Add or delete keys

Options

Options inherited from parent commands

Examples

How to make multiple writes in a transaction

Prerequisites

Terminology

Transactions

How to watch keys

Prerequisites

Watching keys

Options

Options inherited from parent commands

Examples

How to create lease

How to create locks

Prerequisites

Creating a lock

Options

How to conduct leader election in etcd cluster

Prerequisites

Conduct Leader election

Options

How to check Cluster status

Split one large file with `-j`