Task

Purpose Speed up PR management. The etcd PRs are listed at https://github.com/etcd-io/etcd/pulls A PR can have various labels, milestone, reviewer etc. The detailed list of labels can be found at https://github.com/kubernetes/kubernetes/labels …

Purpose Speed up issue management. The etcd issues are listed at https://github.com/etcd-io/etcd/issues and are identified with labels. For example, an issue that is identified as a bug will eventually be set to label area/bug . New issues will start …

In the general case, upgrading from etcd 2.3 to 3.0 can be a zero-downtime, rolling upgrade: one by one, stop the etcd v2.3 processes and replace them with etcd v3.0 processes after running all v3.0 processes, new features in v3.0 are available to …

In the general case, upgrading from etcd 3.0 to 3.1 can be a zero-downtime, rolling upgrade: one by one, stop the etcd v3.0 processes and replace them with etcd v3.1 processes after running all v3.1 processes, new features in v3.1 are available to …

In the general case, upgrading from etcd 3.1 to 3.2 can be a zero-downtime, rolling upgrade: one by one, stop the etcd v3.1 processes and replace them with etcd v3.2 processes after running all v3.2 processes, new features in v3.2 are available to …

In the general case, upgrading from etcd 3.2 to 3.3 can be a zero-downtime, rolling upgrade: one by one, stop the etcd v3.2 processes and replace them with etcd v3.3 processes after running all v3.3 processes, new features in v3.3 are available to …

In the general case, upgrading from etcd v3.6 to v3.7 can be a zero-downtime, rolling upgrade: one by one, stop the etcd v3.6 processes and replace them with etcd v3.7 processes after running all v3.7 processes, new features in v3.7 are available to …

In the general case, upgrading from etcd 3.3 to 3.4 can be a zero-downtime, rolling upgrade: one by one, stop the etcd v3.3 processes and replace them with etcd v3.4 processes after running all v3.4 processes, new features in v3.4 are available to …

In the general case, upgrading from etcd 3.4 to 3.5 can be a zero-downtime, rolling upgrade: one by one, stop the etcd v3.4 processes and replace them with etcd v3.5 processes after running all v3.5 processes, new features in v3.5 are available to …

In the general case, downgrading from etcd v3.6 to v3.5 can be a zero-downtime, rolling downgrade: one by one, stop the etcd v3.6 processes and replace them with etcd v3.5 processes after enabling the downgrade, new features in v3.6 are no longer …

In the general case, downgrading from etcd 3.5 to 3.4 can be a zero-downtime, rolling downgrade: one by one, stop the etcd 3.5 processes and replace them with etcd 3.4 processes after starting any 3.4 processes, new features in 3.5 are not longer …

In the general case, upgrading from etcd v3.5 to v3.6 can be a zero-downtime, rolling upgrade: one by one, stop the etcd v3.5 processes and replace them with etcd v3.6 processes after running all v3.6 processes, new features in v3.6 are available to …

In the general case, downgrading from etcd v3.7 to v3.6 can be a zero-downtime, rolling downgrade: one by one, stop the etcd v3.7 processes and replace them with etcd v3.6 processes after enabling the downgrade, new features in v3.7 are no longer …

This section contains documents specific to upgrading etcd clusters and applications. Upgrade policy Before upgrading, note that etcd only supports the following two upgrade cases: Patch upgrade: Upgrading between patch releases within the same minor …

This section contains documents specific to downgrading etcd clusters and applications. Downgrading an etcd v3.x cluster Downgrade etcd from 3.5 to 3.4 Downgrade etcd from 3.6 to 3.5 Downgrade etcd from 3.7 to 3.6

etcd has built in automated data corruption detection to prevent member state from diverging. Enabling data corruption detection Data corruption detection can be done using: Initial check, enabled with --experimental-initial-corrupt-check flag. …

etcd comes with support for incremental runtime reconfiguration, which allows users to update the membership of the cluster at run time. Reconfiguration requests can only be processed when a majority of cluster members are functioning. It is highly …

Each etcd server provides local monitoring information on its client port through http endpoints. The monitoring data is useful for both system health checking and cluster debugging. Debug endpoint If --log-level=debug is set, the etcd server exports …

Overview An etcd cluster needs periodic maintenance to remain reliable. Depending on an etcd application’s needs, this maintenance can usually be automated and performed without downtime or significantly degraded performance. All etcd maintenance …

The gRPC proxy is a stateless etcd reverse proxy operating at the gRPC layer (L7). The proxy is designed to reduce the total processing load on the core etcd cluster. For horizontal scalability, it coalesces watch and lease API requests. To protect …

What is etcd gateway etcd gateway is a simple TCP proxy that forwards network data to the etcd cluster. The gateway is stateless and transparent; it neither inspects client requests nor interferes with cluster responses. It does not terminate TLS …

etcd is designed to withstand machine failures. An etcd cluster automatically recovers from temporary failures (e.g., machine reboots) and tolerates up to (N-1)/2 permanent failures for a cluster of N members. When a member permanently fails, whether …

The following guide shows how to run etcd with Docker using the static bootstrap process . Docker In order to expose the etcd API to clients outside of Docker host, use the host IP address of the container. Please see docker inspect for more detail …

Below demonstrates how to perform the static bootstrap process as a Kubernetes StatefulSet. Example Manifest This manifest contains a service and statefulset for deploying a static etcd cluster in kubernetes. If you copy the contents of the manifest …

Overview Starting an etcd cluster statically requires that each member knows another in the cluster. In a number of cases, the IPs of the cluster members may be unknown ahead of time. In these cases, the etcd cluster can be bootstrapped with the help …

etcd supports automatic TLS as well as authentication through client certificates for both clients to server as well as peer (server to server / cluster) communication. Note that etcd doesn’t enable RBAC based authentication or the authentication …

Overview Authentication was added in etcd 2.1. The etcd v3 API slightly modified the authentication feature’s API and user interface to better fit the new data model. This guide is intended to help users set up basic authentication and role-based …

auth,user,role for authentication: export ETCDCTL_API=3 ENDPOINTS=localhost:2379 etcdctl --endpoints=${ENDPOINTS} role add root etcdctl --endpoints=${ENDPOINTS} role get root etcdctl --endpoints=${ENDPOINTS} user add root etcdctl …

The etcd embed go package provides a simple way to embed an etcd server directly into your application. For more details, see the embed package documentation .

etcd provides a gRPC resolver to support an alternative name system that fetches endpoints from etcd for discovering gRPC services. The underlying mechanism is based on watching updates to keys prefixed with the service name. Note that this feature …

Users mostly interact with etcd by putting or getting the value of a key. This section describes how to do that by using etcdctl, a command line tool for interacting with etcd server. The concepts described here should apply to the gRPC APIs or …

The default settings in etcd should work well for installations on a local network where the average network latency is low. However, when using etcd across multiple data centers or over networks with high latency, the heartbeat interval and election …

If any part of the etcd project has bugs or documentation mistakes, please let us know by opening an issue . We treat bugs and mistakes very seriously and believe no issue is too small. Before creating a bug report, please check that an issue …

member to add,remove,update membership: # For each machine TOKEN=my-etcd-token-1 CLUSTER_STATE=new NAME_1=etcd-node-1 NAME_2=etcd-node-2 NAME_3=etcd-node-3 HOST_1=10.240.0.13 HOST_2=10.240.0.14 HOST_3=10.240.0.15 …

Requirements Before installing etcd, see the following pages: Supported platforms Hardware recommendations Install pre-built binaries The easiest way to install etcd is from pre-built binaries: Download the compressed archive file for your platform …

Pre-requisites Install etcdctl, etcdutl Setup a local cluster Snapshot a database snapshot to save point-in-time snapshot of etcd database: etcdctl --endpoints=$ENDPOINT snapshot save DB_NAME Global Options etcdctl --endpoints=[127.0.0.1:2379], gRPC …

Prerequisites Install etcd and etcdctl Check Overall Status endpoint status to check the overall status of each endpoint specified in --endpoints flag: etcdctl endpoint status (--endpoints=$ENDPOINTS|--cluster) Options --cluster[=false]: use all …

Prerequisites Ensure etcd and etcdctl is installed. Check for active etcd cluster. Conduct Leader election The etcdctl command is used to conduct leader elections in an etcd cluster. It makes sure that only one client can become leader at a time. …

LOCK acquires a distributed mutex with a given name. Once the lock is acquired, it will be held until etcdctl is terminated. Prerequisites Install etcd and etcdctl Creating a lock lock for distributed lock: etcdctl --endpoints=$ENDPOINTS lock mutex1 …

lease to write with TTL: etcdctl --endpoints=$ENDPOINTS lease grant 300 # lease 2be7547fbc6a5afa granted with TTL(300s) etcdctl --endpoints=$ENDPOINTS put sample value --lease=2be7547fbc6a5afa etcdctl --endpoints=$ENDPOINTS get sample etcdctl …

Prerequisites Install etcd and etcdctl Watching keys watch to get notified of future changes: etcdctl watch $KEY [$END_KEY] Options -i, --interactive[=false]: interactive mode --prefix[=false]: watch on a prefix if prefix is set --rev=0: Revision to …

Prerequisites Install etcd and etcdctl . A running etcd cluster. Terminology Here are definitions of some key terms used in the Example below. Terms Definition etcdctl The command line tool for interacting with the etcd server. txn command txn …

Prerequisites Install etcd and etcdctl Add or delete keys del to remove the specified key or range of keys: etcdctl del $KEY [$END_KEY] Options --prefix[=false]: delete keys with matching prefix --prev-kv[=false]: return deleted key-value pairs …

HAProxy is designed to run with very limited privileges. The standard way to use it is to isolate it into a chroot jail and to drop its privileges to a non-root user without any permissions inside this jail so that if any future vulnerability were …

When HAProxy is started with the “-d” option, it will stay in the foreground and will print one line per event, such as an incoming connection, the end of a connection, and for each request or response header line seen. This debug output is emitted …

Once in a while, someone reports that after a system reboot, the haproxy service wasn’t started, and that once they start it by hand it works. Most often, these people are running a clustered IP address mechanism such as keepalived, to assign the …

It is very common that two HAProxy nodes constituting a cluster share exactly the same configuration modulo a few addresses. Instead of having to maintain a duplicate configuration for each node, which will inevitably diverge, it is possible to …

It is possible to query HAProxy about its status. The most commonly used mechanism is the HTTP statistics page. This page also exposes an alternative CSV output format for monitoring tools. The same format is provided on the Unix socket. Statistics …

For logging, HAProxy always relies on a syslog server since it does not perform any file-system access. The standard way of using it is to send logs over UDP to the log server (by default on port 514). Very commonly this is configured to 127.0.0.1 …

HAProxy normally spends most of its time in the system and a smaller part in userland. A finely tuned 3.5 GHz CPU can sustain a rate about 80000 end-to-end connection setups and closes per second at 100% CPU on a single core. When one core is …

Pre-requisites Install etcdctl Setup a local cluster Get keys by prefix $ etcdctl --endpoints=$ENDPOINTS get PREFIX --prefix Global Options --endpoints=[127.0.0.1:2379], gRPC endpoints Options --prefix, get a range of keys with matching prefix …

HAProxy uses a simple and fast pool-based memory management. Since it relies on a small number of different object types, it’s much more efficient to pick new objects from a pool which already contains objects of the appropriate size than to call …

In order to ensure that all incoming connections will successfully be served, HAProxy computes at load time the total number of file descriptors that will be needed during the process’s life. A regular Unix process is generally granted 1024 file …

HAProxy supports a graceful and a hard stop. The hard stop is simple, when the SIGTERM signal is sent to the haproxy process, it immediately quits and all established connections are closed. The graceful stop is triggered when the SIGUSR1 signal is …

HAProxy is started by invoking the “haproxy” program with a number of arguments passed on the command line. The actual syntax is: $ haproxy [<options>]* where [<options>]* is any number of options. An option always starts with ‘-’ followed by one of …

Prerequisites Install etcdctl Procedure Use the put subcommand to write a key-value pair: etcdctl --endpoints=$ENDPOINTS put foo "Hello World!" where: foo is the key name "Hello World!" is the quote-delimited value

This document describes how to start, stop, manage, and troubleshoot HAProxy, as well as some known limitations and traps to avoid. It does not describe how to configure it (for this please read configuration.txt ). In this document it is assumed …

Prerequisites Install etcdctl Procedure Use the get subcommand to read from etcd: $ etcdctl --endpoints=$ENDPOINTS get foo foo Hello World! $ where: foo is the requested key Hello World! is the retrieved value Or, for formatted output: $ etcdctl …

Chatting If you have a question, looking for an interactive troubleshooting help or want to chat with other Patroni users, join us on channel #patroni in the PostgreSQL Slack . Reporting bugs Before reporting a bug please make sure to reproduce it …

One of HAProxy’s strong points certainly lies is its precise logs. It probably provides the finest level of information available for such a product, which is very important for troubleshooting complex environments. Standard information provided in …

This section describes the process for converting a standalone PostgreSQL instance into a Patroni cluster. To deploy a Patroni cluster without using a pre-existing PostgreSQL instance, see Running and Configuring instead. Procedure You can find …

Patroni can use Kubernetes objects in order to store the state of the cluster and manage the leader key. That makes it capable of operating Postgres in Kubernetes environment without any consistency store, namely, one doesn’t need to run an extra …

pgBadger is maintained as an open project. Bug reports, feature proposals, documentation fixes, and patches are handled through the upstream GitHub repository. Bugs and feature requests Upgrade to the newest released version and confirm the behavior …

The goal Under certain circumstances Patroni needs to temporarily step down from managing the cluster, while still retaining the cluster state in DCS. Possible use cases are uncommon activities on the cluster, such as major version upgrades or …

On each etcd node, specify the cluster members: TOKEN=token-01 CLUSTER_STATE=new NAME_1=machine-1 NAME_2=machine-2 NAME_3=machine-3 HOST_1=10.240.0.17 HOST_2=10.240.0.18 HOST_3=10.240.0.19 …

Patroni also support running cascading replication to a remote datacenter (region) using a feature that is called “standby cluster”. This type of clusters has: “standby leader”, that behaves pretty much like a regular cluster leader, except it …

HAProxy is an open source project covered by the GPLv2 license, meaning that everyone is allowed to redistribute it provided that access to the sources is also provided upon request, especially if any modifications were made. HAProxy evolves as a …

Patroni allows customizing creation of a new replica. It also supports defining what happens when the new empty cluster is being bootstrapped. The distinction between two is well defined: Patroni creates replicas only if the initialize key is …

Incremental mode stores parsed statistics in binary form, then builds one HTML report per day, a cumulative report per week, and a calendar-style index linking them together. It is intended for repeated processing of rotated logs without counting the …

pgBadger has two complementary multiprocessing modes. Choose according to the shape of the input, not simply the number of CPUs. Option Parallel unit Best fit Main constraint -j N / --jobs N chunks of one log file one large, seekable log chunk …

pgBadger can only report information that PostgreSQL writes to the log. Start with a parseable prefix and a deliberate statement-logging policy, then add the operational events you want to analyze. Minimum query logging To include query text and …

Synopsis pgbouncer [-d][-R][-v][-u user] <pgbouncer.ini> pgbouncer -V|-h On Windows, the options are: pgbouncer.exe [-v][-u user] <pgbouncer.ini> pgbouncer.exe -V|-h Additional options for setting up a Windows service: pgbouncer.exe --regservice …