Upgrade etcd from v3.5 to v3.6
In the general case, upgrading from etcd v3.5 to v3.6 can be a zero-downtime, rolling upgrade:
- one by one, stop the etcd v3.5 processes and replace them with etcd v3.6 processes
- after running all v3.6 processes, new features in v3.6 are available to the cluster
Before starting an upgrade , read through the rest of this guide to prepare.
Upgrade checklists
Update 3.5
Before upgrading to 3.6, make sure that all of your 3.5 members are updated to 3.5.32 or later
. Patch releases 3.5.24 through 3.5.26 fix several potential upgrade blockers; 3.5.32
adds --v2-deprecation=write-only-skip-check and extends etcdutl check v2store to inspect WAL records as well as the v2 snapshot.
V2 Store
If the --enable-v2 flag is not configured or is set to false, no further action is required.
If --enable-v2 is configured, run the command etcdutl check v2store to verify whether the v2store contains any non-membership (custom) data. If no custom data is present, the flag can be safely removed. Otherwise, refer to the v2 migration guide
for more details.
Flags added
Flags removed
Flags deprecated
etcd --experimental-bootstrap-defrag-threshold-megabytes flag has been deprecated.
etcd --experimental-compaction-batch-limit flag has been deprecated.
etcd --experimental-compact-hash-check-time flag has been deprecated.
etcd --experimental-compaction-sleep-interval flag has been deprecated.
etcd --experimental-corrupt-check-time flag has been deprecated.
etcd --experimental-enable-distributed-tracing flag has been deprecated.
etcd --experimental-distributed-tracing-address flag has been deprecated.
etcd --experimental-distributed-tracing-instance-id flag has been deprecated.
etcd --experimental-distributed-tracing-sampling-rate flag has been deprecated.
etcd --experimental-distributed-tracing-service-name flag has been deprecated.
etcd --experimental-downgrade-check-time flag has been deprecated.
etcd --experimental-max-learners flag has been deprecated.
etcd --experimental-memory-mlock flag has been deprecated.
etcd --experimental-peer-skip-client-san-verification flag has been deprecated.
etcd --experimental-snapshot-catchup-entries flag has been deprecated.
etcd --experimental-warning-apply-duration flag has been deprecated.
etcd --experimental-warning-unary-request-duration flag has been deprecated.
etcd --experimental-watch-progress-notify-interval flag has been deprecated.
Equivalent flags of v3.5 feature gates
equivalent flag for feature gate etcd --experimental-compact-hash-check-enabled=true
equivalent flag for feature gate etcd --experimental-initial-corrupt-check=true
equivalent flag for feature gate etcd --experimental-enable-lease-checkpoint=true
equivalent flag for feature gate etcd --experimental-enable-lease-checkpoint-persist=true
equivalent flag for feature gate etcd --experimental-stop-grpc-service-on-defrag=true
equivalent flag for feature gate etcd --experimental-txn-mode-write-with-shared-buffer=false
Flags with new defaults
Original default flag etcd --snapshot-count=100000
Original default flag etcd --v2-deprecation='not-yet'
Original default flag etcd --discovery-fallback='proxy'
Difference in Prometheus metrics
Server upgrade checklists
Upgrade requirements
To upgrade an existing etcd deployment to v3.6, the running cluster must be v3.5 or greater. If it’s before v3.5, please upgrade to v3.5 before upgrading to v3.6.
Also, to ensure a smooth rolling upgrade, the running cluster must be healthy. Check the health of the cluster by using the etcdctl endpoint health command before proceeding.
Preparation
Before upgrading etcd, always test the services relying on etcd in a staging environment before deploying the upgrade to the production environment.
Before beginning, download the snapshot backup
. Should something go wrong with the upgrade, it is possible to use this backup to rollback
back to existing etcd version. Please note that the snapshot command only backs up the v3 data.
Mixed versions
While upgrading, an etcd cluster supports mixed versions of etcd members, and operates with the protocol of the lowest common version. The cluster is only considered upgraded once all of its members are upgraded to version v3.6. Internally, etcd members negotiate with each other to determine the overall cluster version, which controls the reported version and the supported features.
Rollback
Before upgrading your etcd cluster, please create and download a snapshot backup of your etcd cluster. This snapshot can be used to restore the cluster to its pre-upgrade state if needed. If users encounter issues during the upgrade, they should first identify and resolve the root cause. If the cluster is still in a mixed-version state—where at least one member remains on v3.5—they can either replace the binary or image with the old v3.5 version or restore the cluster directly using the snapshot. In this mixed state, the cluster continues to operate as a v3.5 cluster, allowing rollback without following a formal downgrade process.
However, once all members have been upgraded to v3.6, the cluster is considered fully upgraded and rollback using binaries is no longer possible. In that case, the only recovery option is to restore from the snapshot taken before the upgrade. If users wish to return to the original version after a full upgrade has completed, they should follow the official downgrade guide to ensure consistency and avoid data corruption.
Upgrade procedure
This example shows how to upgrade a 3-member v3.5 etcd cluster running on a local machine.
Step 1: check upgrade requirements
Is the cluster healthy and running v3.5.x?
Step 2: download snapshot backup from leader
Download the snapshot backup to provide a downgrade path should any problems occur.
etcd leader is guaranteed to have the latest application data, thus fetch snapshot from leader:
Step 3: stop one existing etcd server
When each etcd process is stopped, expected errors will be logged by other cluster members. This is normal since a cluster member connection has been (temporarily) broken:
Step 4: restart the etcd server with same configuration
Restart the etcd server with same configuration but with the new etcd binary.
The new v3.6 etcd will publish its information to the cluster. At this point, cluster still operates as v3.5 protocol, which is the lowest common version.
{"level":"info","ts":"2025-03-01T04:40:36.828+0530","caller":"api/capability.go:76","msg":"enabled capabilities for version","cluster-version":"3.5"}
{"level":"info","ts":"2025-03-01T04:40:36.889+0530","caller":"membership/cluster.go:539","msg":"updated cluster version","cluster-id":"59a05384c9b79ee","local-member-id":"bf9071f4639c75cc","from":"3.0","to":"3.5"}
{"level":"info","ts":"2025-03-01T04:40:36.828+0530","caller":"api/capability.go:76","msg":"enabled capabilities for version","cluster-version":"3.5"}
{"level":"info","ts":"2025-03-01T04:40:36.894+0530","caller":"etcdserver/server.go:1686","msg":"published local member to cluster through raft","local-member-id":"bf9071f4639c75cc","local-member-attributes":"{Name:node1 ClientURLs:[http://127.0.0.1:2379]}","cluster-id":"59a05384c9b79ee","publish-timeout":"7s"}
Verify that each member, and then the entire cluster, becomes healthy with the new v3.6 etcd binary:
Un-upgraded members will log warnings like the following until the entire cluster is upgraded.
This is expected and will cease after all etcd cluster members are upgraded to v3.6:
Step 5: repeat step 3 and step 4 for rest of the members
When all members are upgraded, the cluster will report upgrading to v3.6 successfully:
Member 1:
{"level":"info","ts":"2025-03-01T04:58:32.375+0530","caller":"etcdserver/server.go:2149","msg":"updating cluster version using v3 API","from":"3.5","to":"3.6"}{"level":"info","ts":"2025-03-01T04:58:32.377+0530","caller":"etcdserver/server.go:2164","msg":"cluster version is updated","cluster-version":"3.6"}
Member 2:
{"level":"info","ts":"2025-03-01T04:58:32.377+0530","caller":"membership/cluster.go:539","msg":"updated cluster version","cluster-id":"59a05384c9b79ee","local-member-id":"91bc3c398fb3c146","from":"3.5","to":"3.6"}
Member 3:
{"level":"info","ts":"2025-03-01T04:58:32.377+0530","caller":"membership/cluster.go:539","msg":"updated cluster version","cluster-id":"59a05384c9b79ee","local-member-id":"fd422379fda50e48","from":"3.5","to":"3.6"}