etcd High Latency: Check p99 fsync Below 10 ms and RTT Below 50 ms

Disk write latency and network round-trip time are the two culprits behind most etcd slowdowns, and you can investigate which one is hurting you. Check your WAL fsync p99 and your peer RTT p99 first. If either sits above the healthy range, you have your starting point. Everything else in this guide builds from there.

---

TL;DR:

• Keep WAL fsync p99 below 10 milliseconds, ideally below 1 millisecond on SSD or NVMe, and peer RTT p99 below 50 milliseconds.
• Compare metrics across every member, then test disk and network directly; tail spikes can reach p99.99 or maximum despite healthy p99 readings.
• Keep quorum members in one data center or availability zone; adding voters or spanning regions can increase write latency rather than reduce it.
• Compact regularly to avoid quota limits, and stagger defragmentation across members with at least one minute between runs because each run blocks its member.

---

Table of Contents

• Why etcd latency matters for your Kubernetes control plane
• Common causes of high etcd latency
• Key metrics and Prometheus queries to find the root cause
• Practical diagnostic runbook: what to check right now
• Tuning options that actually reduce latency, and their risks
• Maintenance that keeps latency from creeping back
• When to scale, relocate, or redesign your etcd topology
• A prioritized checklist and the mistakes that waste your afternoon
• How we can help you run this diagnosis end to end
• FAQ
• Sources

Why etcd latency matters for your Kubernetes control plane

Every write to etcd has to clear two gates before it is considered committed: the network round trip to a quorum of peers, and the local disk fsync that makes the write durable. Raft commit time is effectively RTT plus fdatasync time, so when either number grows, every write in your cluster slows down with it.

That delay does not stay contained inside etcd. The API server depends on etcd for every read and write, so sluggish commits turn into slow kubectl commands, scheduler delays, and controller reconciliation lag. Push latency far enough and members start missing heartbeats, which triggers leader elections, and elections briefly pause writes across the whole cluster.

The baseline is not abstract. A healthy 3-node cluster in a typical cloud environment finishes writes in under 1 millisecond under light load and can sustain more than 30,000 requests per second under heavy load. Anything meaningfully worse than that is a signal, not noise.

Common causes of high etcd latency

Most etcd slowdowns trace back to one of a handful of recurring patterns. Knowing which one you are dealing with saves you from tuning the wrong knob.

• Disk fsync spikes: shared or network-backed storage, noisy neighbor workloads, or spinning disks instead of SSDs all push fsync times up.
• Network RTT and jitter: members spread across availability zones or regions add latency that Raft consensus has to absorb on every write.
• Leader overload: a heavy write workload or a client hammering the API with large transactions can saturate the leader before anything else looks wrong.
• Database bloat and fragmentation: an oversized or fragmented backend database slows every read and write, and compaction or defrag running at the wrong time makes it worse temporarily.
• Misconfigured heartbeat and election settings: defaults tuned for low-latency networks can amplify transient blips into full elections on higher-latency links.

SSD-based storage typically keeps fdatasync latency under 1 millisecond, while spinning disks often run around 10 milliseconds. That tenfold gap alone explains why hardware choice is usually the first place to look.

Key metrics and Prometheus queries to find the root cause

A few metrics will tell you almost everything you need to know before you touch a single config file.

• WAL fsync p99: histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m])). Keep this under 10 milliseconds, ideally under 1 millisecond on SSD or NVMe.
• Backend commit p99: histogram_quantile(0.99, rate(etcd_disk_backend_commit_duration_seconds_bucket[5m])). Spikes here often line up with defragmentation or snapshot activity.
• Peer RTT p99: histogram_quantile(0.99, rate(etcd_network_peer_round_trip_time_seconds_bucket[2m])). Target under 50 milliseconds between members.
• Committed versus applied proposals: compare etcd_server_proposals_committed_total against etcd_server_proposals_applied_total. A persistently growing gap means the member is overloaded, usually by slow disk I/O.

Metric	Query focus	Healthy threshold
WAL fsync p99	etcd_disk_wal_fsync_duration_seconds	under 10ms, ideally under 1ms on NVMe
Backend commit p99	etcd_disk_backend_commit_duration_seconds	low and stable, not spiking with defrag
Peer RTT p99	etcd_network_peer_round_trip_time_seconds	under 50ms

Also pull node_disk_write_time and container-level write bytes from your host metrics. A member that looks fine on fsync p99 but still feels slow is often fighting a noisy neighbor for disk bandwidth rather than suffering from raw fsync latency.

Practical diagnostic runbook: what to check right now

Work through these in order. Each step narrows the field before you touch any configuration.

1. Run etcdctl endpoint health and etcdctl endpoint status against every member to confirm basic cluster health and spot an obviously struggling node.
2. Query WAL fsync p99 and peer RTT p99 for each member separately and compare them side by side. One outlier member usually points to a hardware or network problem specific to that host.
3. Validate storage directly with an fio fsync workload against the etcd data directory, and cross-check with iostat and node_disk_write_time to rule out a noisy neighbor saturating shared disk bandwidth.
4. Test the network path between members with ping and iperf, and inspect TCP retransmits and kernel network error counters for jitter that Prometheus histograms might smooth over.
5. Apply safe, reversible mitigations while you investigate further: pause backup jobs or log shipping running on control-plane nodes, throttle any client sending unusually large or frequent writes, and make sure peer traffic is not competing with other workloads for bandwidth.

Pro Tip: Check tail latency, not just p99. Real-world reports show high read latencies can appear even when p99 looks fine, because the real problem shows up in p99.99 or max fsync spikes.

If you are also seeing etcdserver: too many requests errors alongside slow commits, that is a sign the leader is overloaded rather than purely disk or network bound. Our guide to that specific error walks through the client-side causes separately.

Tuning options that actually reduce latency, and their risks

Once you have confirmed where the latency is coming from, a few configuration changes can help, but they work best as refinements, not fixes for broken hardware.

• Heartbeat interval and election timeout: set heartbeat interval close to your measured RTT, and election timeout to at least 10 times that RTT. Raising these values too aggressively just delays detection of a real problem instead of solving it.
• Storage upgrades: move to SSD or NVMe if you have not already, and hold WAL fsync p99 under 10 milliseconds, ideally closer to 1 millisecond.
• Workload isolation: avoid colocating I/O-heavy workloads on control-plane nodes, since disk bandwidth starvation can disrupt etcd even when baseline fsync numbers look acceptable. Dedicated storage or I/O prioritization helps here.
• CPU and network tuning: set a performance CPU governor on control-plane nodes and prioritize peer traffic over best-effort workloads where your network stack allows it.

Pro Tip: Treat timeout tuning as a last resort, not a first move. Stretching election timeouts to paper over fsync latency hides the real problem until the day your disk gets slower still.

Maintenance that keeps latency from creeping back

Ongoing maintenance prevents the slow, quiet latency growth that catches teams off guard months after a cluster goes live.

• Compact regularly: failing to compact lets the keyspace grow until it trips a storage quota alarm, which forces the cluster into limited-operation maintenance mode.
• Stagger defragmentation: defragmentation blocks the member running it, so space it out across members with at least a minute’s wait between each one rather than running it cluster-wide at once.
• Watch snapshot-count trade-offs: a very high snapshot-count setting can let more uncompacted history build up between snapshots, adding memory and garbage-collection pressure.
• Monitor database size: track etcd_mvcc_db_total_size_in_use_in_bytes alongside the raw backend size so you catch growth before it becomes a quota problem.

Our etcd backup and restore guide is worth reading before you schedule compaction or defrag changes, since backup timing interacts with both.

When to scale, relocate, or redesign your etcd topology

Tuning has limits. Adding control-plane nodes improves availability, but each additional voting member adds a round trip to quorum and more batch overhead, which can quietly raise latency rather than lower it.

Keep quorum members in a single data center or availability zone whenever possible: cross-region placement keeps network RTT as the hard limiter on every write, no matter how well you tune everything else. Arbiter nodes or read-offload patterns are worth considering only when your topology and latency requirements genuinely demand them. If storage and RTT targets still cannot be met after addressing hardware and placement, a topology change or capacity increase is the right call, not another round of timeout tweaks.

A prioritized checklist and the mistakes that waste your afternoon

Measure WAL fsync p99 first, isolate disk performance second, validate peer RTT third, and only then touch heartbeat or election settings. That order matters more than any individual number.

The mistakes I see most often: tuning timeouts before ruling out hardware, defragmenting every member at once and causing the exact spike you were trying to avoid, and ignoring storage quotas until the cluster forces itself into maintenance mode. Automate your baseline checks so the next incident starts with data instead of guesswork.

— James

How we can help you run this diagnosis end to end

If you would rather have a second set of eyes run this full diagnostic pass, our Kubernetes Health Check covers exactly this: we pull your etcd metrics, interpret fsync and RTT data against the thresholds above, and hand you a ranked remediation plan. An Observability Review is the right fit if your Prometheus setup itself needs work before you can trust these numbers.

For teams that want to self-serve first, our Kubernetes YAML Validator catches manifest mistakes before they generate the noisy reconfiguration traffic that drives latency up in the first place. For broader infrastructure trade-offs around storage and placement, this guide to optimizing cloud infrastructure from Yslootah Tech is a useful companion read. When you are ready for a deeper engagement, our work with me page lists every audit and consulting option we offer.

FAQ

What is considered high latency for etcd?

WAL fsync p99 above 10 milliseconds or peer RTT p99 above 50 milliseconds both count as high latency and warrant investigation. On NVMe storage, fsync p99 should ideally stay under 1 millisecond.

How do I check etcd latency with Prometheus?

Run histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m])) for fsync latency and the equivalent query against etcd_network_peer_round_trip_time_seconds_bucket for peer RTT. Compare results across all members to spot an outlier.

Does defragmentation cause etcd latency spikes?

Yes, defragmentation blocks the member it runs on while it executes, which is why it should be staggered across members with a wait of at least a minute between each one. Running it on every member simultaneously can cause a cluster-wide latency spike.

Can network RTT alone explain etcd slowness?

Yes, since Raft commit time is network RTT plus disk fsync time, a high RTT between members will slow every write even if disk performance is excellent. This is especially common in clusters spread across multiple availability zones or regions.

Should I increase etcd’s heartbeat interval to fix latency?

Only after confirming disk and network performance are within target ranges. Heartbeat interval should be set close to your measured RTT and election timeout to at least 10 times that RTT, but raising these values to mask a disk or network problem just delays detection of the real issue.

Sources

• Performance | etcd
• Metrics | etcd
• Maintenance | etcd
• docs/performance-metrics.md
• Chapter 2. Recommended etcd practices | etcd | OpenShift Container Platform | 4.22 | Red Hat Documentation

Recommended

• chrony / NTP Time Sync Troubleshooting Prompt
• Kubernetes Error Guide: ‘etcdserver: request timed out’ Slow
• OpenStack Error Guide: ‘ToozConnectionError: Error’