Skip to main content

Chapter 2.3 - etcd

Learning Objectives​

By the end of this chapter, you will be able to:

  • Understand the role of etcd in Kubernetes
  • Explain how etcd stores the cluster state
  • Understand replication and high availability
  • Identify backup best practices

What is etcd?​

etcd is a distributed, consistent, and highly available key-value database used as the single source of truth for Kubernetes. All cluster configuration and state are stored in etcd.

Role in Kubernetes​

Single Source of Truth​

etcd stores:

  • All resources (Pods, Services, Deployments, etc.)
  • Cluster configuration
  • Current state of all resources
  • Metadata and annotations

No Direct Access​

Important: Users and components NEVER access etcd directly. All interactions go through the API Server.

Data Structure​

Key-Value Format​

Data is organized hierarchically:

/registry/pods/default/my-pod
/registry/services/default/my-service
/registry/deployments/default/my-deployment

Example of Stored Data​

{
  "kind": "Pod",
  "apiVersion": "v1",
  "metadata": {
    "name": "my-pod",
    "namespace": "default",
    "uid": "123-456-789"
  },
  "spec": {
    "containers": [...]
  },
  "status": {
    "phase": "Running"
  }
}

High Availability​

Replication​

For production, etcd must be replicated (typically 3 or 5 nodes):

Advantages:

  • Fault tolerance (1 node can fail)
  • Improved performance (distributed reads)
  • Guaranteed consistency

Consensus: Raft​

etcd uses the Raft consensus algorithm:

Characteristics:

  • One elected leader
  • Replication to followers
  • Consensus by majority
  • Guaranteed strong consistency

Backup and Restore​

Critical Importance​

etcd contains ALL the cluster state. Its loss = loss of the cluster!

Regular Backup​

# Back up etcd
ETCDCTL_API=3 etcdctl snapshot save /backup/etcd-snapshot.db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

# Restore from a backup
ETCDCTL_API=3 etcdctl snapshot restore /backup/etcd-snapshot.db

Best Practices​

  • Daily automatic backups
  • Off-site storage
  • Regular restore tests
  • Documented process

Performance​

Factors Impacting Performance​

  1. Cluster size: More resources = more data
  2. Change frequency: Frequent updates
  3. Object size: Large ConfigMaps/Secrets
  4. Compaction: Cleaning up old versions

Optimizations​

  • Regular data compaction
  • Disk defragmentation
  • SSD for better performance
  • Metrics monitoring

Security​

Encryption​

  • In transit: TLS between API Server and etcd
  • At rest: Optional data encryption

Restricted Access​

  • Only the API Server accesses etcd
  • Client certificates required
  • No public network access

Monitoring​

Important Metrics​

  • Database size
  • Operation latency
  • Error rate
  • Leader status
  • Available disk space

Useful Commands​

# Check etcd status
kubectl get componentstatuses

# View metrics (if monitoring configured)
# Via Prometheus or similar tools

Summary​

In this chapter, you learned:

etcd: Distributed database, single source of truth
Storage: All resources and cluster configuration
High Availability: Replication with Raft consensus
Backup: Critical for cluster continuity
Security: Restricted access, encryption in transit

Next Steps​

Now that you understand etcd:

Chapter 2.4: Controller Manager - Maintaining the Desired State
Chapter 2.5: Scheduler - Pod Scheduling

Chapter created: December 2024