Introduction

Apache Zookeeper is a distributed coordination service that provides a centralized infrastructure for distributed applications. It offers services like distributed locks, leader election, configuration management, and group membership, making it essential for building reliable distributed systems.

This guide covers:

  • Zookeeper Fundamentals: Core concepts, architecture, and data model
  • Coordination Services: Locks, leader election, configuration management
  • Consensus Algorithm: ZAB (Zookeeper Atomic Broadcast)
  • Use Cases: Real-world applications and patterns
  • Best Practices: Performance, reliability, and optimization

What is Apache Zookeeper?

Apache Zookeeper is a distributed coordination service that offers:

  • Distributed Locks: Synchronize access to shared resources
  • Leader Election: Elect a leader from a group of nodes
  • Configuration Management: Centralized configuration storage
  • Service Discovery: Register and discover services
  • Group Membership: Track which nodes are active
  • Consensus: Guaranteed consistency across nodes

Key Concepts

ZNode: A node in the Zookeeper data tree (similar to a file/directory)

Session: Connection between client and Zookeeper server

Watcher: Callback mechanism for notifications

Quorum: Majority of servers in a cluster

Leader: Server that coordinates writes

Follower: Server that replicates leader’s writes

Observer: Server that doesn’t participate in voting (read-only)

Architecture

High-Level Architecture

┌─────────────┐     ┌─────────────┐     ┌─────────────┐
│   Client    │────▶│   Client    │────▶│   Client    │
│ Application │     │ Application │     │ Application │
└──────┬──────┘     └──────┬──────┘     └──────┬──────┘
       │                    │                    │
       └────────────────────┴────────────────────┘
                            │
                            ▼
              ┌─────────────────────────┐
              │   Zookeeper Cluster     │
              │                         │
              │  ┌──────────┐           │
              │  │  Leader  │           │
              │  └────┬─────┘           │
              │       │                  │
              │  ┌────┴─────┐           │
              │  │ Followers │           │
              │  │ (Quorum)  │           │
              │  └───────────┘           │
              │                         │
              │  ┌───────────────────┐  │
              │  │   Data Tree        │  │
              │  │   (Znodes)         │  │
              │  └───────────────────┘  │
              └─────────────────────────┘

Explanation:

  • Client Applications: Distributed applications that connect to Zookeeper for coordination services (locks, leader election, configuration).
  • Zookeeper Cluster: A collection of Zookeeper servers working together. Typically 3, 5, or 7 nodes for fault tolerance.
  • Leader: One server in the cluster that coordinates writes and ensures consistency.
  • Followers: Other servers in the cluster that replicate the leader’s writes and participate in voting (quorum).
  • Data Tree (Znodes): Hierarchical namespace similar to a file system, storing configuration, locks, and service information.

Core Architecture

┌─────────────────────────────────────────────────────────┐
│              Zookeeper Cluster                           │
│                                                          │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐              │
│  │  Leader  │  │ Follower  │  │ Follower │              │
│  │          │  │           │  │          │              │
│  └────┬─────┘  └────┬─────┘  └────┬─────┘              │
│       │             │             │                     │
│       └─────────────┴─────────────┘                     │
│                    Quorum                                │
│                                                          │
│  ┌──────────────────────────────────────┐               │
│  │         Data Tree (Znodes)           │               │
│  │  /                                    │               │
│  │  ├── /config                         │               │
│  │  ├── /locks                          │               │
│  │  ├── /services                       │               │
│  │  └── /leader                         │               │
│  └──────────────────────────────────────┘              │
└─────────────────────────────────────────────────────────┘

Data Model

ZNode Types

Persistent ZNode: Exists until explicitly deleted

Ephemeral ZNode: Automatically deleted when session ends

Sequential ZNode: Automatically numbered (e.g., /lock-0000000001)

Example:

/ (root)
├── /config (persistent)
│   ├── /database-url
│   └── /api-key
├── /locks (persistent)
│   ├── /resource-0000000001 (ephemeral, sequential)
│   └── /resource-0000000002 (ephemeral, sequential)
└── /services (persistent)
    ├── /service-1 (ephemeral)
    └── /service-2 (ephemeral)

Common Use Cases

1. Distributed Locks

Coordinate access to shared resources across multiple processes.

Implementation:

import org.apache.zookeeper.*;
import org.apache.zookeeper.data.Stat;

public class DistributedLock {
    private ZooKeeper zk;
    private String lockPath = "/locks/resource";
    private String lockNode;
    
    public boolean acquireLock() throws Exception {
        // Create ephemeral sequential node
        lockNode = zk.create(
            lockPath + "/lock-",
            new byte[0],
            ZooDefs.Ids.OPEN_ACL_UNSAFE,
            CreateMode.EPHEMERAL_SEQUENTIAL
        );
        
        // Get all lock nodes
        List<String> locks = zk.getChildren(lockPath, false);
        Collections.sort(locks);
        
        // Check if we have the smallest number (lock holder)
        String smallestLock = lockPath + "/" + locks.get(0);
        return lockNode.equals(smallestLock);
    }
    
    public void releaseLock() throws Exception {
        zk.delete(lockNode, -1);
    }
}

2. Leader Election

Elect a leader from a group of nodes.

Implementation:

public class LeaderElection {
    private ZooKeeper zk;
    private String electionPath = "/election";
    private String candidateNode;
    private String currentLeader;
    
    public void participateInElection() throws Exception {
        // Create ephemeral sequential node
        candidateNode = zk.create(
            electionPath + "/candidate-",
            new byte[0],
            ZooDefs.Ids.OPEN_ACL_UNSAFE,
            CreateMode.EPHEMERAL_SEQUENTIAL
        );
        
        // Watch for leader changes
        electLeader();
    }
    
    private void electLeader() throws Exception {
        List<String> candidates = zk.getChildren(electionPath, new Watcher() {
            @Override
            public void process(WatchedEvent event) {
                try {
                    electLeader(); // Re-elect on change
                } catch (Exception e) {
                    e.printStackTrace();
                }
            }
        });
        
        Collections.sort(candidates);
        currentLeader = electionPath + "/" + candidates.get(0);
        
        if (candidateNode.equals(currentLeader)) {
            System.out.println("I am the leader!");
            onElectedLeader();
        } else {
            System.out.println("Leader is: " + currentLeader);
        }
    }
}

3. Configuration Management

Store and distribute configuration across services.

Implementation:

public class ConfigurationManager {
    private ZooKeeper zk;
    private String configPath = "/config";
    private Map<String, String> config = new HashMap<>();
    
    public void loadConfig() throws Exception {
        // Watch for config changes
        byte[] data = zk.getData(configPath, new Watcher() {
            @Override
            public void process(WatchedEvent event) {
                try {
                    loadConfig(); // Reload on change
                } catch (Exception e) {
                    e.printStackTrace();
                }
            }
        }, null);
        
        // Parse and update config
        String configJson = new String(data);
        config = parseConfig(configJson);
    }
    
    public void updateConfig(String key, String value) throws Exception {
        config.put(key, value);
        String configJson = serializeConfig(config);
        zk.setData(configPath, configJson.getBytes(), -1);
    }
}

4. Service Discovery

Register services and discover available services.

Implementation:

public class ServiceRegistry {
    private ZooKeeper zk;
    private String servicePath = "/services";
    private String serviceNode;
    
    public void registerService(String serviceName, String serviceUrl) throws Exception {
        // Create ephemeral node for service
        serviceNode = zk.create(
            servicePath + "/" + serviceName + "/instance-",
            serviceUrl.getBytes(),
            ZooDefs.Ids.OPEN_ACL_UNSAFE,
            CreateMode.EPHEMERAL_SEQUENTIAL
        );
        
        System.out.println("Service registered: " + serviceNode);
    }
    
    public List<String> discoverServices(String serviceName) throws Exception {
        String servicePath = this.servicePath + "/" + serviceName;
        List<String> instances = zk.getChildren(servicePath, false);
        
        List<String> serviceUrls = new ArrayList<>();
        for (String instance : instances) {
            byte[] data = zk.getData(servicePath + "/" + instance, false, null);
            serviceUrls.add(new String(data));
        }
        
        return serviceUrls;
    }
}

Consensus Algorithm: ZAB

Zookeeper Atomic Broadcast (ZAB)

Purpose: Ensure all servers agree on the order of transactions

Phases:

  1. Discovery: Find the highest transaction ID
  2. Synchronization: Sync with leader
  3. Broadcast: Broadcast new transactions

Properties:

  • Reliability: All committed transactions are persisted
  • Ordering: All transactions are ordered
  • Atomicity: All servers see the same state

Leader Election

Process:

  1. Each server votes for itself
  2. Server with highest ZXID (transaction ID) becomes leader
  3. Remaining servers become followers
  4. Followers sync with leader

Use Cases and Patterns

1. Distributed Lock Pattern

Use Case: Coordinate access to shared resource

Pattern:

// Acquire lock
String lockNode = zk.create("/locks/resource/lock-", 
    new byte[0], 
    ZooDefs.Ids.OPEN_ACL_UNSAFE,
    CreateMode.EPHEMERAL_SEQUENTIAL);

// Check if we have the lock
List<String> locks = zk.getChildren("/locks/resource", false);
if (isSmallestLock(lockNode, locks)) {
    // We have the lock
    doWork();
    zk.delete(lockNode, -1);
} else {
    // Wait for lock
    waitForLock(lockNode, locks);
}

2. Leader Election Pattern

Use Case: Elect a single leader from multiple candidates

Pattern:

// Participate in election
String candidateNode = zk.create("/election/candidate-",
    new byte[0],
    ZooDefs.Ids.OPEN_ACL_UNSAFE,
    CreateMode.EPHEMERAL_SEQUENTIAL);

// Watch for leader changes
watchLeader(candidateNode);

// If we're the leader, do leader work
if (isLeader(candidateNode)) {
    performLeaderDuties();
}

3. Configuration Management Pattern

Use Case: Centralized configuration storage

Pattern:

// Watch for config changes
zk.getData("/config", new Watcher() {
    @Override
    public void process(WatchedEvent event) {
        reloadConfig();
    }
}, null);

// Update config
zk.setData("/config", newConfig.getBytes(), -1);

4. Service Discovery Pattern

Use Case: Register and discover services

Pattern:

// Register service
zk.create("/services/my-service/instance-",
    serviceUrl.getBytes(),
    ZooDefs.Ids.OPEN_ACL_UNSAFE,
    CreateMode.EPHEMERAL_SEQUENTIAL);

// Discover services
List<String> instances = zk.getChildren("/services/my-service", false);
for (String instance : instances) {
    byte[] data = zk.getData("/services/my-service/" + instance, false, null);
    String serviceUrl = new String(data);
    // Use service
}

Best Practices

Performance Optimization

1. Connection Pooling:

// Reuse ZooKeeper connection
ZooKeeper zk = new ZooKeeper("localhost:2181", 3000, watcher);
// Reuse for multiple operations

2. Batch Operations:

// Batch multiple operations
List<Op> ops = Arrays.asList(
    Op.create("/path1", data1, ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT),
    Op.create("/path2", data2, ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT)
);
zk.multi(ops);

3. Async Operations:

// Async operations for better performance
zk.create("/path", data, ZooDefs.Ids.OPEN_ACL_UNSAFE, 
    CreateMode.PERSISTENT,
    new AsyncCallback.StringCallback() {
        @Override
        public void processResult(int rc, String path, Object ctx, String name) {
            // Handle result
        }
    },
    null);

Reliability

1. Session Management:

// Handle session expiration
ZooKeeper zk = new ZooKeeper("localhost:2181", 3000, new Watcher() {
    @Override
    public void process(WatchedEvent event) {
        if (event.getState() == Event.KeeperState.Expired) {
            // Reconnect
            reconnect();
        }
    }
});

2. Retry Logic:

public void createWithRetry(String path, byte[] data) {
    int maxRetries = 3;
    for (int i = 0; i < maxRetries; i++) {
        try {
            zk.create(path, data, ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);
            return;
        } catch (KeeperException.NodeExistsException e) {
            // Node already exists, OK
            return;
        } catch (Exception e) {
            if (i == maxRetries - 1) throw e;
            Thread.sleep(1000 * (i + 1)); // Exponential backoff
        }
    }
}

3. Watch Management:

// Re-register watches after events
zk.getData("/path", new Watcher() {
    @Override
    public void process(WatchedEvent event) {
        if (event.getType() == Event.EventType.NodeDataChanged) {
            // Re-register watch
            try {
                zk.getData("/path", this, null);
            } catch (Exception e) {
                e.printStackTrace();
            }
        }
    }
}, null);

Common Patterns

Barrier Pattern

Synchronize multiple processes:

public class Barrier {
    private ZooKeeper zk;
    private String barrierPath = "/barrier";
    private int barrierSize;
    
    public void enterBarrier() throws Exception {
        String node = zk.create(barrierPath + "/node-",
            new byte[0],
            ZooDefs.Ids.OPEN_ACL_UNSAFE,
            CreateMode.EPHEMERAL_SEQUENTIAL);
        
        while (true) {
            List<String> nodes = zk.getChildren(barrierPath, true);
            if (nodes.size() >= barrierSize) {
                return; // Barrier reached
            }
            Thread.sleep(100);
        }
    }
}

Queue Pattern

Implement distributed queue:

public class DistributedQueue {
    private ZooKeeper zk;
    private String queuePath = "/queue";
    
    public void enqueue(byte[] data) throws Exception {
        zk.create(queuePath + "/item-",
            data,
            ZooDefs.Ids.OPEN_ACL_UNSAFE,
            CreateMode.PERSISTENT_SEQUENTIAL);
    }
    
    public byte[] dequeue() throws Exception {
        while (true) {
            List<String> items = zk.getChildren(queuePath, false);
            if (items.isEmpty()) {
                Thread.sleep(100);
                continue;
            }
            
            Collections.sort(items);
            String firstItem = items.get(0);
            
            try {
                byte[] data = zk.getData(queuePath + "/" + firstItem, false, null);
                zk.delete(queuePath + "/" + firstItem, -1);
                return data;
            } catch (KeeperException.NoNodeException e) {
                // Item was deleted by another process, retry
                continue;
            }
        }
    }
}

What Interviewers Look For

Distributed Coordination Skills

  1. Coordination Patterns
    • Distributed locks
    • Leader election
    • Configuration management
    • Service discovery
    • Red Flags: No coordination patterns, wrong patterns, poor implementation
  2. Consensus Understanding
    • ZAB algorithm
    • Quorum and voting
    • Consistency guarantees
    • Red Flags: No consensus understanding, wrong algorithm, no guarantees
  3. ZNode Management
    • ZNode types (persistent, ephemeral, sequential)
    • Watchers and notifications
    • Session management
    • Red Flags: Wrong ZNode types, no watchers, poor session handling

Problem-Solving Approach

  1. Coordination Patterns
    • Can implement distributed locks
    • Can implement leader election
    • Can manage configuration
    • Red Flags: Can’t implement patterns, wrong implementation, poor patterns
  2. Fault Tolerance
    • Session expiration handling
    • Connection management
    • Retry logic
    • Red Flags: No fault tolerance, no session handling, poor retry

System Design Skills

  1. Distributed Systems
    • Understanding of coordination needs
    • Consensus algorithms
    • High availability
    • Red Flags: No coordination understanding, wrong algorithms, poor availability
  2. Use Case Application
    • When to use Zookeeper
    • Coordination scenarios
    • Alternative solutions
    • Red Flags: Wrong use cases, no alternatives, poor understanding

Communication Skills

  1. Clear Explanation
    • Explains coordination concepts
    • Discusses trade-offs
    • Justifies design decisions
    • Red Flags: Unclear explanations, no justification, confusing

Meta-Specific Focus

  1. Distributed Coordination Expertise
    • Understanding of coordination
    • Zookeeper mastery
    • Real-world application
    • Key: Demonstrate coordination expertise

Summary

Apache Zookeeper Key Points:

  • Distributed Coordination: Centralized coordination service
  • Consensus: ZAB algorithm for consistency
  • Locks: Distributed locking for resource coordination
  • Leader Election: Elect leaders from groups
  • Configuration: Centralized configuration management
  • Service Discovery: Register and discover services

Common Use Cases:

  • Distributed locks (resource coordination)
  • Leader election (single leader from group)
  • Configuration management (centralized config)
  • Service discovery (register/discover services)
  • Group membership (track active nodes)

Best Practices:

  • Use ephemeral nodes for temporary data
  • Reuse ZooKeeper connections
  • Handle session expiration
  • Implement retry logic
  • Re-register watches after events

Apache Zookeeper is essential for building distributed systems that require coordination, synchronization, and consensus among multiple nodes.