Exploring Apache ZooKeeper

Exploring Apache ZooKeeper :: Part-3

Bhaskar S

05/31/2014

Overview

In Part-2 we explored the ZooKeeper Java API and got our hands dirty with the ZooKeeper primitives exposed via the Java API.

In this part, we will use the primitives defined in the ZkHelper helper class to demonstrate the following use-cases:

Configuration Management
Group Membership
Distributed Synchronization

ZooKeeper Use-Cases

Configuration Management

Traditionally, config information associated with an application is bundled together with the application binaries and deployed as a package.

What if we have to deploy the application in a large enterprise cluster of 1000s of nodes ?

More important, what if we have to change the config information on those 1000s of nodes ?

We can leverage ZooKeeper to host and manage the config information for an application (size cannot exceed 1 MB) and have the application first connect to the ensemble to bootstrap the config information.

We will now demonstrate this use-case using ZooKeeper.

The following Java class com.polarsparc.zookeeper.ZNodeConfigSetup is used for one-time config setup:

Listing.1

/*
 * 
 * Name:   ZNodeConfigSetup
 * 
 * Author: Bhaskar S
 * 
 * Date:   05/31/2014
 * 
 */

package com.polarsparc.zookeeper;

import java.io.File;
import java.io.FileInputStream;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import org.apache.zookeeper.CreateMode;
import org.apache.zookeeper.ZooKeeper;

// ----- Usage -----
// java com.polarsparc.zookeeper.ZNodeConfigSetup <config-file> <host-port-list>
//
public class ZNodeConfigSetup {
    private static final int SESSION_TIMEOUT = 10000;
    private static final String path1 = "/ConfigMgmt";
    private static final String path2 = "/ConfigMgmt/config";
    private static final byte[] data1 = "".getBytes();
    
    private static final Logger log = LoggerFactory.getLogger(ZNodeConfigSetup.class);
    
    // ----- Main Method -----
    
    public static void main(String[] args) {
        if (args.length != 2) {
            System.out.printf("Specify <config-file> <host-port-list>\n");
            System.exit(1);
        }
        
        try {
            byte[] data2 = null;
            
            File file = new File(args[0]);
            
            try (FileInputStream fin = new FileInputStream(file)) {
                data2 = new byte[(int)file.length()];
                fin.read(data2);
            }
            catch (Exception ex) {
                log.error("ZNodeConfigSetup.main(): Exception", ex);
            }
            
            ZooKeeper zk = ZkHelper.connect(args[1], SESSION_TIMEOUT);
            
            boolean status = ZkHelper.checkZnode(zk, path1, null);
            
            log.info("Check ZNode " + path1 + ", Status = " + status);
            
            if (status == false) {
                ZkHelper.createZnode(zk, CreateMode.PERSISTENT, path1, data1);
            }
            
            status = ZkHelper.checkZnode(zk, path2, null);
            
            log.info("Check ZNode " + path2 + ", Status = " + status);
            
            if (status == false) {
                ZkHelper.createZnode(zk, CreateMode.PERSISTENT, path2, data2);
            }
            
            String str = ZkHelper.getDataZnode(zk, path2);
            
            log.info("Get ZNode " + path2 + ", Data = " + str);
            
            ZkHelper.close(zk);
        }
        catch (Exception ex) {
            log.error("ZNodeConfigSetup.main(): Exception", ex);
        }
    }
}

The logic in ZNodeConfigSetup is to perform the following:

Read the contents of the specified properties file as a byte array
Create a persistent znode with the path /ConfigMgmt/config
Set the data for /ConfigMgmt/config with the contents of the byte array

The following Java class com.polarsparc.zookeeper.ZNodeConfigClient is a simple client reading the config information:

Listing.2

/*
 * 
 * Name:   ZNodeConfigClient
 * 
 * Author: Bhaskar S
 * 
 * Date:   05/31/2014
 * 
 */

package com.polarsparc.zookeeper;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import org.apache.zookeeper.ZooKeeper;
import org.apache.zookeeper.Watcher;
import org.apache.zookeeper.WatchedEvent;

// ----- Usage -----
// java com.polarsparc.zookeeper.ZNodeConfigClient <host-port-list>
//
public class ZNodeConfigClient {
    private static final int SESSION_TIMEOUT = 10000;
    private static final String path = "/ConfigMgmt/config";
    
    private static final Logger log = LoggerFactory.getLogger(ZNodeConfigClient.class);
    
    // ----- Main Method -----
    
    public static void main(String[] args) {
        if (args.length != 1) {
            System.out.printf("Specify <host-port-list>\n");
            System.exit(1);
        }
        
        try {
            ZooKeeper zk = ZkHelper.connect(args[0], SESSION_TIMEOUT);
            
            boolean status = ZkHelper.checkZnode(zk, path, new MyWatcher(zk));
            
            log.info("Check ZNode " + path + ", Status = " + status);
            
            if (status == true) {
                String str = ZkHelper.getDataZnode(zk, path);

                log.info("Get ZNode " + path + ", Data = " + str);
            }
            
            log.info("Ready to sleep for 60 seconds !!!");
            
            Thread.sleep(60 * 1000); // 60 Seconds
            
            ZkHelper.close(zk);
            
            log.info("Done !!!");
        }
        catch (Exception ex) {
            log.error("ZNodeConfigClient.main(): Exception", ex);
        }
    }
    
    static class MyWatcher implements Watcher {
        private final ZooKeeper zk;
        
        MyWatcher(ZooKeeper zk) {
            this.zk = zk;
        }
        
        @Override
        public void process(WatchedEvent event) {
            log.info("-----> Notification for ZNode " + event.getPath() + " <-----");
            
            if (event.getType() == Watcher.Event.EventType.NodeDataChanged) {
                String str = ZkHelper.getDataZnode(zk, ZNodeConfigClient.path);
            
                log.info("Get ZNode " + ZNodeConfigClient.path + ", Data = " + str);
            }
        }
    }
}

The logic in ZNodeConfigClient is to perform the following:

Connect to the ensemble and fetch the configuration information at the znode with the path /ConfigMgmt/config
Setup a custom org.apache.zookeeper.Watcher on /ConfigMgmt/config for data changes
Refresh the configuration information via the custom Watcher notification

The following Java class com.polarsparc.zookeeper.ZNodeConfigUpdate is for updating the config information:

Listing.3

/*
 * 
 * Name:   ZNodeConfigUpdate
 * 
 * Author: Bhaskar S
 * 
 * Date:   05/31/2014
 * 
 */

package com.polarsparc.zookeeper;

import java.io.File;
import java.io.FileInputStream;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import org.apache.zookeeper.ZooKeeper;

// ----- Usage -----
// java com.polarsparc.zookeeper.ZNodeConfigUpdate <config-file> <host-port-list>
//
public class ZNodeConfigUpdate {
    private static final int SESSION_TIMEOUT = 10000;
    private static final String path = "/ConfigMgmt/config";
    
    private static final Logger log = LoggerFactory.getLogger(ZNodeConfigUpdate.class);
    
    // ----- Main Method -----
    
    public static void main(String[] args) {
        if (args.length != 2) {
            System.out.printf("Specify <config-file> <host-port-list>\n");
            System.exit(1);
        }
        
        try {
            byte[] data = null;
            
            File file = new File(args[0]);
            
            try (FileInputStream fin = new FileInputStream(file)) {
                data = new byte[(int)file.length()];
                fin.read(data);
            }
            catch (Exception ex) {
                log.error("ZNodeConfigUpdate.main(): Exception", ex);
            }
            
            ZooKeeper zk = ZkHelper.connect(args[1], SESSION_TIMEOUT);
            
            boolean status = ZkHelper.checkZnode(zk, path, null);
            
            log.info("Check ZNode " + path + ", Status = " + status);
            
            if (status == true) {
                status = ZkHelper.setDataZnode(zk, path, data);

                log.info("Set ZNode Data Status = " + status);
            }
            
            ZkHelper.close(zk);
        }
        catch (Exception ex) {
            log.error("ZNodeConfigSetup.main(): Exception", ex);
        }
    }    
}

The logic in ZNodeConfigUpdate is to perform the following:

Read the contents of the specified properties file as a byte array
Set the data for /ConfigMgmt/config with the contents of the byte array

For our tests, we will use two different properties files for the configuration information: config-test-1.properties and config-test-2.properties.

The following are the contents of config-test-1.properties:

#
# For Config Management test
#

app.log.level=INFO

db.driver=com.mysql.jdbc.Driver
db.url=jdbc:mysql://localhost:3306/testDB1
db.user=testUser1
db.secret=testSecret1

And, the following are the contents of config-test-2.properties:

#
# For Config Management test
#

app.log.level=DEBUG

db.driver=com.mysql.jdbc.Driver
db.url=jdbc:mysql://localhost:3306/testDB2
db.user=testUser2
db.secret=testSecret2

Open a Terminal window (Terminal-1) and execute the Java code com.polarsparc.zookeeper.ZNodeConfigSetup as follows:

java -cp /home/zkuser/zookeeper/resources\:/home/zkuser/zookeeper/lib/jline-0.9.94.jar\:/home/zkuser/zookeeper/lib/log4j-1.2.16.jar\:/home/zkuser/zookeeper/lib/netty-3.7.0.Final.jar\:/home/zkuser/zookeeper/lib/slf4j-api-1.6.1.jar\:/home/zkuser/zookeeper/lib/slf4j-log4j12-1.6.1.jar\:/home/zkuser/zookeeper/lib/zookeeper-3.4.6.jar\:/home/zkuser/zookeeper/build/classes com.polarsparc.zookeeper.ZNodeConfigSetup /home/zkuser/zookeeper/resources/config-test-1.properties 192.168.1.1:2181,192.168.1.2:2181,192.168.1.3:2181

The following will be the output:

Output.1

2014-06-01 09:17:19 <main> INFO  ZooKeeper:100 - Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
2014-06-01 09:17:19 <main> INFO  ZooKeeper:100 - Client environment:host.name=my-host
2014-06-01 09:17:19 <main> INFO  ZooKeeper:100 - Client environment:java.version=1.8.0_05
2014-06-01 09:17:19 <main> INFO  ZooKeeper:100 - Client environment:java.vendor=Oracle Corporation
2014-06-01 09:17:19 <main> INFO  ZooKeeper:100 - Client environment:java.home=/usr/lib/jvm/java-8-oracle/jre
2014-06-01 09:17:19 <main> INFO  ZooKeeper:100 - Client environment:java.class.path=/home/zkuser/zookeeper/resources:/home/zkuser/zookeeper/lib/jline-0.9.94.jar:/home/zkuser/zookeeper/lib/log4j-1.2.16.jar:/home/zkuser/zookeeper/lib/netty-3.7.0.Final.jar:/home/zkuser/zookeeper/lib/slf4j-api-1.6.1.jar:/home/zkuser/zookeeper/lib/slf4j-log4j12-1.6.1.jar:/home/zkuser/zookeeper/lib/zookeeper-3.4.6.jar:/home/zkuser/zookeeper/build/classes
2014-06-01 09:17:19 <main> INFO  ZooKeeper:100 - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2014-06-01 09:17:19 <main> INFO  ZooKeeper:100 - Client environment:java.io.tmpdir=/tmp
2014-06-01 09:17:19 <main> INFO  ZooKeeper:100 - Client environment:java.compiler=<NA>
2014-06-01 09:17:19 <main> INFO  ZooKeeper:100 - Client environment:os.name=Linux
2014-06-01 09:17:19 <main> INFO  ZooKeeper:100 - Client environment:os.arch=amd64
2014-06-01 09:17:19 <main> INFO  ZooKeeper:100 - Client environment:os.version=3.13.0-27-generic
2014-06-01 09:17:19 <main> INFO  ZooKeeper:100 - Client environment:user.name=zkuser
2014-06-01 09:17:19 <main> INFO  ZooKeeper:100 - Client environment:user.home=/home/zkuser
2014-06-01 09:17:19 <main> INFO  ZooKeeper:100 - Client environment:user.dir=/home/zkuser
2014-06-01 09:17:19 <main> INFO  ZooKeeper:438 - Initiating client connection, connectString=192.168.1.1:2181,192.168.1.2:2181,192.168.1.3:2181 sessionTimeout=10000 watcher=com.polarsparc.zookeeper.ZkHelper$$Lambda$1/758705033@4d76f3f8
2014-06-01 09:17:19 <main-SendThread(192.168.1.1:2181)> INFO  ClientCnxn:975 - Opening socket connection to server 192.168.1.1/192.168.1.1:2181. Will not attempt to authenticate using SASL (unknown error)
2014-06-01 09:17:19 <main-SendThread(192.168.1.1:2181)> INFO  ClientCnxn:852 - Socket connection established to 192.168.1.1/192.168.1.1:2181, initiating session
2014-06-01 09:17:20 <main-SendThread(192.168.1.1:2181)> INFO  ClientCnxn:1235 - Session establishment complete on server 192.168.1.1/192.168.1.1:2181, sessionid = 0x146577d45700000, negotiated timeout = 10000
2014-06-01 09:17:20 <main-EventThread> INFO  ZkHelper:45 - ZK[192.168.1.1:2181,192.168.1.2:2181,192.168.1.3:2181] -> Connected !!!
2014-06-01 09:17:20 <main> INFO  ZNodeConfigSetup:58 - Check ZNode /ConfigMgmt, Status = false
2014-06-01 09:17:20 <main> INFO  ZNodeConfigSetup:66 - Check ZNode /ConfigMgmt/config, Status = false
2014-06-01 09:17:20 <main> INFO  ZNodeConfigSetup:74 - Get ZNode /ConfigMgmt/config, Data = #
# For Config Management test
#

app.log.level=INFO

db.driver=com.mysql.jdbc.Driver
db.url=jdbc:mysql://localhost:3306/testDB1
db.user=testUser1
db.secret=testSecret1

2014-06-01 09:17:20 <main> INFO  ZooKeeper:684 - Session: 0x146577d45700000 closed
2014-06-01 09:17:20 <main-EventThread> INFO  ClientCnxn:512 - EventThread shut down

As can be inferred from the above Output.1, we created a znode with path /ConfigMgmt/config and set the contents of the properties file config-test-2.properties as the data.

Now, open another Terminal window (Terminal-2) and execute the Java code com.polarsparc.zookeeper.ZNodeConfigClient as follows:

java -cp /home/zkuser/zookeeper/resources\:/home/zkuser/zookeeper/lib/jline-0.9.94.jar\:/home/zkuser/zookeeper/lib/log4j-1.2.16.jar\:/home/zkuser/zookeeper/lib/netty-3.7.0.Final.jar\:/home/zkuser/zookeeper/lib/slf4j-api-1.6.1.jar\:/home/zkuser/zookeeper/lib/slf4j-log4j12-1.6.1.jar\:/home/zkuser/zookeeper/lib/zookeeper-3.4.6.jar\:/home/zkuser/zookeeper/build/classes com.polarsparc.zookeeper.ZNodeConfigClient 192.168.1.1:2181,192.168.1.2:2181,192.168.1.3:2181

The following will be the output:

Output.2

2014-06-01 09:20:42 <main> INFO  ZooKeeper:100 - Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
2014-06-01 09:20:42 <main> INFO  ZooKeeper:100 - Client environment:host.name=my-host
2014-06-01 09:20:42 <main> INFO  ZooKeeper:100 - Client environment:java.version=1.8.0_05
2014-06-01 09:20:42 <main> INFO  ZooKeeper:100 - Client environment:java.vendor=Oracle Corporation
2014-06-01 09:20:42 <main> INFO  ZooKeeper:100 - Client environment:java.home=/usr/lib/jvm/java-8-oracle/jre
2014-06-01 09:20:42 <main> INFO  ZooKeeper:100 - Client environment:java.class.path=/home/zkuser/zookeeper/resources:/home/zkuser/zookeeper/lib/jline-0.9.94.jar:/home/zkuser/zookeeper/lib/log4j-1.2.16.jar:/home/zkuser/zookeeper/lib/netty-3.7.0.Final.jar:/home/zkuser/zookeeper/lib/slf4j-api-1.6.1.jar:/home/zkuser/zookeeper/lib/slf4j-log4j12-1.6.1.jar:/home/zkuser/zookeeper/lib/zookeeper-3.4.6.jar:/home/zkuser/zookeeper/build/classes
2014-06-01 09:20:42 <main> INFO  ZooKeeper:100 - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2014-06-01 09:20:42 <main> INFO  ZooKeeper:100 - Client environment:java.io.tmpdir=/tmp
2014-06-01 09:20:42 <main> INFO  ZooKeeper:100 - Client environment:java.compiler=<NA>
2014-06-01 09:20:42 <main> INFO  ZooKeeper:100 - Client environment:os.name=Linux
2014-06-01 09:20:42 <main> INFO  ZooKeeper:100 - Client environment:os.arch=amd64
2014-06-01 09:20:42 <main> INFO  ZooKeeper:100 - Client environment:os.version=3.13.0-27-generic
2014-06-01 09:20:42 <main> INFO  ZooKeeper:100 - Client environment:user.name=zkuser
2014-06-01 09:20:42 <main> INFO  ZooKeeper:100 - Client environment:user.home=/home/zkuser
2014-06-01 09:20:42 <main> INFO  ZooKeeper:100 - Client environment:user.dir=/home/zkuser
2014-06-01 09:20:42 <main> INFO  ZooKeeper:438 - Initiating client connection, connectString=192.168.1.1:2181,192.168.1.2:2181,192.168.1.3:2181 sessionTimeout=10000 watcher=com.polarsparc.zookeeper.ZkHelper$$Lambda$1/758705033@4d76f3f8
2014-06-01 09:20:42 <main-SendThread(192.168.1.3:2181)> INFO  ClientCnxn:975 - Opening socket connection to server 192.168.1.3/192.168.1.3:2181. Will not attempt to authenticate using SASL (unknown error)
2014-06-01 09:20:42 <main-SendThread(192.168.1.3:2181)> INFO  ClientCnxn:852 - Socket connection established to 192.168.1.3/192.168.1.3:2181, initiating session
2014-06-01 09:20:42 <main-SendThread(192.168.1.3:2181)> INFO  ClientCnxn:1235 - Session establishment complete on server 192.168.1.3/192.168.1.3:2181, sessionid = 0x346577d3eed0000, negotiated timeout = 10000
2014-06-01 09:20:42 <main-EventThread> INFO  ZkHelper:45 - ZK[192.168.1.1:2181,192.168.1.2:2181,192.168.1.3:2181] -> Connected !!!
2014-06-01 09:20:43 <main> INFO  ZkHelper:165 - ZK[/ConfigMgmt/config] -> Exists, Stat: {ctime = Sun Jun 01 09:17:20 EDT 2014, mtime = Sun Jun 01 09:17:20 EDT 2014, version = 0}
2014-06-01 09:20:43 <main> INFO  ZNodeConfigClient:42 - Check ZNode /ConfigMgmt/config, Status = true
2014-06-01 09:20:43 <main> INFO  ZNodeConfigClient:47 - Get ZNode /ConfigMgmt/config, Data = #
# For Config Management test
#

app.log.level=INFO

db.driver=com.mysql.jdbc.Driver
db.url=jdbc:mysql://localhost:3306/testDB1
db.user=testUser1
db.secret=testSecret1

2014-06-01 09:20:43 <main> INFO  ZNodeConfigClient:50 - Ready to sleep for 60 seconds !!!

As can be inferred from the above Output.2, the simple client was able to fetch the config information as expected. The client is also watching for any changes to the config information at the znode with path /ConfigMgmt/config.

Now, open a third Terminal window (Terminal-3) and execute the Java code com.polarsparc.zookeeper.ZNodeConfigUpdate as follows:

java -cp /home/zkuser/zookeeper/resources\:/home/zkuser/zookeeper/lib/jline-0.9.94.jar\:/home/zkuser/zookeeper/lib/log4j-1.2.16.jar\:/home/zkuser/zookeeper/lib/netty-3.7.0.Final.jar\:/home/zkuser/zookeeper/lib/slf4j-api-1.6.1.jar\:/home/zkuser/zookeeper/lib/slf4j-log4j12-1.6.1.jar\:/home/zkuser/zookeeper/lib/zookeeper-3.4.6.jar\:/home/zkuser/zookeeper/build/classes com.polarsparc.zookeeper.ZNodeConfigUpdate /home/zkuser/zookeeper/resources/config-test-2.properties 192.168.1.1:2181,192.168.1.2:2181,192.168.1.3:2181

The following will be the output:

Output.3

2014-06-01 09:21:04 <main> INFO  ZooKeeper:100 - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2014-06-01 09:21:04 <main> INFO  ZooKeeper:100 - Client environment:java.io.tmpdir=/tmp
2014-06-01 09:21:04 <main> INFO  ZooKeeper:100 - Client environment:java.compiler=<NA>
2014-06-01 09:21:04 <main> INFO  ZooKeeper:100 - Client environment:os.name=Linux
2014-06-01 09:21:04 <main> INFO  ZooKeeper:100 - Client environment:os.arch=amd64
2014-06-01 09:21:04 <main> INFO  ZooKeeper:100 - Client environment:os.version=3.13.0-27-generic
2014-06-01 09:21:04 <main> INFO  ZooKeeper:100 - Client environment:user.name=zkuser
2014-06-01 09:21:04 <main> INFO  ZooKeeper:100 - Client environment:user.home=/home/zkuser
2014-06-01 09:21:04 <main> INFO  ZooKeeper:100 - Client environment:user.dir=/home/zkuser
2014-06-01 09:21:04 <main> INFO  ZooKeeper:438 - Initiating client connection, connectString=192.168.1.1:2181,192.168.1.2:2181,192.168.1.3:2181 sessionTimeout=10000 watcher=com.polarsparc.zookeeper.ZkHelper$$Lambda$1/758705033@4d76f3f8
2014-06-01 09:21:04 <main-SendThread(192.168.1.1:2181)> INFO  ClientCnxn:975 - Opening socket connection to server 192.168.1.1/192.168.1.1:2181. Will not attempt to authenticate using SASL (unknown error)
2014-06-01 09:21:04 <main-SendThread(192.168.1.1:2181)> INFO  ClientCnxn:852 - Socket connection established to 192.168.1.1/192.168.1.1:2181, initiating session
2014-06-01 09:21:04 <main-SendThread(192.168.1.1:2181)> INFO  ClientCnxn:1235 - Session establishment complete on server 192.168.1.1/192.168.1.1:2181, sessionid = 0x146577d45700001, negotiated timeout = 10000
2014-06-01 09:21:04 <main-EventThread> INFO  ZkHelper:45 - ZK[192.168.1.1:2181,192.168.1.2:2181,192.168.1.3:2181] -> Connected !!!
2014-06-01 09:21:05 <main> INFO  ZkHelper:165 - ZK[/ConfigMgmt/config] -> Exists, Stat: {ctime = Sun Jun 01 09:17:20 EDT 2014, mtime = Sun Jun 01 09:17:20 EDT 2014, version = 0}
2014-06-01 09:21:05 <main> INFO  ZNodeConfigUpdate:55 - Check ZNode /ConfigMgmt/config, Status = true
2014-06-01 09:21:05 <main> INFO  ZNodeConfigUpdate:60 - Set ZNode Data Status = true
2014-06-01 09:21:05 <main> INFO  ZooKeeper:684 - Session: 0x146577d45700001 closed
2014-06-01 09:21:05 <main-EventThread> INFO  ClientCnxn:512 - EventThread shut down

As can be inferred from the above Output.3, config information at the znode with path /ConfigMgmt/config has been updated with the contents of the properties file config-test-2.properties.

Simultaneously, observe the output in Terminal-2 as shown below:

Output.4

2014-06-01 09:21:05 <main-EventThread> INFO  ZNodeConfigClient:72 - -----> Notification for ZNode /ConfigMgmt/config <-----
2014-06-01 09:21:05 <main-EventThread> INFO  ZNodeConfigClient:77 - Get ZNode /ConfigMgmt/config, Data = #
# For Config Management test
#

app.log.level=DEBUG

db.driver=com.mysql.jdbc.Driver
db.url=jdbc:mysql://localhost:3306/testDB2
db.user=testUser2
db.secret=testSecret2

2014-06-01 09:21:43 <main> INFO  ZooKeeper:684 - Session: 0x346577d3eed0000 closed
2014-06-01 09:21:43 <main> INFO  ZNodeConfigClient:56 - Done !!!
2014-06-01 09:21:43 <main-EventThread> INFO  ClientCnxn:512 - EventThread shut down

As can be inferred from the above Output.4, the simple client running in Terminal-2 was notified of the changes to the config information at the znode with path /ConfigMgmt/config.

Group Membership

When we deploy server-based applications (applications that accept connections from clients and service their requests) in a cluster of nodes, we need to know which nodes are active and ready to serve clients.

We can leverage ZooKeeper to track active nodes in a cluster (group membership).

The following Java class com.polarsparc.zookeeper.ZNodeGroupMonitor monitors for active members in a cluster:

Listing.4

/*
 * 
 * Name:   ZNodeGroupMonitor
 * 
 * Author: Bhaskar S
 * 
 * Date:   05/31/2014
 * 
 */

package com.polarsparc.zookeeper;

import java.util.List;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import org.apache.zookeeper.CreateMode;
import org.apache.zookeeper.ZooKeeper;

// ----- Usage -----
// java com.polarsparc.zookeeper.ZNodeGroupMonitor <count> <delay> <host-port-list>
//
public class ZNodeGroupMonitor {
    private static final int SESSION_TIMEOUT = 10000;
    private static final String parent = "/GroupMember";
    private static final byte[] data = "".getBytes();
    
    private static final Logger log = LoggerFactory.getLogger(ZNodeGroupMonitor.class);
    
    // ----- Main Method -----
    
    public static void main(String[] args) {
        if (args.length != 3) {
            System.out.printf("Specify <count> <delay> <host-port-list>\n");
            System.exit(1);
        }
        
        try {
            int count = Integer.parseInt(args[0]);
            
            long delay = Long.parseLong(args[1]);
            
            ZooKeeper zk = ZkHelper.connect(args[2], SESSION_TIMEOUT);
            
            boolean status = ZkHelper.checkZnode(zk, parent, null);
            
            log.info("Check ZNode " + parent + ", Status = " + status);
            
            if (status == false) {
                ZkHelper.createZnode(zk, CreateMode.PERSISTENT, parent, data);
            }
            
            status = ZkHelper.checkZnode(zk, parent, null);
            
            if (status == true) {
                for (int i = 0; i < count; i++) {
                    List<String> list = ZkHelper.getZnodeChildren(zk, parent, null);
                    if (list != null) {
                        log.info("Members = " + list);
                    }
                    
                    Thread.sleep(delay);
                }
            }
            
            ZkHelper.close(zk);
        }
        catch (Exception ex) {
            log.error("ZNodeGroupMonitor.main(): Exception", ex);
        }
    }
}

The logic in ZNodeGroupMonitor is to perform the following:

Check if the znode with path /GroupMember exists. If not, create it
In a loop, list all the children under /GroupMember and diplay their names. These are the active members of the cluster

The following Java class com.polarsparc.zookeeper.ZNodeGroupMember is a simple group member:

Listing.5

/*
 * 
 * Name:   ZNodeGroupMember
 * 
 * Author: Bhaskar S
 * 
 * Date:   05/31/2014
 * 
 */

package com.polarsparc.zookeeper;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import org.apache.zookeeper.CreateMode;
import org.apache.zookeeper.ZooKeeper;

// ----- Usage -----
// java com.polarsparc.zookeeper.ZNodeGroupMember <unique-name> <delay> <host-port-list>
//
public class ZNodeGroupMember {
    private static final int SESSION_TIMEOUT = 10000;
    private static final String SLASH = "/";
    private static final String parent = "/GroupMember";
    private static final byte[] data = "".getBytes();
    
    private static final Logger log = LoggerFactory.getLogger(ZNodeGroupMember.class);
    
    // ----- Main Method -----
    
    public static void main(String[] args) {
        if (args.length != 3) {
            System.out.printf("Specify <unique-name> <delay> <host-port-list>\n");
            System.exit(1);
        }
        
        try {
            String path = parent + SLASH + args[0];
            
            long delay = Long.parseLong(args[1]);
            
            ZooKeeper zk = ZkHelper.connect(args[2], SESSION_TIMEOUT);
            
            boolean status = ZkHelper.checkZnode(zk, parent, null);
            
            log.info("Check ZNode " + parent + ", Status = " + status);
            
            if (status == false) {
                ZkHelper.createZnode(zk, CreateMode.PERSISTENT, parent, data);
            }
            
            log.info("Client member ZNode = " + path);
            
            status = ZkHelper.checkZnode(zk, path, null);
            
            log.info("Check ZNode " + path + ", Status = " + status);
            
            if (status == false) {
                ZkHelper.createZnode(zk, CreateMode.EPHEMERAL, path, data);
            }
            
            Thread.sleep(delay);
            
            ZkHelper.close(zk);
        }
        catch (Exception ex) {
            log.error("ZNodeGroupMember.main(): Exception", ex);
        }
    }
}

The logic in ZNodeGroupMember is to perform the following:

Connect to the ensemble and create an ephemeral znode under /GroupMember
Sleep for a specified duration and then exit to simulate an inactive member

Open a Terminal window (Terminal-1) and execute the Java code com.polarsparc.zookeeper.ZNodeGroupMonitor as follows:

java -cp /home/zkuser/zookeeper/resources:/home/zkuser/zookeeper/lib/jline-0.9.94.jar:/home/zkuser/zookeeper/lib/log4j-1.2.16.jar:/home/zkuser/zookeeper/lib/netty-3.7.0.Final.jar:/home/zkuser/zookeeper/lib/slf4j-api-1.6.1.jar:/home/zkuser/zookeeper/lib/slf4j-log4j12-1.6.1.jar:/home/zkuser/zookeeper/lib/zookeeper-3.4.6.jar:/home/zkuser/zookeeper/build/classes com.polarsparc.zookeeper.ZNodeGroupMonitor 20 5000 192.168.1.1:2181,192.168.1.2:2181,192.168.1.3:2181

The following will be the output:

Output.5

2014-06-01 13:12:18 <main> INFO  ZooKeeper:100 - Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
2014-06-01 13:12:18 <main> INFO  ZooKeeper:100 - Client environment:host.name=my-host
2014-06-01 13:12:18 <main> INFO  ZooKeeper:100 - Client environment:java.version=1.8.0_05
2014-06-01 13:12:18 <main> INFO  ZooKeeper:100 - Client environment:java.vendor=Oracle Corporation
2014-06-01 13:12:18 <main> INFO  ZooKeeper:100 - Client environment:java.home=/usr/lib/jvm/java-8-oracle/jre
2014-06-01 13:12:18 <main> INFO  ZooKeeper:100 - Client environment:java.class.path=/home/zkuser/zookeeper/resources:/home/zkuser/zookeeper/lib/jline-0.9.94.jar:/home/zkuser/zookeeper/lib/log4j-1.2.16.jar:/home/zkuser/zookeeper/lib/netty-3.7.0.Final.jar:/home/zkuser/zookeeper/lib/slf4j-api-1.6.1.jar:/home/zkuser/zookeeper/lib/slf4j-log4j12-1.6.1.jar:/home/zkuser/zookeeper/lib/zookeeper-3.4.6.jar:/home/zkuser/zookeeper/build/classes
2014-06-01 13:12:18 <main> INFO  ZooKeeper:100 - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2014-06-01 13:12:18 <main> INFO  ZooKeeper:100 - Client environment:java.io.tmpdir=/tmp
2014-06-01 13:12:18 <main> INFO  ZooKeeper:100 - Client environment:java.compiler=<NA>
2014-06-01 13:12:18 <main> INFO  ZooKeeper:100 - Client environment:os.name=Linux
2014-06-01 13:12:18 <main> INFO  ZooKeeper:100 - Client environment:os.arch=amd64
2014-06-01 13:12:18 <main> INFO  ZooKeeper:100 - Client environment:os.version=3.13.0-27-generic
2014-06-01 13:12:18 <main> INFO  ZooKeeper:100 - Client environment:user.name=zkuser
2014-06-01 13:12:18 <main> INFO  ZooKeeper:100 - Client environment:user.home=/home/zkuser
2014-06-01 13:12:18 <main> INFO  ZooKeeper:100 - Client environment:user.dir=/home/zkuser
2014-06-01 13:12:18 <main> INFO  ZooKeeper:438 - Initiating client connection, connectString=192.168.1.1:2181,192.168.1.2:2181,192.168.1.3:2181 sessionTimeout=10000 watcher=com.polarsparc.zookeeper.ZkHelper$$Lambda$1/758705033@4d76f3f8
2014-06-01 13:12:18 <main-SendThread(192.168.1.3:2181)> INFO  ClientCnxn:975 - Opening socket connection to server 192.168.1.3/192.168.1.3:2181. Will not attempt to authenticate using SASL (unknown error)
2014-06-01 13:12:18 <main-SendThread(192.168.1.3:2181)> INFO  ClientCnxn:852 - Socket connection established to 192.168.1.3/192.168.1.3:2181, initiating session
2014-06-01 13:12:18 <main-SendThread(192.168.1.3:2181)> INFO  ClientCnxn:1235 - Session establishment complete on server 192.168.1.3/192.168.1.3:2181, sessionid = 0x34648a66eb50000, negotiated timeout = 10000
2014-06-01 13:12:18 <main-EventThread> INFO  ZkHelper:45 - ZK[192.168.1.1:2181,192.168.1.2:2181,192.168.1.3:2181] -> Connected !!!
2014-06-01 13:12:18 <main> INFO  ZNodeGroupMonitor:48 - Check ZNode /GroupMember, Status = false
2014-06-01 13:12:18 <main> INFO  ZkHelper:165 - ZK[/GroupMember] -> Exists, Stat: {ctime = Sun Jun 01 13:12:18 EDT 2014, mtime = Sun Jun 01 13:12:18 EDT 2014, version = 0}
2014-06-01 13:12:18 <main> INFO  ZNodeGroupMonitor:60 - Members = []

As can be inferred from the above Output.5, we create a znode with path /GroupMember (if it does not exist), fetch and display its children at regular interval. As can be observed, there are no active members.

Now, open 3 more Terminal windows (Terminal-2, Terminal-3, and Terminal-4) and execute the Java code com.polarsparc.zookeeper.ZNodeGroupMember with different command-line arguments as follows:

java -cp /home/zkuser/zookeeper/resources\:/home/zkuser/zookeeper/lib/jline-0.9.94.jar\:/home/zkuser/zookeeper/lib/log4j-1.2.16.jar\:/home/zkuser/zookeeper/lib/netty-3.7.0.Final.jar\:/home/zkuser/zookeeper/lib/slf4j-api-1.6.1.jar\:/home/zkuser/zookeeper/lib/slf4j-log4j12-1.6.1.jar\:/home/zkuser/zookeeper/lib/zookeeper-3.4.6.jar\:/home/zkuser/zookeeper/build/classes com.polarsparc.zookeeper.ZNodeGroupMember client-1 60000 192.168.1.1:2181,192.168.1.2:2181,192.168.1.3:2181

java -cp /home/zkuser/zookeeper/resources\:/home/zkuser/zookeeper/lib/jline-0.9.94.jar\:/home/zkuser/zookeeper/lib/log4j-1.2.16.jar\:/home/zkuser/zookeeper/lib/netty-3.7.0.Final.jar\:/home/zkuser/zookeeper/lib/slf4j-api-1.6.1.jar\:/home/zkuser/zookeeper/lib/slf4j-log4j12-1.6.1.jar\:/home/zkuser/zookeeper/lib/zookeeper-3.4.6.jar\:/home/zkuser/zookeeper/build/classes com.polarsparc.zookeeper.ZNodeGroupMember client-2 40000 192.168.1.1:2181,192.168.1.2:2181,192.168.1.3:2181

java -cp /home/zkuser/zookeeper/resources\:/home/zkuser/zookeeper/lib/jline-0.9.94.jar\:/home/zkuser/zookeeper/lib/log4j-1.2.16.jar\:/home/zkuser/zookeeper/lib/netty-3.7.0.Final.jar\:/home/zkuser/zookeeper/lib/slf4j-api-1.6.1.jar\:/home/zkuser/zookeeper/lib/slf4j-log4j12-1.6.1.jar\:/home/zkuser/zookeeper/lib/zookeeper-3.4.6.jar\:/home/zkuser/zookeeper/build/classes com.polarsparc.zookeeper.ZNodeGroupMember client-3 20000 192.168.1.1:2181,192.168.1.2:2181,192.168.1.3:2181

As we start the 3 group members, the following will be the output in Terminal-1:

Output.6

2014-06-01 13:13:09 <main> INFO  ZNodeGroupMonitor:60 - Members = [client-1]
2014-06-01 13:13:14 <main> INFO  ZNodeGroupMonitor:60 - Members = [client-1]
2014-06-01 13:13:19 <main> INFO  ZNodeGroupMonitor:60 - Members = [client-1]
2014-06-01 13:13:24 <main> INFO  ZNodeGroupMonitor:60 - Members = [client-1]
2014-06-01 13:13:29 <main> INFO  ZNodeGroupMonitor:60 - Members = [client-1, client-2]
2014-06-01 13:13:34 <main> INFO  ZNodeGroupMonitor:60 - Members = [client-1, client-2]
2014-06-01 13:13:39 <main> INFO  ZNodeGroupMonitor:60 - Members = [client-1, client-2]
2014-06-01 13:13:44 <main> INFO  ZNodeGroupMonitor:60 - Members = [client-1, client-3, client-2]
2014-06-01 13:13:49 <main> INFO  ZNodeGroupMonitor:60 - Members = [client-1, client-3, client-2]
2014-06-01 13:13:54 <main> INFO  ZNodeGroupMonitor:60 - Members = [client-1, client-3, client-2]
2014-06-01 13:13:59 <main> INFO  ZNodeGroupMonitor:60 - Members = [client-1, client-3, client-2]
2014-06-01 12:18:04 <main> INFO  ZNodeGroupMonitor:60 - Members = [client-1, client-2]
2014-06-01 12:18:09 <main> INFO  ZNodeGroupMonitor:60 - Members = []

As can be inferred from the above Output.6, we are able to monitor and identify the active members of the application.

Distributed Synchronization

Often times in an application that is distributed across nodes of the cluster there is a need for using synchronization so that only one instance of the application is in a critical section (such as updating a counter).

We will leverage ZooKeeper to demostrate distributed locking.

The following Java class com.polarsparc.zookeeper.ZNodeDistLock implements a simple distributed lock:

Listing.6

/*
 * 
 * Name:   ZNodeDistLock
 * 
 * Author: Bhaskar S
 * 
 * Date:   05/31/2014
 * 
 */

package com.polarsparc.zookeeper;

import java.util.List;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import org.apache.zookeeper.CreateMode;
import org.apache.zookeeper.ZooKeeper;
import org.apache.zookeeper.Watcher;
import org.apache.zookeeper.WatchedEvent;

// ----- Usage -----
// java com.polarsparc.zookeeper.ZNodeDistLock <unique-name> <count> <delay> <host-port-list>
//
public class ZNodeDistLock {
    private static final int SESSION_TIMEOUT = 10000;
    private static final String parent = "/DistLock";
    private static final String lock = "/DistLock/lock";
    private static final byte[] data = "1".getBytes();
    private static final byte[] data1 = "".getBytes();
    
    private static final Logger log = LoggerFactory.getLogger(ZNodeDistLock.class);
    
    private static long delay = 0;
    
    private static String client = null;
    
    // ----- Main Method -----
    
    public static void main(String[] args) {
        if (args.length != 4) {
            System.out.printf("Specify <unique-name> <count> <delay> <host-port-list>\n");
            System.exit(1);
        }
        
        try {
            client = args[0];
            int count = Integer.parseInt(args[1]);
            delay = Long.parseLong(args[2]);
            
            ZooKeeper zk = ZkHelper.connect(args[3], SESSION_TIMEOUT);
            
            boolean status = ZkHelper.checkZnode(zk, parent, null);
            
            log.info("Check ZNode " + parent + ", Status = " + status);
            
            if (status == false) {
                ZkHelper.createZnode(zk, CreateMode.PERSISTENT, parent, data);
            }
            
            status = ZkHelper.checkZnode(zk, parent, null);
            
            if (status == true) {
                for (int i = 0; i < count; i++) {
                    lock(zk);
                    
                    String str = ZkHelper.getDataZnode(zk, parent);
                    
                    log.info("Client " + client + " processing [" + str + "]");
                    
                    int x = Integer.parseInt(str);
                    
                    byte[] data2 = (Integer.toString(x+1)).getBytes();
                    
                    ZkHelper.setDataZnode(zk, parent, data2);
                    
                    unlock(zk);
                    
                    Thread.sleep(delay);
                }
            }
            
            ZkHelper.close(zk);
        }
        catch (Exception ex) {
            log.error("ZNodeDistLock.main(): Exception", ex);
        }
    }
    
    static void lock(ZooKeeper zk) {
        final Object mutex = new Object();
        
        Watcher watcher = new MyLockWatcher(mutex);
        
        for (;;) {
            List<String> list = ZkHelper.getZnodeChildren(zk, parent, watcher);
            if (list.isEmpty()) {
                // Try to grab znode
                boolean status = ZkHelper.createZnode(zk, CreateMode.EPHEMERAL, lock, data1);
                if (status) {
                    // Success
                    log.info("Client " + client + " acquired lock !!!");
                    break;
                }
                else {
                    log.info("Ouch, client " + client + " missed ...");
                    try {
                        synchronized (mutex) {
                            mutex.wait(delay);
                        }
                    }
                    catch (InterruptedException ex) {
                    }
                }
            }
            else {
                log.info("Client " + client + " have to wait ...");
                try {
                    synchronized (mutex) {
                        mutex.wait(delay);
                    }
                }
                catch (InterruptedException ex) {
                }
            }
        }
    }
    
    static void unlock(ZooKeeper zk) {
        boolean status = ZkHelper.deleteZnode(zk, lock);
        if (status) {
            log.info("Client " + client + " released lock !!!");
        }
    }
    
    static class MyLockWatcher implements Watcher {
        private final Object mutex;
        
        MyLockWatcher(Object mutex) {
            this.mutex = mutex;
        }
        
        @Override
        public void process(WatchedEvent event) {
            log.info("-----> Notification for ZNode " + event.getPath() + " <-----");
            
            if (event.getType() == Watcher.Event.EventType.NodeDataChanged) {
                synchronized (mutex) {
                    mutex.notifyAll();
                }
            }
        }
    }
}

The logic in ZNodeDistLock is to perform the following:

Check if the znode with path /DistLock exists. If it does not exist, create it with an initial data value of 1
Try to acquire a lock by creating an ephemeral znode with path /DistLock/lock
If lock was acquired successfully, increment that data value at /DistLock
If lock was not acquired, setup a watch on /DistLock and wait for a callback to retry

Now, open 3 more Terminal windows (Terminal-1, Terminal-2, and Terminal-3) and execute the Java code com.polarsparc.zookeeper.ZNodeDistLock with different command-line arguments as follows:

java -cp /home/zkuser/zookeeper/resources\:/home/zkuser/zookeeper/lib/jline-0.9.94.jar\:/home/zkuser/zookeeper/lib/log4j-1.2.16.jar\:/home/zkuser/zookeeper/lib/netty-3.7.0.Final.jar\:/home/zkuser/zookeeper/lib/slf4j-api-1.6.1.jar\:/home/zkuser/zookeeper/lib/slf4j-log4j12-1.6.1.jar\:/home/zkuser/zookeeper/lib/zookeeper-3.4.6.jar\:/home/zkuser/zookeeper/build/classes com.polarsparc.zookeeper.ZNodeDistLock client-1 10 3000 192.168.1.1:2181,192.168.1.2:2181,192.168.1.3:2181

java -cp /home/zkuser/zookeeper/resources\:/home/zkuser/zookeeper/lib/jline-0.9.94.jar\:/home/zkuser/zookeeper/lib/log4j-1.2.16.jar\:/home/zkuser/zookeeper/lib/netty-3.7.0.Final.jar\:/home/zkuser/zookeeper/lib/slf4j-api-1.6.1.jar\:/home/zkuser/zookeeper/lib/slf4j-log4j12-1.6.1.jar\:/home/zkuser/zookeeper/lib/zookeeper-3.4.6.jar\:/home/zkuser/zookeeper/build/classes com.polarsparc.zookeeper.ZNodeGroupMember client-2 10 1000 192.168.1.1:2181,192.168.1.2:2181,192.168.1.3:2181

java -cp /home/zkuser/zookeeper/resources\:/home/zkuser/zookeeper/lib/jline-0.9.94.jar\:/home/zkuser/zookeeper/lib/log4j-1.2.16.jar\:/home/zkuser/zookeeper/lib/netty-3.7.0.Final.jar\:/home/zkuser/zookeeper/lib/slf4j-api-1.6.1.jar\:/home/zkuser/zookeeper/lib/slf4j-log4j12-1.6.1.jar\:/home/zkuser/zookeeper/lib/zookeeper-3.4.6.jar\:/home/zkuser/zookeeper/build/classes com.polarsparc.zookeeper.ZNodeGroupMember client-3 10 2000 192.168.1.1:2181,192.168.1.2:2181,192.168.1.3:2181

The following will be the output in Terminal-1:

Output.7

2014-06-01 15:16:57 <main> INFO  ZooKeeper:100 - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2014-06-01 15:16:57 <main> INFO  ZooKeeper:100 - Client environment:java.io.tmpdir=/tmp
2014-06-01 15:16:57 <main> INFO  ZooKeeper:100 - Client environment:java.compiler=<NA>
2014-06-01 15:16:57 <main> INFO  ZooKeeper:100 - Client environment:os.name=Linux
2014-06-01 15:16:57 <main> INFO  ZooKeeper:100 - Client environment:os.arch=amd64
2014-06-01 15:16:57 <main> INFO  ZooKeeper:100 - Client environment:os.version=3.13.0-27-generic
2014-06-01 15:16:57 <main> INFO  ZooKeeper:100 - Client environment:user.name=zkuser
2014-06-01 15:16:57 <main> INFO  ZooKeeper:100 - Client environment:user.home=/home/zkuser
2014-06-01 15:16:57 <main> INFO  ZooKeeper:100 - Client environment:user.dir=/home/zkuser
2014-06-01 15:16:57 <main> INFO  ZooKeeper:438 - Initiating client connection, connectString=192.168.1.1:2181,192.168.1.2:2181,192.168.1.3:2181 sessionTimeout=10000 watcher=com.polarsparc.zookeeper.ZkHelper$$Lambda$1/758705033@4d76f3f8
2014-06-01 15:16:57 <main-SendThread(192.168.1.2:2181)> INFO  ClientCnxn:975 - Opening socket connection to server 192.168.1.2/192.168.1.2:2181. Will not attempt to authenticate using SASL (unknown error)
2014-06-01 15:16:57 <main-SendThread(192.168.1.2:2181)> INFO  ClientCnxn:852 - Socket connection established to 192.168.1.2/192.168.1.2:2181, initiating session
2014-06-01 15:16:57 <main-SendThread(192.168.1.2:2181)> INFO  ClientCnxn:1235 - Session establishment complete on server 192.168.1.2/192.168.1.2:2181, sessionid = 0x246577d3ec20000, negotiated timeout = 10000
2014-06-01 15:16:57 <main-EventThread> INFO  ZkHelper:45 - ZK[192.168.1.1:2181,192.168.1.2:2181,192.168.1.3:2181] -> Connected !!!
2014-06-01 15:16:57 <main> INFO  ZNodeDistLock:56 - Check ZNode /DistLock, Status = false
2014-06-01 15:16:57 <main> INFO  ZkHelper:165 - ZK[/DistLock] -> Exists, Stat: {ctime = Sun Jun 01 15:16:57 EDT 2014, mtime = Sun Jun 01 15:16:57 EDT 2014, version = 0}
2014-06-01 15:16:57 <main-EventThread> INFO  ZNodeDistLock:146 - -----> Notification for ZNode /DistLock <-----
2014-06-01 15:16:57 <main> INFO  ZNodeDistLock:103 - Client client-1 acquired lock !!!
2014-06-01 15:16:57 <main> INFO  ZNodeDistLock:70 - Client client-1 processing [1]
2014-06-01 15:16:57 <main> INFO  ZNodeDistLock:133 - Client client-1 released lock !!!
2014-06-01 15:17:00 <main-EventThread> INFO  ZNodeDistLock:146 - -----> Notification for ZNode /DistLock <-----
2014-06-01 15:17:00 <main> INFO  ZNodeDistLock:103 - Client client-1 acquired lock !!!
2014-06-01 15:17:00 <main> INFO  ZNodeDistLock:70 - Client client-1 processing [6]
2014-06-01 15:17:01 <main> INFO  ZNodeDistLock:133 - Client client-1 released lock !!!
2014-06-01 15:17:04 <main> INFO  ZNodeDistLock:118 - Client client-1 have to wait ...
2014-06-01 15:17:04 <main-EventThread> INFO  ZNodeDistLock:146 - -----> Notification for ZNode /DistLock <-----
2014-06-01 15:17:07 <main-EventThread> INFO  ZNodeDistLock:146 - -----> Notification for ZNode /DistLock <-----
2014-06-01 15:17:07 <main> INFO  ZNodeDistLock:103 - Client client-1 acquired lock !!!
2014-06-01 15:17:07 <main> INFO  ZNodeDistLock:70 - Client client-1 processing [15]
2014-06-01 15:17:07 <main> INFO  ZNodeDistLock:133 - Client client-1 released lock !!!
2014-06-01 15:17:10 <main-EventThread> INFO  ZNodeDistLock:146 - -----> Notification for ZNode /DistLock <-----
2014-06-01 15:17:10 <main> INFO  ZNodeDistLock:103 - Client client-1 acquired lock !!!
2014-06-01 15:17:10 <main> INFO  ZNodeDistLock:70 - Client client-1 processing [20]
2014-06-01 15:17:10 <main> INFO  ZNodeDistLock:133 - Client client-1 released lock !!!
2014-06-01 15:17:13 <main-EventThread> INFO  ZNodeDistLock:146 - -----> Notification for ZNode /DistLock <-----
2014-06-01 15:17:13 <main> INFO  ZNodeDistLock:103 - Client client-1 acquired lock !!!
2014-06-01 15:17:13 <main> INFO  ZNodeDistLock:70 - Client client-1 processing [22]
2014-06-01 15:17:13 <main> INFO  ZNodeDistLock:133 - Client client-1 released lock !!!
2014-06-01 15:17:16 <main-EventThread> INFO  ZNodeDistLock:146 - -----> Notification for ZNode /DistLock <-----
2014-06-01 15:17:16 <main> INFO  ZNodeDistLock:103 - Client client-1 acquired lock !!!
2014-06-01 15:17:16 <main> INFO  ZNodeDistLock:70 - Client client-1 processing [25]
2014-06-01 15:17:16 <main> INFO  ZNodeDistLock:133 - Client client-1 released lock !!!
2014-06-01 15:17:19 <main-EventThread> INFO  ZNodeDistLock:146 - -----> Notification for ZNode /DistLock <-----
2014-06-01 15:17:19 <main> INFO  ZNodeDistLock:103 - Client client-1 acquired lock !!!
2014-06-01 15:17:19 <main> INFO  ZNodeDistLock:70 - Client client-1 processing [27]
2014-06-01 15:17:19 <main> INFO  ZNodeDistLock:133 - Client client-1 released lock !!!
2014-06-01 15:17:22 <main-EventThread> INFO  ZNodeDistLock:146 - -----> Notification for ZNode /DistLock <-----
2014-06-01 15:17:22 <main> INFO  ZNodeDistLock:103 - Client client-1 acquired lock !!!
2014-06-01 15:17:22 <main> INFO  ZNodeDistLock:70 - Client client-1 processing [28]
2014-06-01 15:17:22 <main> INFO  ZNodeDistLock:133 - Client client-1 released lock !!!
2014-06-01 15:17:25 <main-EventThread> INFO  ZNodeDistLock:146 - -----> Notification for ZNode /DistLock <-----
2014-06-01 15:17:25 <main> INFO  ZNodeDistLock:103 - Client client-1 acquired lock !!!
2014-06-01 15:17:25 <main> INFO  ZNodeDistLock:70 - Client client-1 processing [29]
2014-06-01 15:17:26 <main> INFO  ZNodeDistLock:133 - Client client-1 released lock !!!
2014-06-01 15:17:29 <main-EventThread> INFO  ZNodeDistLock:146 - -----> Notification for ZNode /DistLock <-----
2014-06-01 15:17:29 <main> INFO  ZNodeDistLock:103 - Client client-1 acquired lock !!!
2014-06-01 15:17:29 <main> INFO  ZNodeDistLock:70 - Client client-1 processing [30]
2014-06-01 15:17:29 <main> INFO  ZNodeDistLock:133 - Client client-1 released lock !!!
2014-06-01 15:17:32 <main> INFO  ZooKeeper:684 - Session: 0x246577d3ec20000 closed
2014-06-01 15:17:32 <main-EventThread> INFO  ClientCnxn:512 - EventThread shut down

The following will be the output in Terminal-2:

Output.8

2014-06-01 15:16:58 <main> INFO  ZooKeeper:100 - Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
2014-06-01 15:16:58 <main> INFO  ZooKeeper:100 - Client environment:host.name=my-host
2014-06-01 15:16:58 <main> INFO  ZooKeeper:100 - Client environment:java.version=1.8.0_05
2014-06-01 15:16:58 <main> INFO  ZooKeeper:100 - Client environment:java.vendor=Oracle Corporation
2014-06-01 15:16:58 <main> INFO  ZooKeeper:100 - Client environment:java.home=/usr/lib/jvm/java-8-oracle/jre
2014-06-01 15:16:58 <main> INFO  ZooKeeper:100 - Client environment:java.class.path=/home/zkuser/zookeeper/resources:/home/zkuser/zookeeper/lib/jline-0.9.94.jar:/home/zkuser/zookeeper/lib/log4j-1.2.16.jar:/home/zkuser/zookeeper/lib/netty-3.7.0.Final.jar:/home/zkuser/zookeeper/lib/slf4j-api-1.6.1.jar:/home/zkuser/zookeeper/lib/slf4j-log4j12-1.6.1.jar:/home/zkuser/zookeeper/lib/zookeeper-3.4.6.jar:/home/zkuser/zookeeper/build/classes
2014-06-01 15:16:58 <main> INFO  ZooKeeper:100 - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2014-06-01 15:16:58 <main> INFO  ZooKeeper:100 - Client environment:java.io.tmpdir=/tmp
2014-06-01 15:16:58 <main> INFO  ZooKeeper:100 - Client environment:java.compiler=<NA>
2014-06-01 15:16:58 <main> INFO  ZooKeeper:100 - Client environment:os.name=Linux
2014-06-01 15:16:58 <main> INFO  ZooKeeper:100 - Client environment:os.arch=amd64
2014-06-01 15:16:58 <main> INFO  ZooKeeper:100 - Client environment:os.version=3.13.0-27-generic
2014-06-01 15:16:58 <main> INFO  ZooKeeper:100 - Client environment:user.name=zkuser
2014-06-01 15:16:58 <main> INFO  ZooKeeper:100 - Client environment:user.home=/home/zkuser
2014-06-01 15:16:58 <main> INFO  ZooKeeper:100 - Client environment:user.dir=/home/zkuser
2014-06-01 15:16:58 <main> INFO  ZooKeeper:438 - Initiating client connection, connectString=192.168.1.1:2181,192.168.1.2:2181,192.168.1.3:2181 sessionTimeout=10000 watcher=com.polarsparc.zookeeper.ZkHelper$$Lambda$1/758705033@4d76f3f8
2014-06-01 15:16:58 <main-SendThread(192.168.1.2:2181)> INFO  ClientCnxn:975 - Opening socket connection to server 192.168.1.2/192.168.1.2:2181. Will not attempt to authenticate using SASL (unknown error)
2014-06-01 15:16:58 <main-SendThread(192.168.1.2:2181)> INFO  ClientCnxn:852 - Socket connection established to 192.168.1.2/192.168.1.2:2181, initiating session
2014-06-01 15:16:58 <main-SendThread(192.168.1.2:2181)> INFO  ClientCnxn:1235 - Session establishment complete on server 192.168.1.2/192.168.1.2:2181, sessionid = 0x246577d3ec20001, negotiated timeout = 10000
2014-06-01 15:16:58 <main-EventThread> INFO  ZkHelper:45 - ZK[192.168.1.1:2181,192.168.1.2:2181,192.168.1.3:2181] -> Connected !!!
2014-06-01 15:16:58 <main> INFO  ZkHelper:165 - ZK[/DistLock] -> Exists, Stat: {ctime = Sun Jun 01 15:16:57 EDT 2014, mtime = Sun Jun 01 15:16:57 EDT 2014, version = 1}
2014-06-01 15:16:58 <main> INFO  ZNodeDistLock:56 - Check ZNode /DistLock, Status = true
2014-06-01 15:16:58 <main> INFO  ZkHelper:165 - ZK[/DistLock] -> Exists, Stat: {ctime = Sun Jun 01 15:16:57 EDT 2014, mtime = Sun Jun 01 15:16:57 EDT 2014, version = 1}
2014-06-01 15:16:58 <main-EventThread> INFO  ZNodeDistLock:146 - -----> Notification for ZNode /DistLock <-----
2014-06-01 15:16:58 <main> INFO  ZNodeDistLock:103 - Client client-2 acquired lock !!!
2014-06-01 15:16:58 <main> INFO  ZNodeDistLock:70 - Client client-2 processing [2]
2014-06-01 15:16:58 <main> INFO  ZNodeDistLock:133 - Client client-2 released lock !!!
2014-06-01 15:16:59 <main-EventThread> INFO  ZNodeDistLock:146 - -----> Notification for ZNode /DistLock <-----
2014-06-01 15:16:59 <main> INFO  ZNodeDistLock:103 - Client client-2 acquired lock !!!
2014-06-01 15:16:59 <main> INFO  ZNodeDistLock:70 - Client client-2 processing [4]
2014-06-01 15:16:59 <main> INFO  ZNodeDistLock:133 - Client client-2 released lock !!!
2014-06-01 15:17:00 <main-EventThread> INFO  ZNodeDistLock:146 - -----> Notification for ZNode /DistLock <-----
2014-06-01 15:17:00 <main> INFO  ZNodeDistLock:103 - Client client-2 acquired lock !!!
2014-06-01 15:17:00 <main> INFO  ZNodeDistLock:70 - Client client-2 processing [5]
2014-06-01 15:17:00 <main> INFO  ZNodeDistLock:133 - Client client-2 released lock !!!
2014-06-01 15:17:01 <main-EventThread> INFO  ZNodeDistLock:146 - -----> Notification for ZNode /DistLock <-----
2014-06-01 15:17:01 <main> INFO  ZNodeDistLock:103 - Client client-2 acquired lock !!!
2014-06-01 15:17:01 <main> INFO  ZNodeDistLock:70 - Client client-2 processing [8]
2014-06-01 15:17:01 <main> INFO  ZNodeDistLock:133 - Client client-2 released lock !!!
2014-06-01 15:17:02 <main-EventThread> INFO  ZNodeDistLock:146 - -----> Notification for ZNode /DistLock <-----
2014-06-01 15:17:02 <main> INFO  ZNodeDistLock:103 - Client client-2 acquired lock !!!
2014-06-01 15:17:02 <main> INFO  ZNodeDistLock:70 - Client client-2 processing [9]
2014-06-01 15:17:02 <main> INFO  ZNodeDistLock:133 - Client client-2 released lock !!!
2014-06-01 15:17:04 <main-EventThread> INFO  ZNodeDistLock:146 - -----> Notification for ZNode /DistLock <-----
2014-06-01 15:17:04 <main> INFO  ZNodeDistLock:103 - Client client-2 acquired lock !!!
2014-06-01 15:17:04 <main> INFO  ZNodeDistLock:70 - Client client-2 processing [11]
2014-06-01 15:17:04 <main> INFO  ZNodeDistLock:133 - Client client-2 released lock !!!
2014-06-01 15:17:05 <main-EventThread> INFO  ZNodeDistLock:146 - -----> Notification for ZNode /DistLock <-----
2014-06-01 15:17:05 <main> INFO  ZNodeDistLock:103 - Client client-2 acquired lock !!!
2014-06-01 15:17:05 <main> INFO  ZNodeDistLock:70 - Client client-2 processing [12]
2014-06-01 15:17:05 <main> INFO  ZNodeDistLock:133 - Client client-2 released lock !!!
2014-06-01 15:17:06 <main-EventThread> INFO  ZNodeDistLock:146 - -----> Notification for ZNode /DistLock <-----
2014-06-01 15:17:06 <main> INFO  ZNodeDistLock:103 - Client client-2 acquired lock !!!
2014-06-01 15:17:06 <main> INFO  ZNodeDistLock:70 - Client client-2 processing [14]
2014-06-01 15:17:06 <main> INFO  ZNodeDistLock:133 - Client client-2 released lock !!!
2014-06-01 15:17:07 <main-EventThread> INFO  ZNodeDistLock:146 - -----> Notification for ZNode /DistLock <-----
2014-06-01 15:17:07 <main> INFO  ZNodeDistLock:103 - Client client-2 acquired lock !!!
2014-06-01 15:17:07 <main> INFO  ZNodeDistLock:70 - Client client-2 processing [16]
2014-06-01 15:17:07 <main> INFO  ZNodeDistLock:133 - Client client-2 released lock !!!
2014-06-01 15:17:08 <main-EventThread> INFO  ZNodeDistLock:146 - -----> Notification for ZNode /DistLock <-----
2014-06-01 15:17:08 <main> INFO  ZNodeDistLock:103 - Client client-2 acquired lock !!!
2014-06-01 15:17:08 <main> INFO  ZNodeDistLock:70 - Client client-2 processing [18]
2014-06-01 15:17:08 <main> INFO  ZNodeDistLock:133 - Client client-2 released lock !!!
2014-06-01 15:17:09 <main-EventThread> INFO  ClientCnxn:512 - EventThread shut down
2014-06-01 15:17:09 <main> INFO  ZooKeeper:684 - Session: 0x246577d3ec20001 closed

The following will be the output in Terminal-3:

Output.9

2014-06-01 15:16:58 <main> INFO  ZooKeeper:100 - Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
2014-06-01 15:16:58 <main> INFO  ZooKeeper:100 - Client environment:host.name=my-host
2014-06-01 15:16:58 <main> INFO  ZooKeeper:100 - Client environment:java.version=1.8.0_05
2014-06-01 15:16:58 <main> INFO  ZooKeeper:100 - Client environment:java.vendor=Oracle Corporation
2014-06-01 15:16:58 <main> INFO  ZooKeeper:100 - Client environment:java.home=/usr/lib/jvm/java-8-oracle/jre
2014-06-01 15:16:58 <main> INFO  ZooKeeper:100 - Client environment:java.class.path=/home/zkuser/zookeeper/resources:/home/zkuser/zookeeper/lib/jline-0.9.94.jar:/home/zkuser/zookeeper/lib/log4j-1.2.16.jar:/home/zkuser/zookeeper/lib/netty-3.7.0.Final.jar:/home/zkuser/zookeeper/lib/slf4j-api-1.6.1.jar:/home/zkuser/zookeeper/lib/slf4j-log4j12-1.6.1.jar:/home/zkuser/zookeeper/lib/zookeeper-3.4.6.jar:/home/zkuser/zookeeper/build/classes
2014-06-01 15:16:58 <main> INFO  ZooKeeper:100 - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2014-06-01 15:16:58 <main> INFO  ZooKeeper:100 - Client environment:java.io.tmpdir=/tmp
2014-06-01 15:16:58 <main> INFO  ZooKeeper:100 - Client environment:java.compiler=<NA>
2014-06-01 15:16:58 <main> INFO  ZooKeeper:100 - Client environment:os.name=Linux
2014-06-01 15:16:58 <main> INFO  ZooKeeper:100 - Client environment:os.arch=amd64
2014-06-01 15:16:58 <main> INFO  ZooKeeper:100 - Client environment:os.version=3.13.0-27-generic
2014-06-01 15:16:58 <main> INFO  ZooKeeper:100 - Client environment:user.name=zkuser
2014-06-01 15:16:58 <main> INFO  ZooKeeper:100 - Client environment:user.home=/home/zkuser
2014-06-01 15:16:58 <main> INFO  ZooKeeper:100 - Client environment:user.dir=/home/zkuser
2014-06-01 15:16:58 <main> INFO  ZooKeeper:438 - Initiating client connection, connectString=192.168.1.1:2181,192.168.1.2:2181,192.168.1.3:2181 sessionTimeout=10000 watcher=com.polarsparc.zookeeper.ZkHelper$$Lambda$1/758705033@4d76f3f8
2014-06-01 15:16:58 <main-SendThread(192.168.1.3:2181)> INFO  ClientCnxn:975 - Opening socket connection to server 192.168.1.3/192.168.1.3:2181. Will not attempt to authenticate using SASL (unknown error)
2014-06-01 15:16:58 <main-SendThread(192.168.1.3:2181)> INFO  ClientCnxn:852 - Socket connection established to 192.168.1.3/192.168.1.3:2181, initiating session
2014-06-01 15:16:58 <main-SendThread(192.168.1.3:2181)> INFO  ClientCnxn:1235 - Session establishment complete on server 192.168.1.3/192.168.1.3:2181, sessionid = 0x346577d3eed0001, negotiated timeout = 10000
2014-06-01 15:16:58 <main-EventThread> INFO  ZkHelper:45 - ZK[192.168.1.1:2181,192.168.1.2:2181,192.168.1.3:2181] -> Connected !!!
2014-06-01 15:16:58 <main> INFO  ZkHelper:165 - ZK[/DistLock] -> Exists, Stat: {ctime = Sun Jun 01 15:16:57 EDT 2014, mtime = Sun Jun 01 15:16:58 EDT 2014, version = 2}
2014-06-01 15:16:58 <main> INFO  ZNodeDistLock:56 - Check ZNode /DistLock, Status = true
2014-06-01 15:16:58 <main> INFO  ZkHelper:165 - ZK[/DistLock] -> Exists, Stat: {ctime = Sun Jun 01 15:16:57 EDT 2014, mtime = Sun Jun 01 15:16:58 EDT 2014, version = 2}
2014-06-01 15:16:58 <main-EventThread> INFO  ZNodeDistLock:146 - -----> Notification for ZNode /DistLock <-----
2014-06-01 15:16:58 <main> INFO  ZNodeDistLock:103 - Client client-3 acquired lock !!!
2014-06-01 15:16:58 <main> INFO  ZNodeDistLock:70 - Client client-3 processing [3]
2014-06-01 15:16:59 <main> INFO  ZNodeDistLock:133 - Client client-3 released lock !!!
2014-06-01 15:17:01 <main-EventThread> INFO  ZNodeDistLock:146 - -----> Notification for ZNode /DistLock <-----
2014-06-01 15:17:01 <main> INFO  ZNodeDistLock:103 - Client client-3 acquired lock !!!
2014-06-01 15:17:01 <main> INFO  ZNodeDistLock:70 - Client client-3 processing [7]
2014-06-01 15:17:01 <main> INFO  ZNodeDistLock:133 - Client client-3 released lock !!!
2014-06-01 15:17:03 <main-EventThread> INFO  ZNodeDistLock:146 - -----> Notification for ZNode /DistLock <-----
2014-06-01 15:17:03 <main> INFO  ZNodeDistLock:103 - Client client-3 acquired lock !!!
2014-06-01 15:17:03 <main> INFO  ZNodeDistLock:70 - Client client-3 processing [10]
2014-06-01 15:17:03 <main> INFO  ZNodeDistLock:133 - Client client-3 released lock !!!
2014-06-01 15:17:05 <main-EventThread> INFO  ZNodeDistLock:146 - -----> Notification for ZNode /DistLock <-----
2014-06-01 15:17:05 <main> INFO  ZNodeDistLock:103 - Client client-3 acquired lock !!!
2014-06-01 15:17:05 <main> INFO  ZNodeDistLock:70 - Client client-3 processing [13]
2014-06-01 15:17:05 <main> INFO  ZNodeDistLock:133 - Client client-3 released lock !!!
2014-06-01 15:17:07 <main-EventThread> INFO  ZNodeDistLock:146 - -----> Notification for ZNode /DistLock <-----
2014-06-01 15:17:07 <main> INFO  ZNodeDistLock:103 - Client client-3 acquired lock !!!
2014-06-01 15:17:07 <main> INFO  ZNodeDistLock:70 - Client client-3 processing [17]
2014-06-01 15:17:07 <main> INFO  ZNodeDistLock:133 - Client client-3 released lock !!!
2014-06-01 15:17:09 <main-EventThread> INFO  ZNodeDistLock:146 - -----> Notification for ZNode /DistLock <-----
2014-06-01 15:17:09 <main> INFO  ZNodeDistLock:103 - Client client-3 acquired lock !!!
2014-06-01 15:17:09 <main> INFO  ZNodeDistLock:70 - Client client-3 processing [19]
2014-06-01 15:17:09 <main> INFO  ZNodeDistLock:133 - Client client-3 released lock !!!
2014-06-01 15:17:11 <main-EventThread> INFO  ZNodeDistLock:146 - -----> Notification for ZNode /DistLock <-----
2014-06-01 15:17:11 <main> INFO  ZNodeDistLock:103 - Client client-3 acquired lock !!!
2014-06-01 15:17:11 <main> INFO  ZNodeDistLock:70 - Client client-3 processing [21]
2014-06-01 15:17:12 <main> INFO  ZNodeDistLock:133 - Client client-3 released lock !!!
2014-06-01 15:17:14 <main-EventThread> INFO  ZNodeDistLock:146 - -----> Notification for ZNode /DistLock <-----
2014-06-01 15:17:14 <main> INFO  ZNodeDistLock:103 - Client client-3 acquired lock !!!
2014-06-01 15:17:14 <main> INFO  ZNodeDistLock:70 - Client client-3 processing [23]
2014-06-01 15:17:14 <main> INFO  ZNodeDistLock:133 - Client client-3 released lock !!!
2014-06-01 15:17:16 <main-EventThread> INFO  ZNodeDistLock:146 - -----> Notification for ZNode /DistLock <-----
2014-06-01 15:17:16 <main> INFO  ZNodeDistLock:103 - Client client-3 acquired lock !!!
2014-06-01 15:17:16 <main> INFO  ZNodeDistLock:70 - Client client-3 processing [24]
2014-06-01 15:17:16 <main> INFO  ZNodeDistLock:133 - Client client-3 released lock !!!
2014-06-01 15:17:18 <main-EventThread> INFO  ZNodeDistLock:146 - -----> Notification for ZNode /DistLock <-----
2014-06-01 15:17:18 <main> INFO  ZNodeDistLock:103 - Client client-3 acquired lock !!!
2014-06-01 15:17:18 <main> INFO  ZNodeDistLock:70 - Client client-3 processing [26]
2014-06-01 15:17:18 <main> INFO  ZNodeDistLock:133 - Client client-3 released lock !!!
2014-06-01 15:17:20 <main> INFO  ZooKeeper:684 - Session: 0x346577d3eed0001 closed
2014-06-01 15:17:20 <main-EventThread> INFO  ClientCnxn:512 - EventThread shut down

As can be inferred from the above Output.7, Output.8, and Output.9, the counter value is icrementing sequentially as expected without any duplicates or misses.

References

Exploring Apache ZooKeeper :: Part-1

Exploring Apache ZooKeeper :: Part-2