Zookeeper ensemble doesn't fit
I am trying to set up an ensemble of 3 nodes following the documentation . They are all on Linux Ubuntu. on all three node config files looks like this:
zoo.cfg under $ ZOOKEEPER_HOME / conf
tickTime=2000
dataDir=/home/zkuser/zookeeper_data
clientPort=2181
initLimit=5
syncLimit=2
server.1=ip.of.zk1:2888:3888
server.2=ip.of.zk2:2888:3888
server.3=ip.of.zk3:2888:3888
I have also placed the relevant files myid
in the directory /home/zkuser/zookeeper_data/
. These myid files contain 1, which is on node (ip.of.zk1), so on and so forth.
When I start the zk server by running bin / zkServer.sh without showing any exceptions in the console. However, when I open the zookeeper.out files in the bin directory, I see the following errors.
2014-11-04 00:23:49,120 [myid:3] - WARN [WorkerSender[myid=3]:QuorumCnxManager@382] - Cannot open channel to 1 at election address /ip.of.zk1:3888
java.net.NoRouteToHostException: No route to host
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:327)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:193)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:180)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:385)
at java.net.Socket.connect(Socket.java:546)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:368)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:341)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:449)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:430)
at java.lang.Thread.run(Thread.java:701)
2014-11-04 00:23:49,123 [myid:3] - WARN [WorkerSender[myid=3]:QuorumCnxManager@382] - Cannot open channel to 2 at election address /ip.of.zk2:3888
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:327)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:193)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:180)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:385)
at java.net.Socket.connect(Socket.java:546)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:368)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:341)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:449)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:430)
at java.lang.Thread.run(Thread.java:701)
Note . I have opened the appropriate ports using iptables on each machine. For example: INPUT chain (ACCEPT policy)
target prot opt source destination
ACCEPT all -- IP.of.ZK1 anywhere
ACCEPT all -- IP.of.ZK2 anywhere
ACCEPT all -- IP.of.ZK3 anywhere
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
Can anyone tell me what I am missing?
Regards, JE
source to share
Make sure that:
- you have started the Zookeeper server on all 3 servers.
- all servers are running in error-free mode by starting
echo ruok | netcat ip.of.zk2 2181
. If everything is ok, the server should respondimok
(FYI, here is a list of all supported 4-letter commands ) -
/home/zkuser/zookeeper_data/myid
contains values 1/2/3 for each server, respectively - you can ping other 2 servers from the first server
If interested, I created a firewall + ability script to create a virtual Zookeeper 3 node cluster, see https://github.com/mkrcah/virtual-zookeeper-cluster
source to share
I had a similar problem. I have some hints that the problem might be from here and here . In my case, the command output was netstat -plutn
showing something including 127.0.0.1:3888
for selective port 3888. I solved the problem by changing the zoo.cfg part on server n, from something like
server.1=name.of.s1:2888.3888
...
server.n=localhost:2888:3888
...
to
server.1=name.of.s1:2888.3888
...
server.n=0.0.0.0:2888:3888
...
After restarting ZooKeeper, the output netstat -plutn
turns on :::3888
.
Apparently this is necessary for the ZooKeeper to display the election port correctly, in this case 3888.
source to share