HDFS CDH5 balancing error

'hdfs balancer' is running, I can see it shows the rack topology installed, overused and underutilized nodes, it says "5TB needs to be moved to balance the cluster" and "Decided to move 10GB bytes from 10.150.11.24: 50010 - 10.150.11.164:50010 "

then i see this error,

WARN balancer.Balancer: Dispatcher thread failed java.lang.NullPointerException when org.apache.hadoop.hdfs.server.balancer.Balancer.isGoodBlockCandidate (Balancer.java:1233) in org.apache.hadoop.hdfs.server.balancer. Balancer.access $ 400 (Balancer.java:183) at org.apache.hadoop.hdfs.server.balancer.Balancer $ Source.isGoodBlockCandidate (Balancer.java:686) at org.apache.hadoop.hdfs.server.balancer.Balancer $ Source.getBlockList (Balancer.java:674) at org.apache.hadoop.hdfs.server.balancer.Balancer $ Source.dispatchBlocks (Balancer.java:776) at org.apache.hadoop.hdfs.server.balancer.Balancer $ Source.access $ 1600 (Balancer.java:607) at org.apache.hadoop.hdfs.server.balancer.Balancer $ Source $ BlockMoveDispatcher.run (Balancer.java:614) at java.util.concurrent.Executors $ RunnableAdapter. call (Executors.java:471) in java.util.concurrent.FutureTask.run (FutureTask.java:262) in java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1145) in java.util.concurrent.ThreadPoolExecutor $ Worker.run (ThreadPoolExecutor.java:615) in java.lang.Thread .run (Thread.java:744)

that is,

No block was moved for 5 iterations. Exit from ...

WARN hdfs.DFSClient: DataStreamer Exception org.apache.hadoop.ipc.RemoteException (org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException): No permission on /system/balancer.id: File does not exist. holder DFSClient_NONMAPREDUCE_-201468433_1 has no open files. at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease (FSNamesystem.java:2937) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.analyzeFileState (FSNamesystem.java:2757) at org.apache .hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock (FSNamesystem.java:2665) in org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock (NameNodeRpcServer.java:569) in org.apdf.hado .protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock (ClientNamenodeProtocolServerSideTranslatorPB.java:440) in org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos $ ClientNamenodeProtocol $ 2.callBlockingMethod (ClientNamenodeProtocolProtos.java) in org.apache.hadoop.ipc.ProtobufRpcEngine $ Server $ ProtoBufRpcInvoker.call (ProtobufRpcEngine.java:585)

and the namenode log shows this,

WARN org.apache.hadoop.security.UserGroup Information: PriviledgedActionException as: hdfs (auth: SIMPLE) Reason: org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on / system / balancer.id: File does not exist. holder DFSClient_NONMAPREDUCE_-201468433_1 has no open files.

INFO org.apache.hadoop.ipc.Server: IPC server handler 118 on 8020, call org.apache.hadoop.hdfs.protocol.ClientProtocol.addBlock from 10.150.11.126:56490 Call # 71 Retry # 0: error: org.apache .hadoop.hdfs.server.namenode.LeaseExpiredException: no lease on / system / balancer.id: File does not exist. holder DFSClient_NONMAPREDUCE_-201468433_1 has no open files.

I searched and found some discussion regarding duplicate balancing processes, but I could not find this issue in our cluster. have any other ideas? we are using cdh5.0.1

+3


source to share


1 answer


In CDH, you must have a node that is running HDFS balancing service. This prevents balancing hdfs from the command line.

See here:



http://community.cloudera.com/t5/Storage-Random-Access-HDFS/Running-HDFS-Balancer-fail-report-namenode-LeaseExpiredException/td-p/14052

0


source







All Articles