Clever Geek Handbook

Datanode decommisioning vs Write

What happens when we log out of the datanode when writing happens to HDFS on this node?

Will it stop writing data to HDFS on that node and disconnect that node, or will it finish writing and then disconnect it.

+3


source to share


3 answers


I found a solution for this in cloudera.

Decommissioning applies only to HDFS DataNode, MapReduce TaskTracker, YARN NodeManager, and HBase RegionServer objects. If the host has other roles executed on it, these roles are stopped / killed.



http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cm_mc_decomm_host.html

+1


source


As stated in the Hadoop wiki :

Hadoop offers a deactivation function to exit a set of existing data nodes. The nodes to be removed must be included in the exclusion file, and the name of the exclusion file must be specified as the dfs.hosts.exclude configuration parameter. This file should have been specified when namenode was started.



The exclude file property contains a file containing a list of hosts that are not allowed to connect to namenode.

When this happens I think I IOException

got it while processing the process before the decommissioned node.

0


source


After adding entries for the nodes you want to decommission. You need to run the namenode -refreshnodes command to start the actual decommissioning process. The actual decompression process is very slow. All current tasks, both reading and writing, will be performed normally. But all future writes are not allowed, but reads are allowed with the lowest priority. Meanwhile, all blocks are replicated to other nodes.

0


source






More articles:


All Articles