Bandwidth issue
I am doing distcp from one haop cluster (version 0.20.2) to another hasoop cluster (version 2.2.0) using below command.
hadoop distcp -update -skipcrccheck "hftp://x.x.x.x:50070//hive/warehouse//staging_eventlog_arpu_comma" "hdfs://y.y.y.y:9000//hive/warehouse/staging_eventlog_arpu_comma"
therefore, bandwidth usage must be the source for the destination. but network utilization is greater from destination to source compared to original destination.
hasoop distcp -bandwidth indicates:
**Each map will be restricted to consume only the specified bandwidth. This is not always exact. The map throttles back its bandwidth consumption during a copy, such that the net bandwidth used tends towards the specified value.**
So what is it throttling back.?
+3
source to share
No one has answered this question yet
Check out similar questions: