Installing hadoop.tmp.dir on Windows gives error: URI has permissions component
I am trying to specify the base directory of HDFS files in my hdfs-site.xml
under Windows 7 (Hadoop 2.7.1 which I built from source using Java SDK 1.8.0_45 and Windows SDK 7.1). I cannot figure out how to specify the path that defines the drive.
Mine hdfs-site.xml
looks like this:
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>XXX</value>
</property>
</configuration>
and I tried various values for XXX
which I tested with hdfs namenode -format
, which resulted in one of these two errors:
-
XXX=D:/tmp/hdp
:15/07/10 23:38:33 ERROR namenode.NameNode: Failed to start namenode. java.lang.IllegalArgumentException: URI has an authority component at java.io.File.<init>(File.java:423) at org.apache.hadoop.hdfs.server.namenode.NNStorage.getStorageDirectory(NNStorage.java:329)
-
XXX=D:\tmp\hdp
:ERROR common.Util: Syntax error in URI file://D:\tmp\hdp/dfs/name
Other options that gave similar errors: file:///D:/tmp/hdp
(from http://hortonworks.com/community/forums/topic/hadoop-configuration-files-issues/ ) file://D:/tmp/hdp
,D:\\tmp\\hdp
And if I use /D/tmp/hdp
it does not crash but goes to a folder D
on my current drive.
I have no ideas, suggestions? (NB: also, using Cygwin, this is not an option for me)
source to share
You can specify the drive spec in the hadoop.tmp.dir
core-site.xml file by adding '/' in front of the absolute path and using '/' as path separator instead of '\' for all path elements, For example, if the desired absolute path is D: \ tmp \ hdp, then it will look like this:
<property>
<name>hadoop.tmp.dir</name>
<value>/D:/tmp/hdp</value>
</property>
This is because the defaults for many HDFS directories are configured as file://${hadoop.tmp.dir}/suffix
. See the default definitions dfs.namenode.name.dir
, dfs.datanode.data.dir
and dfs.namenode.checkpoint.dir
here:
http://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml
Substituting the above value for for hadoop.tmp.dir
yields a valid file:
disk spec and unauthorized URI that satisfies the HDFS configuration requirements. It is important to use '/' instead of '\' because the bare, unencoded '\' character is not valid in URL syntax.
http://www.ietf.org/rfc/rfc1738.txt
If you prefer not to rely on this substitution behavior, it is also acceptable to override any configuration properties that you use hadoop.tmp.dir
in your hdfs-site.xml file. Each value must be a fully file:
qualified URI. For example:
<property>
<name>dfs.namenode.name.dir</name>
<value>file:///D:/tmp/hdp/dfs/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:///D:/tmp/hdp/dfs/data</value>
</property>
<property>
<name>dfs.namenode.checkpoint.dir</name>
<value>file:///D:/tmp/hdp/dfs/namesecondary</value>
</property>
You may find this more readable overall.
source to share