Clever Geek Handbook

Failed to list files in hadoop

I am new to hadoop.

I am trying to list all the files I have in my hdfs with this recursive code.

public void list(String path) throws IOException {
    Path dirPath = new Path(path);
    FileStatus[] fileStatus = fileSystem.listStatus(dirPath);
    if(fileStatus != null) {
        for (FileStatus fs : fileStatus) {
            String name = fs.getPath().getName();
            if(fs.isDir()) {
                System.out.println("dir --> " + name);
                list(dirPath.getName() + "/" + name);
            } else {
                System.out.println("file --> " + name);
            }
        }
    }
}

However, I am not getting the list of files.

Thanks in advance.

+3


source to share


2 answers


As stated, you must initialize your filesystem object correctly. It should be something like:

Configuration conf = new Configuration();
FileSystem fs = FileSystem.get(URI.create(uri), conf);


And uri

must be prefixed hdfs:///

if you are trying to get data from HDFS.

+2


source


Try the following:

`



FileSystem fs = FileSystem.get(new Configuration());
RemoteIterator<LocatedFileStatus> rmIterator = fs.listLocatedStatus(new Path(srcPath));
while (rmIterator.hasNext()) {
Path path = rmIterator.next().getPath();
if(fs.isDirectory(path)){
System.out.println("Directory Name: "+path.getName());
}else if(fs.isFile(path)){
System.out.println("File Name: "+path.getName());
}

`



+1


source






More articles:


All Articles