Using java to read files in HDFS and match multi-line blocks with regex

NB EDITED after clarification in comments

You might be able to find multi-line blocks using Regex - you can of course write a Regex that will match them for example. .*\[MAP\]((?s).*)\[\\MAP\]

- noting that in Java you will also have to escape all characters \

, while (?s)

letting the character .

match newlines, i.e.

String mapBlockRegex = ".*\\[MAP\\]((?s).*)\\[\\\\MAP\\]";`  

      

        
        
        
      

    

However - as you pointed out - this creates difficulties if the file does not fit in memory, and splitting also has some difficulties.

I suggest another idea - scan the file line by line and use a state variable to indicate if you are in a block or not. The main algorithm is as follows

• When you match the start of a block, set the state variable to true.
• While the state is true, add text to StringBuilder
  
  
• When you agree with block ending, set the state variable to false and use the String
  
  one you created for example. output it to a file, to the console, or use it programmatically.

Java solution

I propose one way to implement the above - using Scanner

- that traverses the stream line by line, discarding how it happens, while avoiding OutOfMemoryError

. Please note that this code can throw exceptions - I threw them, but you can put them in a block try..catch..finally

. Also note that Scanner

swallows the IO exception, but as the docs say , if it's important to you:

The most recent IOException thrown by the underlying readable can be obtained with the ioException () method.

import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.util.Scanner;


public class LogScanner
{

    public static void main(String[] args) throws FileNotFoundException
    {
        FileInputStream inputStream = null;
        Scanner sc = null;

        String path = "D:\\hadoopTest.log";
        String blockStart= ".*\\[MAP\\].*";
        String blockEnd = ".*\\[\\\\MAP\\].*";
        boolean inBlock = false;
        StringBuilder block = null;

        inputStream = new FileInputStream(path);
        sc = new Scanner(inputStream, "UTF-8");
        while (sc.hasNextLine()) {
            String line = sc.nextLine();
            if (line.matches(blockStart)) {
                inBlock = true;
                block = new StringBuilder();
            }

            if (inBlock) {
                block.append(line);
                block.append("\n");
            }

            if (line.matches(blockEnd)) {
                inBlock = false;
                String completeBlock = block.toString();
                System.out.println(completeBlock);
                // I'm outputting the blockto stdout, you could append to a file\whatever.
            }
        }

        sc.close();
    }
}

      

        
        
        
      

    

Caveat There may be characteristics in your file where this will not work without any changes. If you can have nested blocks [map]

, then there inBlock

must be an int where you increment, if you agree with the start and decrement of the block, if you match the end - add for any, inblock > 0

and only output the full string when inBlock

it vanishes.

Splitting command line when searching for matches on the same line

If you've searched on a string and the matches were guaranteed to be on the same line, then the split will be OK as long as the split occurs only at the end of complete lines.

In this case, you can use the command line to split the file. If you are on Linux (or any * nix I think) you can use the split command, eg.

split --lines=75000

      

        
        
        
      

    

This discusses the question and answer in more detail

There is no equivalent command on Windows that I know of, but you can install things that will work similarly - eg. GNU CoreUtils for Windows or 7-Zip. Caveat: I've never used them to separate.