Multiple levels of multiline in logstash

Question

Multiple levels of multiline in logstash

I want to use a multi-line filter and then another multi-line at a deeper level.

To be more precise, I want to have a stack trace of java exceptions, for example:

2014-06-20 Some-arbitrary-log
java.lang.IndexOutOfBoundsException: Index: 8, Size: 1
    at java.util.ArrayList.rangeCheck(ArrayList.java:604)
    at java.util.ArrayList.get(ArrayList.java:382)

And then, after that, combine a couple of them together like this using another multi-line and throttle:

2014-06-19 Some-arbitrary-log
java.lang.IndexOutOfBoundsException: Index: 2, Size: 1
   at java.util.ArrayList.rangeCheck(ArrayList.java:604)
   at java.util.ArrayList.get(ArrayList.java:382)
2014-06-20 Some-arbitrary-log
java.lang.IndexOutOfBoundsException: Index: 8, Size: 1
    at java.util.ArrayList.rangeCheck(ArrayList.java:604)
    at java.util.ArrayList.get(ArrayList.java:382)

My filter looks like this:

filter {
    if [type] =~ /test.+
    {
        multiline {
            pattern => "(^.+Exception.*)|(^\tat.+)"
            negate => false
            what => "previous"
        }
        if ("multiline" in [tags]) {
            mutate {
                add_field => [ "ERROR_TYPE", "java_exception" ]
            }
        }

        if ([ERROR_TYPE] == "java_exception") {
            throttle{
                key => ".*"
                period => 10
                before_count => 2
                after_count => -1
                add_tag => "throttled"
            }

            if ("throttled" not in [tags]) {
                multiline {
                    pattern => ".*"
                    negate => false
                    what => "previous"
                }
            }
        }

    }
}

The first level is only one stack trace. As is the case, this works as intended:

    multiline {
        pattern => "(^.+Exception.*)|(^\tat.+)"
        negate => false
        what => "previous"
    }
    if ("multiline" in [tags]) {
        mutate {
            add_field => [ "ERROR_TYPE", "java_exception" ]
        }
    }

However, concatenating multiple stacks of stack does not work. The output I'm using is this:

output {
    if [ERROR_TYPE] == "java_exception"{
    stdout {codec => rubydebug }
    elasticsearch {
        cluster => "logstash"
    }
    }
}

However, there are no combined stack traces. And they all have the tags "throttled".

To check if there are any chokes I did:

output {
    if [ERROR_TYPE] == "java_exception" and "throttled" not in [tags]{
    stdout {codec => rubydebug }
    elasticsearch {
        cluster => "logstash"
    }
    }
}