Grok template for different types of log in log file
I am trying to write a grok template for my log file that has three different types of logs, I want to put a filter in the type names (TYPE1, TYPE2, TYPE3) and then write three different grok templates for that one log file. Also, my log file is a split csv file.
Log file:
TYPE1, word, word, word, Num
TYPE2, word, word, word, word
TYPE3, num, word, num, word
Here's what I've tried so far:
filter {
if [message] =~ /TYPE1/ {
grok {
match => [ "message", "%{WORD:type},%{WORD:a1"},%{WORD:a2"},%{WORD:a3"},%{POSINT:a4"}]
}
}
}
This does not work. Also, in this config file, I wrote grok templates for other files (which work well), for example:
filter {
if [type] == "sometype1" or [type] == "sometype2" {
grok {
match => [ "message", "%{POSINT:moduleid}%{SPACE}%{NUMBER:date}"]
}
}
}
And the log file that is giving me the problem is of type = sometype3, which I haven't mentioned anywhere.
thank
source to share
I guess you don't need to conditionally do this. If you have static TYPE values ("TYPE1", "TYPE2" or "TYPE3"), then why not specify one grok template for each TYPE:
filter {
grok {
match => { "message" => [
"TYPE1,%{WORD:a1},%{WORD:a2},%{WORD:a3},%{POSINT:a4}",
"TYPE2,%{WORD:b1},%{WORD:b2},%{WORD:b3},%{WORD:b4}",
"TYPE3,%{POSINT:c1},%{WORD:c2},%{POSINT:c3},%{WORD:c4}" ]
}
}
}
I tried and it works for your data formats:
TYPE1,word,word,word,num TYPE2,word,word,word,word TYPE3,num,word,num,word
The log file will look like this:
TYPE1,a,b,c,4 TYPE2,a,b,c,d TYPE3,1,b,3,d
source to share
start with a successful match of one type, for example:
filter {
if [type] == "sometype1" {
grok {
match => [ "message", "%{WORD:type",%{WORD:abc"}]
}
}
}
If that doesn't work, you either don't have a type field with a matching value in your log data, or your grok template is wrong.
Test it using the grok debugger
If you were able to parse one type, try adding other types by adding
if [type] == "sometype1" or [type] == "sometype2" or [type] == "sometype3"
an alternative for this could be
if [type] == "sometype1" {
}
else if [type] == "sometype2" {
}
source to share
In your example, you are using regex to see if the regex should run. This is too much overhead.
Here are two ideas:
First use grok to wrap the first word into a variable and return the rest of the information in the message:
"%{WORD:myType},%{GREEDYDATA:message}"
(you need to use overwrite
in this configuration).
Then you can use exact conditions to determine which subsequent grok pattern to use:
if [myType] == "type1" {
}
...
Second, it's also possible to list multiple patterns in a single grok statement:
match => [ "message", "pattern1", "pattern2", "pattern3" ]
But that's expensive too! (make sure the syntax is against the document to be sure!).
source to share