Syslog forwared HAProxy protocol filtering in Logstash
I am having trouble figuring out how to do this correctly.
I have the following Logstash configuration:
input {
lumberjack {
port => 5000
host => "127.0.0.1"
ssl_certificate => "/etc/ssl/star_server_com.crt"
ssl_key => "/etc/ssl/server.key"
type => "somelogs"
}
}
output {
elasticsearch {
protocol => "http"
host => "es01.server.com"
}
}
With logstash-forwarder I push my haproxy.log file generated by syslog to logstash. Kibana then shows me _source
which looks like this:
{"message":"Dec 8 11:32:20 localhost haproxy[5543]: 217.116.219.53:47746 [08/Dec/2014:11:32:20.938] es_proxy es_proxy/es02.server.com 0/0/1/18/20 200 305 - - ---- 1/1/1/0/0 0/0 \"GET /_cluster/health HTTP/1.1\"","@version":"1","@timestamp":"2014-12-08T11:32:21.603Z","type":"syslog","file":"/var/log/haproxy.log","host":"haproxy.server.com","offset":"4728006"}
Now this needs to be filtered out (somehow) and I have to admit that I have no idea how.
Looking at the grok documentation and playing with the grok debugger I still haven't gotten anything useful from Logstash and Kibana.
I've been going through the templates directory and their files and I can't say I understand how to use them. I was hoping that providing a filter with a haproxy Logstash pattern would match the pattern from mine _source
, but that was no luck.
source to share
You're in luck as there is already a predefined grok pattern that seems to parse this exact type of log. All you have to do is reference it in the grok filter :
filter {
grok {
match => ["message", "%{HAPROXYHTTP}"]
}
}
%{HAPROXYHTTP}
will recursively expand according to the template definition , and each interesting chunk in each input line will be extracted into its own field, you can also remove the "message" field after successfully applying the grok filter, since it contains redundant data anyway; just add remove_field => ["message"]
grok to your filter declaration.
source to share