Getting RESTful GET parameters in logstash
I am trying to get logstash to parse key-value pairs in an HTTP request request from my ELB log files.
the request field looks like this:
http://aaa.bbb/get?a=1&b=2
I would like the log line to have a field for a
and b
, and I am having a hard time understanding it.
My logstash conf (formatted for clarity) is below which doesn't load any extra key fields. I am guessing that I need to strip the address portion of the URI, but didn't get it.
input {
file {
path => "/home/ubuntu/logs/**/*.log"
type => "elb"
start_position => "beginning"
sincedb_path => "log_sincedb"
}
}
filter {
if [type] == "elb" {
grok {
match => [ "message", "%{TIMESTAMP_ISO8601:timestamp}
%{NOTSPACE:loadbalancer} %{IP:client_ip}:%{NUMBER:client_port:int}
%{IP:backend_ip}:%{NUMBER:backend_port:int}
%{NUMBER:request_processing_time:float}
%{NUMBER:backend_processing_time:float}
%{NUMBER:response_processing_time:float}
%{NUMBER:elb_status_code:int}
%{NUMBER:backend_status_code:int}
%{NUMBER:received_bytes:int} %{NUMBER:sent_bytes:int}
%{QS:request}" ]
}
date {
match => [ "timestamp", "ISO8601" ]
}
kv {
field_split => "&?"
source => "request"
exclude_keys => ["callback"]
}
}
}
output {
elasticsearch { host => localhost }
}
source to share
kv will take the url and strip the parameters. This configuration works:
input {
stdin { }
}
filter {
mutate {
add_field => { "request" => "http://aaa.bbb/get?a=1&b=2" }
}
kv {
field_split => "&?"
source => "request"
}
}
output {
stdout {
codec => rubydebug
}
}
stdout shows:
{
"request" => "http://aaa.bbb/get?a=1&b=2",
"a" => "1",
"b" => "2"
}
However, I would suggest that you create your own versions of the default URI templates to specify fields. Then you can pass the querystring field to kv. It's so clean.
UPDATE:
To "create my own templates" I wanted to use the existing ones and modify them as needed. In logstash 1.4, installing them was as easy as putting them in a new file in the "patterns" directory; I don't know about templates for> 1.4 yet.
MY_URIPATHPARAM %{URIPATH}(?:%{URIPARAM:myuriparams})?
MY_URI %{URIPROTO}://(?:%{USER}(?::[^@]*)?@)?(?:%{URIHOST})?(?:%{MY_URIPATHPARAM})?
Then you can use MY_URI in your grok {} template and create a myuriparams field that you could pass to kv {}.
source to share