Can I specify two different codecs in the lumberjack?
I just installed the ELK stack, but I am having problems with the logstash configuration in /etc/logstash/conf.d. I have two sources of input that are forwarded from the same linux server that has a logsto forwarder installed with "files" similar to:
{
"paths": ["/var/log/syslog","/var/log/auth.log"],
"fields": { "type": "syslog" }
},
{
"paths": ["/var/log/osquery/osqueryd.results.log"],
"fields": { "type": "osquery_json" }
}
As you can see, one input is osquery (json formatted) output and the other is syslog. My current configuration for logstash is osquery.conf:
input {
lumberjack {
port => 5003
ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
codec => "json"
}
}
filter {
if [type] == "osquery_json" {
date {
match => [ "unixTime", "UNIX" ]
}
}
}
output {
elasticsearch { host => localhost }
stdout { codec => rubydebug }
}
Which works great for one input source, but I don't know how to add another syslog input source in the same configuration as the "codec" field in the input file - I can't change it to syslog ...
I am also planning to add another input source to the Windows log format that is not forwarded by log forwarding. Is there a way to structure this differently?
source to share
It's probably better to just remove the codec from your input if you're going to handle different codecs on the same input:
input {
lumberjack {
port => 5003
ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
}
}
filter {
if [type] == "osquery_json" {
json {
source => "field_name_the_json_encoded_data_is_stored_in"
}
date {
match => [ "unixTime", "UNIX" ]
}
}
if [type] == "syslog" {
}
}
output {
elasticsearch { host => localhost }
stdout { codec => rubydebug }
}
Then you just need to decide what you want to do with the syslog messages.
I would suggest splitting your config into multiple files as well. I try to use 01-filename.conf - 10-filename.conf for inputs, 11-29 as filters and something higher than outputs. These files will be loaded into logstash in the order in which they are printed in ls.
source to share