Minor "template doesn't match" error on fluentd
Can anyone tell if its ok that fluentd will raise this error in the td-agent.log file ?
2015-07-31 13:15:19 +0000 [warn]: pattern not match: "- - - [31/Jul/2015:13:15:19 +0000] GET http://172.31.108.218/ HTTP/1.1 200 0 \"-\" \"ELB-HealthChecker/1.0\""
Although this is a well-formed apache2 log:
- - - [31/Jul/2015:13:15:19 +0000] GET http://172.31.108.218/ HTTP/1.1 200 0 \"-\" \"ELB-HealthChecker/1.0\"
And here is the original configuration:
<source>
type tail
format apache2
path /var/log/varnish/varnishncsa.log
pos_file /var/log/td-agent/tmp/access.log.pos
tag "apache2.varnish-access"
</source>
I cannot figure out what is wrong there.
source to share
Instead of finding a way to filter out the logs from ELB-HealthChecker, you can set your own format
for Apache access log, which is a little more flexible in terms of the first pair fields. I ran into this same error while getting / checking server status from collectd (using it to monitor SignalFx).
Source setup as follows:
<source>
type tail
format /^(?<host>[^ ]*(?:\s+[^ ]+)*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$/
time_format %d/%b/%Y:%H:%M:%S %z
path /var/log/apache2/access.log
pos_file /var/log/td-agent/apache2.pos
tag apache2.log
</source>
Allows both log lines:
172.18.0.2:80 127.0.0.1 - - [08/Aug/2017:19:58:38 +0000] "GET /server-status?auto HTTP/1.1" 200 508 "-" "collectd/5.7.2.sfx0"
As well as:
192.168.0.1 - - [28/Feb/2013:12:00:00 +0900] "GET / HTTP/1.1" 200 777 "-" "Opera/12.0"
You can check for format
regex matching with Fluentular .
See related: Fluentd Log Format with Multiple Host IPs
source to share