Log4j registers directly with elasticsearch server
I am a little confused as to how I can put my log entries directly to elasticsearch
(not logstash). So far I've found a few additions ( log4j.appender.SocketAppender
, log4j.appender.server
etc.) that allow logs to be sent to a remote host, as well as a ConversionPattern
feature that seems to allow us to convert logs to "elastic" format, but this approach looks strange .. . or I'm wrong? Is this the only way to send logs to elastic
?
So far I have a configuration like this:
log4j.rootLogger=DEBUG, server
log4j.appender.server=org.apache.log4j.net.SocketAppender
log4j.appender.server.Port=9200
log4j.appender.server.RemoteHost=localhost
log4j.appender.server.ReconnectionDelay=10000
log4j.appender.server.layout.ConversionPattern={"debug_level":"%p","debug_timestamp":"%d{ISO8601}","debug_thread":"%t","debug_file":"%F", "debug_line":"%L","debug_message":"%m"}%n
But I am getting the error:
log4j:WARN Detected problem with connection: java.net.SocketException: Broken pipe (Write failed)
I cannot find a useful example, so I cannot figure out what I am doing wrong and how to fix it. Thank.
source to share
I found the solution that best suits my requirements. It is graylog . Since it's built on top elasticsearch
, the usage is familiar, so I was able to jump straight to it.
To use it, I added this dependency along with the basic log4j2 dependencies:
<dependency>
<groupId>org.graylog2.log4j2</groupId>
<artifactId>log4j2-gelf</artifactId>
<version>1.3.2</version>
</dependency>
and use the config log4j2.json
:
{
"configuration": {
"status": "info",
"name": "LOGGER",
"packages": "org.graylog2.log4j2",
"appenders": {
"GELF": {
"name": "GELF",
"server": "log.myapp.com",
"port": "12201",
"hostName": "my-awsome-app",
"JSONLayout": {
"compact": "false",
"locationInfo": "true",
"complete": "true",
"eventEol": "true",
"properties": "true",
"propertiesAsList": "true"
},
"ThresholdFilter": {
"level": "info"
}
}
},
"loggers": {
"logger": [
{
"name": "io.netty",
"level": "info",
"additivity": "false",
"AppenderRef": {
"ref": "GELF"
}
}
],
"root": {
"level": "info",
"AppenderRef": [
{
"ref": "GELF"
}
]
}
}
}
}
source to share
I wrote this appender here Log4J2 Elastic REST Appender if you want to use it. It has the ability to buffer log events based on time and / or number of events before sending it to Elastic (using the _bulk API so it dispatches everything in one go). It was published on Maven Central, so it is pretty straight forward.
source to share