Java app in Docker container not logging to syslog correctly
My goal
I have several different containers running inside the host. They all share the volume with each other socket /dev/log
. The host forwards these logs to a central log server. All other container logs are displayed on the host /var/log/messages
. Other containers are python-based programs that allow logging directly to be added to a /dev/log
unix domain socket .
Basic configuration
I have a docker container running a scala app from a jar file. Sockets are /dev/log
shared as volumes between host and container. The file log4j.properties
for the application seems good to me and it installs like this:
# Root logger option
log4j.rootLogger=INFO, file, stdout, SYSLOG
# Direct log messages to a log file
log4j.appender.file=org.apache.log4j.RollingFileAppender
log4j.appender.file.File=log/associationRules.log
log4j.appender.file.MaxFileSize=10MB
log4j.appender.file.MaxBackupIndex=10
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} [%c{1}:%L] %-5p <%X{jobID}> %m%n
# Direct log messages to stdout
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.Target=System.out
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} [%c{1}:%L] %-5p <%X{jobID}> %m%n
# Log to syslog
log4j.appender.SYSLOG=org.apache.log4j.net.SyslogAppender
log4j.appender.SYSLOG.syslogHost=localhost
log4j.appender.SYSLOG.layout=org.apache.log4j.PatternLayout
log4j.appender.SYSLOG.layout.conversionPattern=%d{yyyy-MM-dd HH:mm:ss} [%c{1}:%L] %-5p <%X{jobID}> %m%n
log4j.appender.SYSLOG.Facility=LOCAL0
log4j.appender.SYSLOG.Threshold=debug
log4j.appender.SYSLOG.FacilityPrinting=true
The stdout and file logs are correct, but SYSLOG is not working correctly. The logs are displayed in the container /var/log/syslog
, but they are not displayed on the host /var/log/messages
. I thought SyslogAppender also outputs to /dev/log
, but it looks like it doesn't.
Additional Information
Here is the result I get in the container /var/log/syslog
. I see some problems with imuxsock not working, but I don't understand how the logs are stored on /var/log/syslog
. The last two lines are taken from the application and are consistent with the file applications stdout and file
Jul 6 18:07:18 26056b722779 rsyslogd: [origin software="rsyslogd" swVersion="7.4.4" x-pid="25" x-info="http://www.rsyslog.com"] start
Jul 6 18:07:18 26056b722779 rsyslogd: cannot create '/dev/log': Address already in use
Jul 6 18:07:18 26056b722779 rsyslogd: imuxsock does not run because we could not aquire any socket
Jul 6 18:07:18 26056b722779 rsyslogd-3000: activation of module imuxsock failed
Jul 6 18:07:18 26056b722779 rsyslogd: imklog: cannot open kernel log (/proc/kmsg): Operation not permitted.
Jul 6 18:07:18 26056b722779 rsyslogd-2145: activation of module imklog failed [try http://www.rsyslog.com/e/2145 ]
Jul 6 18:07:19 localhost local0: 2015-07-06 18:07:19 [AssocApp$:112] INFO <Undefined> Listening to queue ASSOCIATIONRULES
Jul 6 18:07:19 localhost local0: 2015-07-06 18:07:19 [AssocApp$:113] INFO <Undefined> Listening to queue ASSOCIATIONRULES
The container base image is ubuntu 14.04 with java installed. My CMD in dockerfile for container image:
CMD sudo service rsyslog start; java -cp /root/AssociationRules.jar AssocApp
and I start the container like this:
docker run -d -v /dev/log:/dev/log <IMAGE>
/etc/rsyslog.conf
#################
#### MODULES ####
#################
$ModLoad imuxsock # provides support for local system logging
$ModLoad imklog # provides kernel logging support
#$ModLoad immark # provides --MARK-- message capability
# provides UDP syslog reception
#$ModLoad imudp
#$UDPServerRun 514
# provides TCP syslog reception
#$ModLoad imtcp
#$InputTCPServerRun 514
# Enable non-kernel facility klog messages
$KLogPermitNonKernelFacility on
###########################
#### GLOBAL DIRECTIVES ####
###########################
#
# Use traditional timestamp format.
# To enable high precision timestamps, comment out the following line.
#
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
# Filter duplicated messages
$RepeatedMsgReduction on
#
# Set the default permissions for all log files.
#
$FileOwner syslog
$FileGroup adm
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022
#$PrivDropToUser syslog
#$PrivDropToGroup syslog
#
# Where to place spool and state files
#
$WorkDirectory /var/spool/rsyslog
#
# Include all config files in /etc/rsyslog.d/
#
$IncludeConfig /etc/rsyslog.d/*.conf
$ModLoad imudp
$UDPServerRun 514
and / etc / rsyslog.d / 50-default.conf
auth,authpriv.* /var/log/auth.log
*.*;auth,authpriv.none -/var/log/syslog
#cron.* /var/log/cron.log
#daemon.* -/var/log/daemon.log
kern.* -/var/log/kern.log
#lpr.* -/var/log/lpr.log
mail.* -/var/log/mail.log
#user.* -/var/log/user.log
#
# Logging for the mail system. Split it up so that
# it is easy to write scripts to parse these files.
#
#mail.info -/var/log/mail.info
#mail.warn -/var/log/mail.warn
mail.err /var/log/mail.err
#
# Logging for INN news system.
#
news.crit /var/log/news/news.crit
news.err /var/log/news/news.err
news.notice -/var/log/news/news.notice
#
# Some "catch-all" log files.
#
#*.=debug;\
# auth,authpriv.none;\
# news.none;mail.none -/var/log/debug
#*.=info;*.=notice;*.=warn;\
# auth,authpriv.none;\
# cron,daemon.none;\
# mail,news.none -/var/log/messages
#
# Emergencies are sent to everybody logged in.
#
*.emerg :omusrmsg:*
#
# I like to have messages displayed on the console, but only on a virtual
# console I usually leave idle.
#
#daemon,mail.*;\
# news.=crit;news.=err;news.=notice;\
# *.=debug;*.=info;\
# *.=notice;*.=warn /dev/tty8
# The named pipe /dev/xconsole is for the `xconsole' utility. To use it,
# you must invoke `xconsole' with the `-file' option:
#
# $ xconsole -file /dev/xconsole [...]
#
# NOTE: adjust the list below, or you'll go crazy if you have a reasonably
# busy site..
#
daemon.*;mail.*;\
news.err;\
*.=debug;*.=info;\
*.=notice;*.=warn |/dev/xconsole
source to share
I think you think too much about it. If I read your question correctly, you want your Docker container to write its syslog to the syslog of the host machine.
The error you are getting is probably because you are starting a second syslog daemon inside your container, which then wants to open a socket /dev/log
(which already exists).
Several ideas come to mind (all untested though. Enjoy with care):
-
Why don't you configure Log4j directly to write to the remote syslog daemon? Will this eliminate the need for syslog daemons inside the container and the need to mount
/dev/log
to the container?# Log to syslog log4j.appender.SYSLOG=org.apache.log4j.net.SyslogAppender log4j.appender.SYSLOG.syslogHost=<your-host-ip> # <-- INSERT HOST IP HERE log4j.appender.SYSLOG.layout=org.apache.log4j.PatternLayout
To transfer the container, it is better to configure the syslog server IP address when creating the container using the flag
--add-host
:docker run -d --add-host sysloghost:<host-ip-here> <IMAGE>
So you can just use
sysloghost
Log4J as the hostname in the config file. -
If you insist on running the syslog server inside the application container, you should be able (remember: all unverified!) To configure it to send all messages to the remote syslog server (somewhere in
/etc/rsyslog.conf
or/etc/rsyslog.d
):*.* @sysloghost:512 # UDP forwarding # *.* @@sysloghost:512 # TCP forwarding
-
Better yet, why not run a syslog daemon inside your own Docker container and link that container to your application containers?
Be aware that for both solutions, your syslog daemon on your host must be configured to listen on the TCP or UDP socket [ref] :
$ModLoad imudp
$UDPServerRun 514
Alternatively [ref] :
$ModLoad imtcp # needs to be done just once
$InputTCPMaxSessions 500
$InputTCPServerRun 514
source to share