Configuring Python protocol for multiple child log files while disabling main log
I have several related but separate Python scripts that use two internal modules that use logging.
The first script works fine using a root logger and commits entries from two modules. However, with the second script, I want to have the main log, but when it iterates through the server list, it sends the logs to the log file for each computer, pausing the logging in the main log file and console. At the moment I have a hacky solution which I will show below.
import logging
DEFAULT_LOG_FORMAT = "%(asctime)s [%(levelname)s]: %(message)s"
DEFAULT_LOG_LEVEL = logging.INFO
def get_log_file_handler(filename, level=None, log_format=None):
file_handler = logging.FileHandler(filename=filename, encoding="utf-8", mode="w")
file_handler.setLevel(level or DEFAULT_LOG_LEVEL)
file_handler.setFormatter(logging.Formatter(log_format or DEFAULT_LOG_FORMAT))
return file_handler
def process(server):
server_file_handler = get_log_file_handler("%s.log" % server.name)
root_logger = logging.getLogger()
# This works, but is hacky
main_handlers = list(root_logger.handlers) # copy list of root log handlers
root_logger.handlers = [] # empty the list on the root logger
root_logger.addHandler(server_file_handler)
try:
# do some stuff with the server
logging.info("This should show up only in the server-specific log file.")
finally:
root_logger.removeHandler(server_file_handler)
# Add handlers back in
for handler in main_handlers:
root_logger.addHandler(handler)
def main():
logging.basicConfig(level=DEFAULT_LOG_LEVEL)
logging.getLogger().addHandler(get_log_file_handler("main.log"))
servers = [] # retrieved from another function, just here for iteration
logging.info("This should show up in the console and main.log.")
for server in servers:
process(server)
logging.info("This should show up in the console and main.log again.")
if __name__ == "__main__":
main()
I'm looking for a less brave way to do this. I understand that just calling logging.info () and similar is the problem and changed the code in two modules:
logger = logging.getLogger("moduleA")
and
logger = logging.getLogger("moduleB")
So the main script, be it scriptA.py or scriptB.py, using the root log, you will receive events from these two modules, which are propagated and written to main.log. Several other solutions I've tried use a filter for all existing handlers, which will ignore everything from "moduleA" and "moduleB".
My next thought is to create a new named logger for individual servers using server_file_handler as the only handler for them and add that as a handler for the two registrars and also remove those handlers at the end of the process (). Then I could set the root logger level to WARNING, so all INFO / DEBUG statements from the two modules would only access the server logger.
I cannot accurately use the hierarchical log naming convention unless it supports wildcards, since I end up with:
logging.getLogger("org.company") # main logger for script
logging.getLogger("org.company.serverA")
logging.getLogger("org.company.serverB")
logging.getLogger("org.company.moduleA")
logging.getLogger("org.company.moduleB")
The two-module recording will only apply to the main recorder, not the two server logs.
Basically it is the "they-waiting", "me-need-schedule" problem. Has anyone done something like this before, and what's the most Pythonic way to do it?
source to share
This is an interesting problem. My first instinct was to use it logger.getChild
, but the default implementation won't do what you want. Assuming you can add handlers dynamically to a single logger, it still won't do what you want as you will have to add filters to both the main file handler and server handlers to filter messages that shouldn't go into the server magazines and vice versa.
However, the good news is that the custom log that generates the handler for each child is actually quite simple and can be done with a simple subclass that changes getChild
and not much more.
Big changes below - it only HandlerPerChildLogger
, Logger
which is different from the usual Logger
that two arguments are required for it, not just one parameter name
.
import logging
DEFAULT_LOG_FORMAT = "%(asctime)s [%(levelname)s]: %(message)s"
DEFAULT_LOG_LEVEL = logging.INFO
class HandlerPerChildLogger(logging.Logger):
selector = "server"
def __init__(self, name, handler_factory, level=logging.NOTSET):
super(HandlerPerChildLogger, self).__init__(name, level=level)
self.handler_factory = handler_factory
def getChild(self, suffix):
logger = super(HandlerPerChildLogger, self).getChild(suffix)
if not logger.handlers:
logger.addHandler(self.handler_factory(logger.name))
logger.setLevel(DEFAULT_LOG_LEVEL)
return logger
def file_handler_factory(name):
handler = logging.FileHandler(filename="{}.log".format(name), encoding="utf-8", mode="a")
formatter = logging.Formatter(DEFAULT_LOG_FORMAT)
handler.setFormatter(formatter)
return handler
logger = HandlerPerChildLogger("my.company", file_handler_factory)
logger.setLevel(DEFAULT_LOG_LEVEL)
ch = logging.StreamHandler()
fh = logging.FileHandler(filename="my.company.log", encoding="utf-8", mode="a")
ch.setLevel(DEFAULT_LOG_LEVEL)
fh.setLevel(DEFAULT_LOG_LEVEL)
formatter = logging.Formatter(DEFAULT_LOG_FORMAT)
ch.setFormatter(formatter)
fh.setFormatter(formatter)
logger.addHandler(ch)
logger.addHandler(fh)
def process(server):
server_logger = logger.getChild(server)
server_logger.info("This should show up only in the server-specific log file for %s", server)
server_logger.info("another log message for %s", server)
def main():
# servers list retrieved from another function, just here for iteration
servers = ["server1", "server2", "server3"]
logger.info("This should show up in the console and main.log.")
for server in servers:
process(server)
logger.info("This should show up in the console and main.log again.")
if __name__ == "__main__":
main()
source to share
Use naming and log proxy
If your modules used a named logger org.company.moduleX
then you can simply add a file handler to the named log org.company
and block propagation to your root logger handlers using Logger.propogate
Implementing it as a context manager to make it enjoyable.
import contextlib
log = logging.getLogger("org.company.scriptB")
@contextlib.contextmanager
def block_and_divert_logging(logger, new_handler):
logger.propagate = False
logger.addHandler(new_handler)
try:
yield
finally:
logger.propogate = True
logger.removeHandler(new_handler)
def process(server):
server_file_handler = get_log_file_handler("%s.log" % server.name)
logger_block = logging.getLogger("org.company")
with block_and_divert_logging(logger_block, server_file_handler):
# do some stuff with the server
log.info("This should show up only in the server-specific log file.")
This will stop any messages from loggers at or below the level org.company
from accessing the root logger handlers. Instead, they will be processed by your file handler.
However, this means that any registrars not named as org.company.something
will still access the root registrars.
source to share