Clever Geek Handbook

Python logging (logutils) with QueueHandler and QueueListener

Logging is required for a multiprocessing Python application. Using queues seems to be the best solution. The logutils library provides this.

Can log levels be set for two handlers independently? For example, in the test below, I would like STREAM 1 to have WARNING messages and STREAM 2 to have INFO messages. An INFO message is generated in the test log at the end of the code and should not be printed to the console from the STREAM 1 handler (WARNING). However, it outputs to both handlers.

For reference, I use this page http://plumberjack.blogspot.co.uk/2010/09/improved-queuehandler-queuelistener.html by Vinay Sajip who is the author of the library.

# System imports
import logging
import logging.handlers
try:
    import Queue as queue
except ImportError:
    import queue

# Custom imports
from logutils.queue import QueueHandler, QueueListener

# Get queue
q = queue.Queue(-1)

# Setup stream handler 1 to output WARNING to console
h1 = logging.StreamHandler()
f1 = logging.Formatter('STREAM 1 WARNING: %(threadName)s: %(message)s')
h1.setFormatter(f1)
h1.setLevel(logging.WARNING) # NOT WORKING. This should log >= WARNING

# Setup stream handler 2 to output INFO to console
h2 = logging.StreamHandler()
f2 = logging.Formatter('STREAM 2 INFO: %(threadName)s: %(message)s')
h2.setFormatter(f2)
h2.setLevel(logging.INFO) # NOT WORKING. This should log >= WARNING

# Start queue listener using the stream handler above
ql = QueueListener(q, h1, h2)
ql.start()

# Create log and set handler to queue handle
root = logging.getLogger()
root.setLevel(logging.DEBUG) # Log level = DEBUG
qh = QueueHandler(q)
root.addHandler(qh)

root.info('Look out!') # Create INFO message

ql.stop()
+5


source to share


3 answers


This is a limitation in the implementation of the method QueueListener.handle()

. It is currently:

def handle(self, record):
    record = self.prepare(record)
    for handler in self.handlers:
        handler.handle(record)

To do what you want it must be



def handle(self, record):
    record = self.prepare(record)
    for handler in self.handlers:
        # CHANGED HERE TO ADD A CONDITION TO CHECK THE HANDLER LEVEL
        if record.levelno >= handler.level:
            handler.handle(record)

I'll fix this at some point because I think it is better, but now you can subclass QueueListener

and override the method handle

in the subclass.

+1


source


I used this subclass to override the queue listener class. The other two methods addHandler and removeHandler let you add and remove handlers. CustomQueueListener

should be used in the same way as QueueListener

. Follow other logging examples for using addHandler()

and removeHandler()

.



class CustomQueueListener(QueueListener):
    def __init__(self, queue, *handlers):
        super(CustomQueueListener, self).__init__(queue, *handlers)
        """
        Initialise an instance with the specified queue and
        handlers.
        """
        # Changing this to a list from tuple in the parent class
        self.handlers = list(handlers)

    def handle(self, record):
        """
        Override handle a record.

        This just loops through the handlers offering them the record
        to handle.

        :param record: The record to handle.
        """
        record = self.prepare(record)
        for handler in self.handlers:
            if record.levelno >= handler.level: # This check is not in the parent class
                handler.handle(record)

    def addHandler(self, hdlr):
        """
        Add the specified handler to this logger.
        """
        if not (hdlr in self.handlers):
            self.handlers.append(hdlr)

    def removeHandler(self, hdlr):
        """
        Remove the specified handler from this logger.
        """
        if hdlr in self.handlers:
            hdlr.close()
            self.handlers.remove(hdlr)
+1


source


You can use an argument respect_handler_levels

that was added to the initializer QueueListener

from Python 3.5.

From QueueListener

doc :

If respect_handler_level

equal True

, the level of the handler is taken into account (compared to the level for the message) when deciding whether to pass messages to this handler; otherwise, the behavior is the same as in previous versions of Python - always pass every message to every handler.

In your case, you should replace initialization QueueListener

with:

ql = QueueListener(q, h1, h2, respect_handler_levels=True)
0


source






More articles:


All Articles