Disable logging for python module in one context, but not
I have some code that uses a module requests
to communicate with the logging API. However, requests
he himself, through urllib3
, performs the logging. Naturally, I need to disable logging so that requests to the logging API do not cause an endless loop of logs. So in the module I am logging into the log I do logging.getLogger("requests").setLevel(logging.CRITICAL)
to disable normal query logs.
However, this code is meant to load and run arbitrary user code. Since the python module logging
seems to use global state to manage settings for a given logger, I am worried that the user code might log back in and cause problems, for example if they naively use the requests module in their code without realizing that they have I have to disable logging for this for some reason.
How can I turn off logging for a query module when it is executed from the context of my code, but does not affect the logger state for the module from the user's perspective? Some kind of context manager that silences registration calls for code in the manager would be ideal. The ability to load a query module with a unique one __name__
, so the log might use a different name, but it's a little confusing. However, I cannot find a way to do this.
Unfortunately the solution will have to handle multiple threads, so procedurally disabling the logging and then running the API call and then enabling it again won't work as the global state is mutated.
source to share
I think I have a solution for you:
The module is logging
built for thread safety :
The logging module must be thread safe with no special work to be done by its clients. He achieves this by using threaded locks; there is one lock to serialize access to shared data modules and each handler also creates a lock to serialize access to its underlying I / O.
Fortunately, it provides a second lock, mentioned in the public API: Handler.acquire()
allows you to acquire a lock for a specific log handler (and Handler.release()
releases it again). Acquiring a lock blocks all other threads that are trying to write a record to be processed by this handler until the lock is released.
This allows you to manage the state of the handler in a thread-safe way. The caveat is that because it is intended to be a blocking handler I / O, the lock will only be acquired in emit()
. This way, only once, when the entry goes through the filter and log levels and is released by a special handler, a lock will be obtained. This is why I had to subclass the handler and create SilencableHandler
.
So the idea is this:
- Get the topmost registrar for a module
requests
and stop it from propagating - Create your own
SilencableHandler
and add it to the query log - Use a context manager
Silenced
to selectively disableSilencableHandler
main.py
from Queue import Queue
from threading import Thread
from usercode import fetch_url
import logging
import requests
import time
logging.basicConfig(level=logging.INFO)
log = logging.getLogger(__name__)
class SilencableHandler(logging.StreamHandler):
def __init__(self, *args, **kwargs):
self.silenced = False
return super(SilencableHandler, self).__init__(*args, **kwargs)
def emit(self, record):
if not self.silenced:
super(SilencableHandler, self).emit(record)
requests_logger = logging.getLogger('requests')
requests_logger.propagate = False
requests_handler = SilencableHandler()
requests_logger.addHandler(requests_handler)
class Silenced(object):
def __init__(self, handler):
self.handler = handler
def __enter__(self):
log.info("Silencing requests logger...")
self.handler.acquire()
self.handler.silenced = True
return self
def __exit__(self, exc_type, exc_value, traceback):
self.handler.silenced = False
self.handler.release()
log.info("Requests logger unsilenced.")
NUM_THREADS = 2
queue = Queue()
URLS = [
'http://www.stackoverflow.com',
'http://www.stackexchange.com',
'http://www.serverfault.com',
'http://www.superuser.com',
'http://travel.stackexchange.com',
]
for i in range(NUM_THREADS):
worker = Thread(target=fetch_url, args=(i, queue,))
worker.setDaemon(True)
worker.start()
for url in URLS:
queue.put(url)
log.info('Starting long API request...')
with Silenced(requests_handler):
time.sleep(5)
requests.get('http://www.example.org/api')
time.sleep(5)
log.info('Done with long API request.')
queue.join()
usercode.py
import logging
import requests
import time
logging.basicConfig(level=logging.INFO)
log = logging.getLogger(__name__)
def fetch_url(i, q):
while True:
url = q.get()
response = requests.get(url)
logging.info("{}: {}".format(response.status_code, url))
time.sleep(i + 2)
q.task_done()
Output example:
(Note that the call is http://www.example.org/api
not logged and all threads that try to log requests are blocked for the first 10 seconds).
INFO:__main__:Starting long API request...
INFO:__main__:Silencing requests logger...
INFO:__main__:Requests logger unsilenced.
INFO:__main__:Done with long API request.
Starting new HTTP connection (1): www.stackoverflow.com
Starting new HTTP connection (1): www.stackexchange.com
Starting new HTTP connection (1): stackexchange.com
Starting new HTTP connection (1): stackoverflow.com
INFO:root:200: http://www.stackexchange.com
INFO:root:200: http://www.stackoverflow.com
Starting new HTTP connection (1): www.serverfault.com
Starting new HTTP connection (1): serverfault.com
INFO:root:200: http://www.serverfault.com
Starting new HTTP connection (1): www.superuser.com
Starting new HTTP connection (1): superuser.com
INFO:root:200: http://www.superuser.com
Starting new HTTP connection (1): travel.stackexchange.com
INFO:root:200: http://travel.stackexchange.com
Threading code is based on Doug Hellman's articles on threading and queues .
source to share