Python query library throws exceptions on registration
The Python library requests
has some pretty strange quirks when it comes to its logging. Using the latest Python 2.7.8, I have the following code:
import requests
import logging
logging.basicConfig(
filename='mylog.txt',
format='%(asctime)-19.19s|%(task)-36s|%(levelname)s:%(name)s: %(message)s',
level=eval('logging.%s' % 'DEBUG'))
logger = logging.getLogger(__name__)
logger.info('myprogram starting up...', extra={'task': ''}) # so far so good
...
(ommited code)
...
payload = {'id': 'abc', 'status': 'ok'}
# At this point the program continues but throws an exception.
requests.get('http://localhost:9100/notify', params=payload)
print 'Task is complete! NotifyURL was hit! - Exiting'
My program seems to exit fine, however, inside the generated log file (mylog.txt), I always find the following exception:
KeyError: 'task'
Logged from file connectionpool.py, line 362
If I remove this:
requests.get('http://localhost:9100/notify', params=payload)
then there is no exception.
What exactly am I doing wrong here and how can I fix it? I am using requests v2.4.3.
source to share
As stated in t-8ch's answer , the logger is used internally by the requests
library and this library doesn't know anything about your parameters. A possible solution is to implant a custom filter in the library recorder (in this case, one of its modules):
class TaskAddingFilter(logging.Filter):
def __init__(self):
logging.Filter.__init__(self)
def filter(self, record):
record.args = record.args + ('task', '')
# ...
requestsLogger = logging.getLogger('requests.packages.urllib3.connectionpool')
requestsLogger.addFilter(TaskAddingFilter())
Potentially you need to add such filtering to all registrars from requests
that:
-
requests.packages.urllib3.util
-
requests.packages.urllib3.connectionpool
-
requests.packages
-
requests.packages.urllib3
-
requests.packages.urllib3.util.retry
-
requests
-
requests.packages.urllib3.poolmanager
in my version you can find them using the attribute logging.Logger.manager.loggerDict
. So, you can do something like this:
for name,logger in logging.Logger.manager.loggerDict.iteritems():
logger = logging.getLogger(name) # because of lazy initialization
if name.startswith('requests.'):
logger.addFilter(TaskAddingFilter())
TaskAddingFilter
might be a little smarter, for example adding a specific entry task
depending on where you are in your code. I only added the simplest solution for the code you provided - the exception is no longer encountered, but the wide range of possibilities seems obvious;)
source to share