Clever Geek Handbook

Django logging format for Logstash

I am trying to set up a django app to write logs in a format that is easy for Logstash to use. (Inspired by the Winston registration package for Node)

Logstash expects a JSON object with a log message in the key "@message" and a timestamp "@timestamp". Does anyone know how to format Django logs?

So far, I have something similar to what is in the django docs.

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'formatters': {
        'verbose': {
            'format': '%(levelname)s %(asctime)s %(module)s %(process)d %(thread)d %(message)s'
        }
    },
    'handlers': {
        'file': {
            'level': 'DEBUG',
            'class': 'logging.FileHandler',
            'filename': './logs/app.log',
        },
    },
    'loggers': {
        'django.request': {
            'handlers': ['file'],
            'level': 'DEBUG',
            'propagate': True,
        },
    },
}

This makes logs like this:

WARNING 2015-05-22 21:20:19,082 base 3360 4588670976 Not Found: /register/fwq

Anyone have any suggestions on how JSON encodes a message in Logstash format?

Using Python 2.7.9, Django 1.8, and Logstash 1.5

+3


source to share


2 answers


You can catch Django logs with the following:

template:

DJANGO_LOGLEVEL (DEBUG|INFO|ERROR|WARNING|CRITICAL)
DJANGO_LOG %{DJANGO_LOGLEVEL:log_level}\s+%{TIMESTAMP_ISO8601:log_timestamp}\s+%{TZ:log_tz}\s+%{NOTSPACE:logger}\s+%{WORD:module}\s+%{POSINT:proc_id}\s+%{GREEDYDATA:content}


Filter:

filter {
     if [type] == "django" {
        grok {
             match => ["message", "%{DJANGO_LOG}" ]
        }

        date {
            match => [ "timestamp", "ISO8601", "YYYY-MM-dd HH:mm:ss,SSS"]
            target => "@timestamp"
        }
     }
}
+3


source


This package [1] provides a solution, but instead of writing to a file with formatters, it directly provides a handler for writing to logstash.

Hope it helps.



[1] https://github.com/vklochan/python-logstash

+4


source






More articles:


All Articles