Reading syslog from Python script

I am trying to emulate this shell command in Python using the systemd libraries http://www.freedesktop.org/software/systemd/python-systemd/journal.html

I'm really trying to emulate this command, but inside Python.

journalctl --since=-5m --no-pager

      

I've seen others do this in Python by invoking the log executable, but this is a pretty bad way to do it.

I wrote this simple script based on the documentation linked above

import select
from systemd import journal

j = journal.Reader()
j.log_level(journal.LOG_INFO)
# j.add_match(_SYSTEMD_UNIT="systemd-udevd.service")

j.seek_tail()
j.get_next()

while j.get_next():
    for entry in j:
        if entry['MESSAGE'] != "":
            print(str(entry['__REALTIME_TIMESTAMP'] )+ ' ' + entry['MESSAGE'])

      

There are several problems here.

  • the log seems to start from about 5 days ago, which means seek_tail is not working.
  • I am getting a lot of garbage, is there a specific filter here that I should use to match the data that I am getting from the journalctl command given at the beginning of the question?

Ideally for a longer time frame, I just want to follow this log based on a set of filters / matches to emulate the 'journalctl -f' command, but I just need to address this first. I want to get something like this, but it doesn't work either.

import select
from systemd import journal

j = journal.Reader()
j.log_level(journal.LOG_INFO)

# j.add_match(_SYSTEMD_UNIT="systemd-udevd.service")
j.seek_tail()

p = select.poll()
p.register(j, j.get_events())
while p.poll():

    while j.get_next():
        for entry in j:
            if entry['MESSAGE'] != "":
                print(str(entry['__REALTIME_TIMESTAMP'] )+ ' ' + entry['MESSAGE'])

      

+3


source to share


2 answers


I am also working on a similar python module.

According to the following links, we should call sd_journal_previous (in the python systemd module, that is, in the log .Reader (). Get_previous ()).

http://www.freedesktop.org/software/systemd/man/sd_journal_seek_tail.html

https://bugs.freedesktop.org/show_bug.cgi?id=64614

Also, your example code will consume 80 to 100% load because the read state remains "readable" even after a write is received, which results in too many polls ().



According to the following link, it seems to us that we should call sd_journal_process (in the python systemd module, i.e. journal.Reader (). Process ()) after each poll () in order to reset the readable state of the file descriptor.

http://www.freedesktop.org/software/systemd/man/sd_journal_get_events.html

In conclusion, your example code would be,

import select
from systemd import journal

j = journal.Reader()
j.log_level(journal.LOG_INFO)

# j.add_match(_SYSTEMD_UNIT="systemd-udevd.service")
j.seek_tail()
j.get_previous()
# j.get_next() # it seems this is not necessary.

p = select.poll()
p.register(j, j.get_events())

while p.poll():
    if j.process() != journal.APPEND:
        continue

    # Your example code has too many get_next() (i.e, "while j.get_next()" and "for event in j") which cause skipping entry.
    # Since each iteration of a journal.Reader() object is equal to "get_next()", just do simple iteration.
    for entry in j:
        if entry['MESSAGE'] != "":
            print(str(entry['__REALTIME_TIMESTAMP'] )+ ' ' + entry['MESSAGE'])

      

EDIT : Removed "j.get_next ()" after j.get_previous ().

+6


source


The previous answer works, and a hint about calling get_next () after seek_tail () is really needed.

An easier way (but obviously the previous version using polling is more flexible in large applications)

import systemd.journal

def main():
  j = systemd.journal.Reader()
  j.seek_tail()
  j.get_previous()
  while True:
    event = j.wait(-1)
    if event == systemd.journal.APPEND:
      for entry in j:
         print entry['MESSAGE']

if __name__ == '__main__':
  main()

      



If you want to see what's going on in detail, the next version of debug output support might be helpful (calling it with some arguments will enable it)

import sys
import systemd.journal

def main(debug):
  j = systemd.journal.Reader()
  j.seek_tail()
  j.get_previous()
  while True:
    event = j.wait(-1)
    if event == systemd.journal.APPEND:
      for entry in j:
         print entry['MESSAGE']
    elif debug and event == systemd.journal.NOP:
      print "DEBUG: NOP"
    elif debug and event == systemd.journal.INVALIDATE:
      print "DEBUG: INVALIDATE"
    elif debug:
      raise ValueError, event   

if __name__ == '__main__':
  main(len(sys.argv) > 1)

      

For me this result is always in one INVALIDATE at the beginning. Not sure what that means. Something that is being used as an invalid iterator sounds to me that I could somehow recreate / reopen / update it. But at least during my main testing, the code above works. Don't know if there could be any race conditions. Actually it would be difficult for me to explain how this code is race free.

+2


source







All Articles