Clever Geek Handbook

Reading file rotation and locking log files

I have a Python service spitting out logs into text files. It rotates them every ~ 400KB. Thus, the Python service opens the file descriptor by calling it in app.log. Then it writes the content to a file each time and flushes it to disk again. When it reaches a certain size, it closes the descriptor and moves it to app.log.1 and starts a new descriptor on app.log.

So I cannot change this service, but I have a C # application that will read these logs. I came across three scenarios:

  • If I just try to read these logs with new FileStream(path, FileMode.Open);

    , it won't let me, since the Python service has a handle.
  • If I try to open it with new FileStream(path, FileMode.Open, FileAccess.Read);

    , it allows me to read it, but if the service tries to rotate the log, it fails, since my C # application now has a file descriptor.
  • And if I try to open the file with new FileStream(path, FileMode.Open, FileAccess.Read, FileShare.Delete);

    , my Python service doesn't crash when the file is deleted, but it fails to create a new descriptor in app.log, since the C # app will still have a descriptor for it.

The only solution I know of would be to use Windows Shadow Copy (VSS) to create a snapshot of the logs and then read that snapshot, but that would be quite expensive as we need to query the logs every 5 minutes.

Also, I'm not interested in reading rotated logs, app.log.1, app.log.2, etc.

Writing to text files under Windows seems sick with all the locks / knobs. Anyone have any suggestions?

+3


source to share


3 answers


You should be able to open the file as suggested by Dmitry Popov in his answer as shown below, and not affect the writing of it in Python, however this depends on what the Python application is blocking in the file, it might block you completely, you have nothing to do. to prevent this without jailbreaking Windows.

FileSream fs = File.Open(@"c:\Test.txt", FileMode.Open, FileAccess.Read, FileShare.ReadWrite | FileShare.Delete)

The FileStream object created in this way will still be attached to the same file after the operating system file move operation was performed to rename it.

So let's assume your python application opens the Test.log file and starts writing to it. You can read any data written to it (after python flushes its buffer) using the file stream returned from the line above. The python application can close and reopen the file as often as it wants to write each time, and the reader application will stay connected with it. When the python application issues a file move operation to rename the file to Test1.log, the file stream returned above will still be attached to the file now called Test1.log so that you can continue reading to the end of the file before starting a new log file. if that's what you want. This is one caveat. Python application should use Move / Rename operation and not copy file to new and delete old one,I would be surprised if this is what he does, though.

There is a chance that your reading application will reach the end of the file before your written application finishes reading. In this case, fs.Read will continue to return 0 after a timeout until the write application opens the file and writes some more. You can make the timeout very long / infinite if you like.



Since you don't want to read to the end of one file before starting a new one, you can simply close and reopen the file at regular intervals. The log file without a numeric suffix must always be the last one.

If you want your reader application to read to the end of one log file before starting at the beginning of the next, you will need to work when the writing application has finished writing to the log file. He also needs to figure out what is now called a file so that he can read n-1. Is there some marker written by a python application that you can look for to mark the end of a file? Is he writing "End of the Journal" or something similar?

It should also be warned that there will be short periods of time when LogFile n-1 does not exist. This is because if you have log files 0, 1, 2 and 3, it has to make log file 3 in log file 4 before it can make log file 2 in log file 3. While it does so, it will short period when you have log files 0, 1, 2, 4 and 3.

Personally, I would find a developer who wrote a log for your Python application gave him the evil eye to cause this headache in the first place. What happened to the most recent log file having the highest number?

+3


source 


using (FileStream fs = new FileStream(path, FileMode.Open, FileAccess.Read, FileShare.Read))
{
  //Do works
}


C # thread does not lock the file in this case, your Python script can write and close the file to create another without deadlock.

+1


source


You can combine flags FileShare

:

FileShare.Write | FileShare.Delete

Here's a demo:

using (var cSharp = new FileStream(filename, FileMode.Open, FileAccess.Read, FileShare.Write | FileShare.Delete))
{
    // The Python service will be able to change and to rename the file:
    using (var python = new FileStream(filename, FileMode.Open, FileAccess.Write, FileShare.Read))
    {
    }
    File.Move(filename, newFilename);
}

You will have to deal with concurrency. You can use FileSystemWatcher to monitor file changes.

0


source






More articles:


All Articles