In logstash, do I need separate file entries for logically different application logs?

(Logstash 1.4.2 on Windows)

In our system, a "product" is a high-level group of related web applications. Each web application is configured to record a dedicated log file named after the application name (for example, MyProduct.ApplicationA.log and MyProduct.ApplicationB.log). All web applications for a given product write their log files to the same folder (c: \ Logs \ MyProduct \; c: \ Logs \ MyOtherProduct).

I need to configure logstash to monitor all log files for all applications for all products. I was hoping to use:

input {
    file {
        path => "c:/Logs/**/*.log"
        exclude => ["Info.*", "Warn.*", "Error.*"]
        sincedb_path => "c:/logstash/.sincedb"
        sincedb_write_interval => 1
    }
}

      

In the first run I see a lot of output to stdout, which I assume are documents called "first contact".

After all the log files (from multiple applications) have been parsed first, if applications generate log entries, they seem to be collected and output. Things are good.

However, if I restart logstash, all the logs seem to be parsed again - as if sincedb is not being honored. I have looked at other SO questions that describe a similar duplicate and repair experience (e.g. logstash + elasticsearch: reloads the same data ), however I believe I have additional information that might indicate that I am actually using the file incorrectly ...

If I installed multiple input files like so:

file {
    path => "c:/Logs/MyProduct/MyProduct.ApplicationA.log"
    exclude => ["Info.*", "Warn.*", "Error.*"]
    sincedb_path => "c:/logstash/.sincedb_A"
    sincedb_write_interval => 1
}
file {
    path => "c:/Logs/MyProduct/MyProduct.ApplicationB.log"
    exclude => ["Info.*", "Warn.*", "Error.*"]
    sincedb_path => "c:/logstash/.sincedb_B"
    sincedb_write_interval => 1
}

      

Then restarting logstash does not update the existing files and does sincedb's honest work for logical grouping. This leads me to think that maybe I was thinking about file input the wrong way: do I need to set up separate file inputs for each application?

(Looking at the contents of sincedb, there is only one line, for example

0 0 2 661042

      

and it becomes apparent that multiple files cannot be tracked)

Am I missing something that would allow me to have a generic globular styling globally without having to customize each application?

+3


source to share


1 answer


It looks like you are facing the known sincedb error on Windows



Your workaround to add a file {} block with a separate sincedb_path for each file is probably the best solution until the bug is fixed.

+3


source







All Articles