How to use Log4perl to rotate log files in a multithreaded perl application

Question

How to use Log4perl to rotate log files in a multithreaded perl application

Below is some sample code I tried to rotate a log file in a multithreaded application using log4perl. But it works great if it is not a multi-threaded application. The logs do not change, but the log file grows. Can anyone point me to where I am going wrong?

use strict;
use warnings;
use Log::Log4perl;
use POSIX;
use threads;
use threads::shared;

my @InputFiles;

my $InputDirectory=$ARGV[0];
my $LogName=$ARGV[1];
opendir(DIR,$InputDirectory) or die "could not open the input directory";
@InputFiles=readdir(DIR);
close(DIR);
my $file;

    #logger_configuration
my $log_conf ="
   log4perl.rootLogger              = DEBUG, LOG1

   log4perl.appender.LOG1           = Log::Dispatch::FileRotate
   log4perl.appender.LOG1.filename  = $LogName
   log4perl.appender.LOG1.mode      = append
   log4perl.appender.LOG1.autoflush = 1
   log4perl.appender.LOG1.size      = 10000
   log4perl.appender.LOG1.max       = 20
   log4perl.appender.LOG1.layout    = Log::Log4perl::Layout::PatternLayout
   log4perl.appender.LOG1.layout.ConversionPattern = \%d{yyyy-MM-dd HH:mm:ss}|\%P|\%m|\%n
";

#loading the configuration file
Log::Log4perl::init(\$log_conf);

#creating logger instance
my $logger = Log::Log4perl->get_logger();

my $thread_count=5;
my $file_total= scalar @InputFiles;
#print STDERR "$file_total\n";

#dividing total files among the no of given threads
my $div = $file_total/$thread_count;
$div = ceil($div);
my $start = 0;
my $end = $div;
my @threads;
for (my $count = 1; $count <=$thread_count ; $count++) 
{
    my $thread = threads->new(\&process,$start,$end);
    push(@threads,$thread);        
    $start = $end;
    $end = $end + $div;
    if ($end > $file_total)
    {
        $end = $file_total;
    }
}

foreach (@threads) 
{
   $_->join;
}

sub process
{
    my $lstart = shift;
    my $lend = shift;
    my $id = threads->tid();
    for (my $index = $lstart; $index < $lend; ++$index) 
    {   
      $logger->info($InputFiles[$index]);
    }
}

+3

multithreading perl log4perl

Doe 02 dec. '14 at 15:00

source to share

1 answer

Sobrique · Answer 1 · 2014-12-02T16:02:54+0000

OK, pretty fundamentally your problem is this: your "logger" is created before your streams start. This means that all your streams will have the same file descriptors.

This will inherently cause problems if you don't have some sort of arbitration mechanism for the IO file. Think of your threads as separate programs, everyone trying to open and write to the same file - and you can see how messy it can be.

Instead, I would suggest what you need to do is create another stream for the logger and send the IO via something like Thread::Queue

use Thread::Queue;
my $log_q = Thread::Queue -> new(); 

sub logger_thread {
     #init logger here 

    while ( my $log_item = $log_q -> dequeue() ) {
        $logger -> info ( $log_item );
    }
}

my $log_thread = threads -> create ( \&logger_thread );

And then replace $logger -> info (....)

with:

$log_q -> enqueue($message_to_log);

Then, once you have attached all your "process" threads (for example, as you are now), close the log thread:

$log_q -> end();
$log_thread -> join();

This will cause each of the threads to queue log messages, and once they finish (and join) you close the queue so the logger knows it is "done" - and therefore exits as soon as the queue is empty. and maybe joined.

Multi-threaded IO file is confusing, so it's best to avoid as much as possible.

How to use Log4perl to rotate log files in a multithreaded perl application

More articles: