How to use Log4perl to rotate log files in a multithreaded perl application
Below is some sample code I tried to rotate a log file in a multithreaded application using log4perl. But it works great if it is not a multi-threaded application. The logs do not change, but the log file grows. Can anyone point me to where I am going wrong?
use strict;
use warnings;
use Log::Log4perl;
use POSIX;
use threads;
use threads::shared;
my @InputFiles;
my $InputDirectory=$ARGV[0];
my $LogName=$ARGV[1];
opendir(DIR,$InputDirectory) or die "could not open the input directory";
@InputFiles=readdir(DIR);
close(DIR);
my $file;
#logger_configuration
my $log_conf ="
log4perl.rootLogger = DEBUG, LOG1
log4perl.appender.LOG1 = Log::Dispatch::FileRotate
log4perl.appender.LOG1.filename = $LogName
log4perl.appender.LOG1.mode = append
log4perl.appender.LOG1.autoflush = 1
log4perl.appender.LOG1.size = 10000
log4perl.appender.LOG1.max = 20
log4perl.appender.LOG1.layout = Log::Log4perl::Layout::PatternLayout
log4perl.appender.LOG1.layout.ConversionPattern = \%d{yyyy-MM-dd HH:mm:ss}|\%P|\%m|\%n
";
#loading the configuration file
Log::Log4perl::init(\$log_conf);
#creating logger instance
my $logger = Log::Log4perl->get_logger();
my $thread_count=5;
my $file_total= scalar @InputFiles;
#print STDERR "$file_total\n";
#dividing total files among the no of given threads
my $div = $file_total/$thread_count;
$div = ceil($div);
my $start = 0;
my $end = $div;
my @threads;
for (my $count = 1; $count <=$thread_count ; $count++)
{
my $thread = threads->new(\&process,$start,$end);
push(@threads,$thread);
$start = $end;
$end = $end + $div;
if ($end > $file_total)
{
$end = $file_total;
}
}
foreach (@threads)
{
$_->join;
}
sub process
{
my $lstart = shift;
my $lend = shift;
my $id = threads->tid();
for (my $index = $lstart; $index < $lend; ++$index)
{
$logger->info($InputFiles[$index]);
}
}
source to share
OK, pretty fundamentally your problem is this: your "logger" is created before your streams start. This means that all your streams will have the same file descriptors.
This will inherently cause problems if you don't have some sort of arbitration mechanism for the IO file. Think of your threads as separate programs, everyone trying to open and write to the same file - and you can see how messy it can be.
Instead, I would suggest what you need to do is create another stream for the logger and send the IO via something like Thread::Queue
use Thread::Queue;
my $log_q = Thread::Queue -> new();
sub logger_thread {
#init logger here
while ( my $log_item = $log_q -> dequeue() ) {
$logger -> info ( $log_item );
}
}
my $log_thread = threads -> create ( \&logger_thread );
And then replace $logger -> info (....)
with:
$log_q -> enqueue($message_to_log);
Then, once you have attached all your "process" threads (for example, as you are now), close the log thread:
$log_q -> end();
$log_thread -> join();
This will cause each of the threads to queue log messages, and once they finish (and join) you close the queue so the logger knows it is "done" - and therefore exits as soon as the queue is empty. and maybe joined.
Multi-threaded IO file is confusing, so it's best to avoid as much as possible.
source to share