Improving the performance of Perl code

Do you have a better idea as to what "bit" is costing you productivity?

The reason I'm asking is some holy trinity of performance bottlenecks:

• CPU - the actual operations performed on the processor
• "Active" memory (size of memory profile or available RAM and how much you are shuffling).
• IO - transfer data to / from disk.

You can often compromise one against the other - improve CPU efficiency by creating lookup tables, things like that.

Operations like map are the ones I'm starting to get closer to - things like map / sort / grep are very efficient, but may use a less optimal algorithm.

If you are CPU bound you can try using multithreading or forking to increase CPU access. At first glance, it looks like you have no dependencies on the processing of "matrix.txt" (for example, each row is self-contained), so it might be a good candidate for parallelism.

I would consider using Parallel :: ForkManager to wrap this loop while

. The downside to this is that you will have a non-deterministic order of your output to refer to.

So a starter for 10 could be:

use List::Util 'sum';
use Data::Dumper;
use Fcntl qw(:flock);

use Parallel::ForkManager;

my $mgr = Parallel::ForkManager->new(10);

my @indexes = do {
    open my $fh, '<', "INPUT.txt";
    map {
        [ map { $_ - 1 } split ' ' ]
    } <$fh>;
};
open my $infh,   '<', "matrix.txt";
open my $out_fh, '>', "output.txt";
while (<$infh>) {
    $mgr->start and next;
    my @vals = split ' ';
    my $output_line = join( '    ', map { sum( @vals[@$_] ) } @indexes ),
        "\n";
    {
        flock( $out_fh, LOCK_EX );
        print {$out_fh} $output_line;
    }
}
close $out_fh;

      

        
        
        
      

    

Note. This will work, but you end up with a random order of output, which is almost certainly not what you want. But it will use 10 processors at the same time to perform the "join / map / sum" operation.

(Of course, if you're tied to IO, this won't help.)

But for IO synchronization, I think multithreading is not a bad option:

 use warnings;
 use strict;

use List::Util 'sum';

use threads; 
use Thread::Queue;

my $line_q = Thread::Queue -> new(); 
my $output_q = Thread::Queue -> new(); 

my %line_output : shared; 

    my @indexes = do {
        open my $fh, '<', "INPUT.txt";
        map {
            [ map { $_ - 1 } split ' ' ]
        } <$fh>;
};


sub generate_output {
   while ( my $item = $line_q -> dequeue() ) {
   print "processing $item \n";
       my ( $line_num, @vals ) = split ( ' ', $item );           
       $output_q -> enqueue($line_num.":". join('    ', map {sum(@vals[@$_])} @indexes ). "\n");
   }
}

sub coalesce_output {
    open my $out_fh, '>', "output.txt";
    my $current_line = 0; 
    my %lines;
    while ( my $item = $output_q -> dequeue ) {
        my ( $line_num, $output_line ) = split ( ":", $item );
        if ( $line_num = $current_line ) { 
            print {$out_fh} $output_line;
            $current_line++; 
        }
        else {
           $lines{$line_num} = $output_line; 
        }
        while ( defined $lines{$current_line} ) {
            print {$out_fh} $lines{$current_line};
            delete $lines{$current_line};
            $current_line++;
        }
    }
}




open my $infh,   '<', "matrix.txt";

my @workers;
for ( 1..10 ) {
  push ( @workers, threads -> create ( \&generate_output ) ); 
}

threads -> create ( \&coalesce_output );

while (my $line = <$infh>) {
    $line_q -> enqueue ( "$.: $line" );
}

$line_q -> end();
foreach my $thr ( @workers ) {
  $thr -> join(); 
}

$output_q -> end(); 

      

        
        
        
      

    

For example. Rotates 10 "workers" to perform parallel summing operations, and one "output" stream to write data in the correct order.

So something like:

use warnings;
use strict;

use List::Util 'sum';

use threads;
use Thread::Queue;

my $line_q   = Thread::Queue->new();
my $output_q = Thread::Queue->new();

my @indexes = do {
    open my $fh, '<', "INPUT.txt";
    map {
        [ map { $_ - 1 } split ' ' ]
    } <$fh>;
};


sub generate_output {
    while ( my $item = $line_q->dequeue() ) {

        #print "processing $item \n";
        my ( $line_num, @vals ) = split( ' ', $item );
        $output_q->enqueue( $line_num . ":"
                . join( '    ', map { sum( @vals[@$_] ) } @indexes )
                . "\n" );
    }
}

sub coalesce_output {
    open my $out_fh, '>', "output.txt";
    my $current_line = 1;
    my %lines;
    while ( my $item = $output_q->dequeue ) {

        my ( $line_num, $output_line ) = split( ":", $item );

        #     print "Got $line_num ($current_line) $item\n";
        if ( $line_num = $current_line ) {

            #   print "printing $current_line = $output_line\n";
            print {$out_fh} $output_line;
            $current_line++;
        }
        else {
            $lines{$line_num} = $output_line;
        }
        while ( defined $lines{$current_line} ) {

    #   print "printing  (while) $current_line = $lines{$current_line}\n";
            print {$out_fh} $lines{$current_line};
            delete $lines{$current_line};
            $current_line++;
        }
    }
}


open my $infh, '<', "matrix.txt";

my @workers;
for ( 1 .. 40 ) {
    push( @workers, threads->create( \&generate_output ) );
}

threads->create( \&coalesce_output );

while ( my $line = <$infh> ) {
    $line_q->enqueue("$. $line");
}

$line_q->end();
foreach my $thr (@workers) {
    $thr->join();
}

$output_q->end();
foreach my $thr ( threads -> list ) { $thr -> join(); }

      

        
        
        
      

    

Produces (+ more):

 1    1    1    1    1    1    1    1    1    1    1    1    2    2    2    2
 2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2
 2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2
 2    2    2    2    2    2    2    2    2    2    2    2    2    2    2    2
 2    2    2    2    2    2    2    2    2    2    2    2    2    2    3    3
 3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3
 3    3    3    3    3    3    3    3    3    3    3    3    3    3    3    3

      

        
        
        
      

    

At the end of the day, though, it rather depends on what your limiting factor is.

Running a quick and dirty test gives

Started at 1417007048,
finished at 1417007064
Took:16s

      

        
        
        
      

    

Vs.

Started at 1417007118
finished at 1417007161
Took:43s

      

        
        
        
      

    

(I have not confirmed the outputs of both exhaustively)