Clever Geek Handbook

How to run a shell script in multiple jobs

I have a build script that is very slow, especially on Solaris. I want to improve its performance by running it on multiple jobs. How can i do this?

+3


source to share


2 answers


Try GNU Parallel , it's pretty easy to use:

GNU parallel is a shell tool for executing jobs in parallel using one or more computers. The task can be a single command or a small script that needs to be run for each of the input lines. Typical input is a list of files, a list of hosts, a list of users, a list of URLs, or a list of tables. The job can also be a command that reads from a pipe. GNU parallel can split the input and connect it to commands in parallel.

If you use xargs and tee today, you will find that GNU parallel is very easy to use, as GNU parallel is written in the same way as xargs. If you write loops in the shell, you'll find that GNU Parallel can replace most loops and make them run faster by doing multiple jobs in parallel.

GNU parallel ensures that the output from the commands is the same as you would get if you ran the commands sequentially. This allows the output from GNU parallel to be used as input for other programs.

For each line of input, GNU parallel will execute the command with line as arguments. If no command is specified, the input line is executed. Several lines will run in parallel. GNU parallel can often be used as a replacement for xargs or cat | bash.

You mentioned that this is a build script. If you are using a command line utility make

, you can parallelize assemblies using the make -j<N>

option :



GNU make knows how to execute multiple recipes at once. Usually make only executes one recipe at a time, waiting for it to complete before executing the next. However, the '-j or' --jobs option tells make to run multiple recipes at the same time.

Additionally, there is distcc

one that can be used with make

to distribute compilation to multiple hosts:

export DISTCC_POTENTIAL_HOSTS='localhost red green blue'
cd ~/work/myproject; 
make -j8 CC=distcc
+6


source


Parallel to GNU not bad. @Maxim is a good +1 suggestion.

At one time, if you cannot install new software, try this for a slow command that needs to run multiple times, run slowcommand 17 times. Change things to suit your needs:



#!/bin/bash
cnt=0

while [ $cnt -le 17  ]    # loop 17 times
do
   slow_command  &
   cnt=$(( $cnt + 1 ))
   [  $(( $cnt % 5 )) -eq  0 ] && wait  # 5 jobs at a time in parallel 
done
wait      # you will have 2 jobs you di not wait for in the loop 17 % 5 == 2
+2


source






More articles:


All Articles