Run a shell script in parallel in bash / linux
I have a shell script job.sh
.
below:
#!/bin/bash
table=$1
sqoop job --exec ${table}
Now that I do. /job.sh table1
The script runs successfully.
I have table names in a file tables.txt
.
Now I want to loop over a file tables.txt
and execute the job.sh
script 10 times in parallel .
How can i do this?
Ideally, when I execute the script, I want it to do as shown below:
./job.sh table1 ./job.sh table2 ./job.sh table3 ./job.sh table4 ./job.sh table5 ./job.sh table6 ./job.sh table7 ./job.sh table8 ./job.sh table9 ./job.sh table10
What are the available options?
source to share
Simple with GNU Parallel
parallel -a tables.txt --dry-run sqoop job --exec {}
Output result
sqoop job --exec table7
sqoop job --exec table8
sqoop job --exec table9
sqoop job --exec table6
sqoop job --exec table5
sqoop job --exec table4
sqoop job --exec table3
sqoop job --exec table2
sqoop job --exec table1
sqoop job --exec table10
If it looks correct, just uninstall --dry-run
and run again for real.
If you want to complete 4 tasks at a time, use:
parallel -j 4 ....
If you want to do one job per processor core, this is the default, so you don't need to do anything.
If you want the jobs to be kept in order, add the option -k
:
parallel -k ...
source to share
Option 1
Run all scripts as background processes by adding &
eg.
./job.sh table1 & ./job.sh table2 & ./job.sh table3 &
However, this will execute all tasks at the same time!
Option 2
For more time or memory scenarios, you can run a limited number of tasks at the same time using xargs
, as described, for example, here .
source to share