How to use GNU to run a list of commands that execute 4 commands at the same time
I have a list of shell commands that I would name. Up to four processes must be running at the same time.
Once the number of processes drops below 4, the next command will be sent to the shell until the entire command has finished executing.
I already have a "partially" working solution from stackoverflow stickers which, however, rely on wait -n, which is not available on my debian wheezy install.
One guy mentioned GNU Parallel. Can this be achieved? Any recommendations are really appreciated!
nohup scrapy crawl urlMonitor -a slice=0 &
nohup scrapy crawl urlMonitor -a slice=1 &
nohup scrapy crawl urlMonitor -a slice=2 &
nohup scrapy crawl urlMonitor -a slice=3 &
nohup scrapy crawl urlMonitor -a slice=4 &
nohup scrapy crawl urlMonitor -a slice=5 &
nohup scrapy crawl urlMonitor -a slice=6 &
nohup scrapy crawl urlMonitor -a slice=7 &
nohup scrapy crawl urlMonitor -a slice=8 &
nohup scrapy crawl urlMonitor -a slice=9 &
nohup scrapy crawl urlMonitor -a slice=10 &
nohup scrapy crawl urlMonitor -a slice=11 &
nohup scrapy crawl urlMonitor -a slice=12 &
nohup scrapy crawl urlMonitor -a slice=13 &
nohup scrapy crawl urlMonitor -a slice=14 &
nohup scrapy crawl urlMonitor -a slice=15 &
nohup scrapy crawl urlMonitor -a slice=16 &
nohup scrapy crawl urlMonitor -a slice=17 &
nohup scrapy crawl urlMonitor -a slice=18 &
nohup scrapy crawl urlMonitor -a slice=19 &
nohup scrapy crawl urlMonitor -a slice=20 &
nohup scrapy crawl urlMonitor -a slice=21 &
nohup scrapy crawl urlMonitor -a slice=22 &
nohup scrapy crawl urlMonitor -a slice=23 &
nohup scrapy crawl urlMonitor -a slice=24 &
nohup scrapy crawl urlMonitor -a slice=25 &
nohup scrapy crawl urlMonitor -a slice=26 &
nohup scrapy crawl urlMonitor -a slice=27 &
nohup scrapy crawl urlMonitor -a slice=28 &
nohup scrapy crawl urlMonitor -a slice=29 &
nohup scrapy crawl urlMonitor -a slice=30 &
nohup scrapy crawl urlMonitor -a slice=31 &
nohup scrapy crawl urlMonitor -a slice=32 &
nohup scrapy crawl urlMonitor -a slice=33 &
nohup scrapy crawl urlMonitor -a slice=34 &
nohup scrapy crawl urlMonitor -a slice=35 &
nohup scrapy crawl urlMonitor -a slice=36 &
nohup scrapy crawl urlMonitor -a slice=37 &
nohup scrapy crawl urlMonitor -a slice=38 &
source to share
If you have GNU Parallel, you can do this:
parallel -j4 scrapy crawl urlMonitor -a slice={} ::: {1..38}
GNU Parallel is a general parallelizer that makes it easy to run jobs in parallel on a single computer, or across multiple computers that you have ssh access to.
If you have 32 different jobs that you want to run on 4 processors, the direct way to parallelize is to run 8 jobs for each processor:
GNU Parallel instead starts a new process when it ends - keeping active processors and thus saving time:
Installation
If GNU Parallel is not packaged for your distribution, you can do a personal installation that does not require root access. This can be done in 10 seconds by following these steps:
(wget -O - pi.dk/3 || curl pi.dk/3/ || fetch -o - http://pi.dk/3) | bash
For other installation options see http://git.savannah.gnu.org/cgit/parallel.git/tree/README
More details
More examples: http://www.gnu.org/software/parallel/man.html
Watch videos: https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1
Go through the tutorial: http://www.gnu.org/software/parallel/parallel_tutorial.html
Subscribe to an email list for support: https://lists.gnu.org/mailman/listinfo/parallel
source to share