GNU Parallel: distribute files from one source to remote hosts while distributing destination files
Scenario: S3 bucket has 1000 files. I have two cars. Each of these machines has two disks / dev / sda and / dev / sdb. Limitations: No disc can hold all 1000 files. And no machine can hold all 1000 files. Desired result: Spread 1000 files on 4 disks on two machines using GNU parallel.
I've tried things like:
parallel --xapply --joblog out.txt -S:, R echo {1} {2} ::: "/ dev / sda" "/ dev / sdb" ::: {0..10}
But I am getting:
Seq Host Starttime JobRuntime Send Receive Exitval Signal Command 2: 1414040436.607 0.037 0 0 0 0 echo / dev / sda 1 4: 1414040436.615 0.030 0 0 0 0 echo / dev / sda 3 6: 1414040436.623 0.024 0 0 0 0 echo / dev / sda 5 8: 1414040436.632 0.015 0 0 0 0 echo / dev / sda 7 10: 1414040436.640 0.006 0 0 0 0 echo / dev / sda 9 1 R 1414040436.603 0.088 0 0 0 0 echo / dev / sdb 0 3 R 1414040436.611 0.092 0 0 0 0 echo / dev / sdb 2 5 R 1414040436.619 0.095 0 0 0 0 echo / dev / sdb 4 7 R 1414040436.628 0.095 0 0 0 0 echo / dev / sdb 6 9 R 1414040436.636 0.096 0 0 0 0 echo / dev / sdb 8 11 R 1414040436.645 0.094 0 0 0 0 echo / dev / sdb 10
Where "R" is the IP address of the remote host. How do I distribute files (I have all the names in the file) from S3 to 4 drives? Thank.
source to share
GNU Parallel is good for starting a new job when the old one has finished: it divides jobs into servers on the fly, rather than ahead of time.
What you are looking for is a way to do it ahead of time.
Your --xapply approach seems sound, but you need to force GNU Parallel to distribute hosts evenly. Your current approach depends on how quickly each host finishes, and it won't work at all.
So something like:
parallel echo {1}//{2} ::: sda sdb ::: server1 server2 | parallel --colsep '//' --xapply echo copy {3} to {1} on {2} :::: - filenames.txt
Or:
parallel --xapply echo copy {3} to {1} on {2} ::: sda sda sdb sdb ::: server1 server2 server1 server2 :::: filenames.txt
source to share