How to parallelize a bash script containing nested for loops on a huge dataset with GNU parallel?
I am actually running a lot of IP connections thanks to OpenSSL to determine if the server is accepting or not specified cipher. I am running this script on 1,000,000 servers (which are contained in "listeIpShuffle.txt"). So my script contains 2 for the loop: the first is used to get every line of my file containing the IP addresses, and the second is used to test every encryption available in my version of OpenSSL if the server accepts or rejects it.
I saw on the parallel GNU doc that it is possible to parallelize this type of loop:
*(for x in `cat xlist` ; do
for y in `cat ylist` ; do
do_something $x $y
done
done) | process_output*
... can be written like this:
*parallel do_something {1} {2} :::: xlist ylist | process_output*
But I am not trying to apply this on my script ... Indeed, my file containing my IPs is too large and I am getting the known "Too many arguments" error ... How can I deal with this problem and make my script parallel?
Thanks in advance!
Here's my script:
#!/usr/bin/env bash
#on recupere toutes les ciphers d'openssl
ciphers=$(openssl ciphers 'ALL:eNULL' | sed -e 's/:/ /g')
fichier="./serveursMail/listeIpShuffle.txt"
port=":25"
>resultDeprecatedCipher.txt
nbInconnu=0
echo Cipher list de $(openssl version).
for ligne in $(<$fichier)
do
ligneIp=$(echo $ligne | tr "|" "\n")
ip=($ligneIp)
ipPort=$ip$port
dns=$(echo $ligneIp |cut -f 2 -d ' ')
ciphers=$(openssl ciphers 'ALL:eNULL' | sed -e 's/:/ /g')
for cipher in ${ciphers[@]}
do
if [[ $nbInconnu < 4 ]] ; then
echo -n Test $ligneIp " : " $cipher...
result=$(echo -n | timeout 10s openssl s_client -starttls smtp -cipher "$cipher" -connect $ipPort -servername $dns 2>&1) #pas de reponse apres dasn les 15sec => FAIL
if [[ "$result" =~ ":error:" ]] ; then
error=$(echo -n $result | cut -d':' -f6)
echo NON \($error\)
let "nbInconnu=0"
else
if [[ "$result" =~ "Cipher is ${cipher}" || "$result" =~ "Cipher :" ]] ; then
echo OUI
let "nbInconnu=0"
echo $ligneIp " : " $cipher >> resultDeprecatedCipher.txt
else
echo REPONSE INCONNUE
let "nbInconnu++" #incrementation
echo $nbInconnu
echo $result
fi
fi
else
let "nbInconnu=0"
break
fi
done
done
source to share
echo alt3.gmail-smtp-in.l.google.com > serverlist
echo fo-ds-ats.member.g02.yahoodns.net >> serverlist
doit() {
ip="$1"
port="$2"
cipher="$3"
openssl s_client -starttls smtp -cipher "$cipher" -connect $ip:$port -servername $ip < /dev/null
}
export -f doit
parallel --tag --timeout 10 --retries 4 doit $1 :::: serverlist ::: 25 ::: $(openssl ciphers 'ALL:eNULL' | sed -e 's/:/ /g') >tmp_results
# Post process tmp_results as needed
To avoid overloading one server, add --shuf
. To run as much in parallel as possible, add -j0
.
source to share