How to parallelize a bash script containing nested for loops on a huge dataset with GNU parallel?

I am actually running a lot of IP connections thanks to OpenSSL to determine if the server is accepting or not specified cipher. I am running this script on 1,000,000 servers (which are contained in "listeIpShuffle.txt"). So my script contains 2 for the loop: the first is used to get every line of my file containing the IP addresses, and the second is used to test every encryption available in my version of OpenSSL if the server accepts or rejects it.

I saw on the parallel GNU doc that it is possible to parallelize this type of loop:

  *(for x in `cat xlist` ; do
    for y in `cat ylist` ; do
      do_something $x $y
    done
  done) | process_output*

      

... can be written like this:

*parallel do_something {1} {2} :::: xlist ylist | process_output*

      

But I am not trying to apply this on my script ... Indeed, my file containing my IPs is too large and I am getting the known "Too many arguments" error ... How can I deal with this problem and make my script parallel?

Thanks in advance!

Here's my script:

#!/usr/bin/env bash

#on recupere toutes les ciphers d'openssl
ciphers=$(openssl ciphers 'ALL:eNULL' | sed -e 's/:/ /g')
fichier="./serveursMail/listeIpShuffle.txt"
port=":25"
>resultDeprecatedCipher.txt
nbInconnu=0
echo Cipher list de $(openssl version).

for ligne in $(<$fichier)
do
    ligneIp=$(echo $ligne | tr "|" "\n")    
    ip=($ligneIp)
    ipPort=$ip$port
    dns=$(echo $ligneIp |cut -f 2 -d ' ')


    ciphers=$(openssl ciphers 'ALL:eNULL' | sed -e 's/:/ /g')

    for cipher in ${ciphers[@]}
    do
        if [[ $nbInconnu < 4 ]] ; then
            echo -n Test $ligneIp " : " $cipher...

            result=$(echo -n | timeout 10s openssl s_client -starttls smtp -cipher "$cipher" -connect $ipPort -servername $dns 2>&1) #pas de reponse apres dasn les 15sec => FAIL

            if [[ "$result" =~ ":error:" ]] ; then
            error=$(echo -n $result | cut -d':' -f6)
            echo NON \($error\)
            let "nbInconnu=0"
            else
                if [[ "$result" =~ "Cipher is ${cipher}" || "$result" =~ "Cipher    :" ]] ; then
                    echo OUI
                    let "nbInconnu=0"
                    echo $ligneIp " : " $cipher >> resultDeprecatedCipher.txt
                else
                    echo REPONSE INCONNUE
                    let "nbInconnu++" #incrementation
                    echo $nbInconnu
                    echo $result
                fi
            fi
        else
          let "nbInconnu=0"
          break
        fi
    done 
done

      

+3


source to share


1 answer


echo alt3.gmail-smtp-in.l.google.com > serverlist
echo fo-ds-ats.member.g02.yahoodns.net >> serverlist

doit() {
    ip="$1"
    port="$2"
    cipher="$3"
    openssl s_client -starttls smtp -cipher "$cipher" -connect $ip:$port -servername $ip < /dev/null
}
export -f doit
parallel --tag --timeout 10 --retries 4 doit $1 :::: serverlist ::: 25 ::: $(openssl ciphers 'ALL:eNULL' | sed -e 's/:/ /g') >tmp_results
# Post process tmp_results as needed

      



To avoid overloading one server, add --shuf

. To run as much in parallel as possible, add -j0

.

+1


source







All Articles