Bash parameter expansion, indirection and background

After trying this problem for a few hours and searching here and unable to find a suitable solution, it's time to ask:

In bash (4.3), I am trying to make a combination of the following:

  • Create array
  • Loop the array values โ€‹โ€‹with a command that isn't super fast (curl to the webserver to get the value), so we loop each loop to parallelize everything to speed it up.
  • Set the names of the values โ€‹โ€‹in the array to the variables assigned to the values โ€‹โ€‹redirected to it from the command via "read"
  • Commit each loop and get their PIDs into a regular array and concatenate each PID with the corresponding array value in the associative array, so I have key pairs = array value value for PID
  • Use "wait" to wait for each PID to complete 0 or throw an error telling us which name (s) of name in the array failed to complete with 0 by referencing the associative array
  • I need to export all VAR names in the original array and their time-bound values โ€‹โ€‹(from curl command results) because I am using this script from another bash script that will use the resulting exported VARs / values.

The reason why I use "read" instead of "export" with "export var = $ (command)" or similar is that when I use "PID" to use "wait" in next for loop, I am on in fact (incorrectly) I get the PID of the "export" command which always goes out of 0, so I don't see the error. When I use redirect read to set the VAR value (from the name in the array) and background, it actually gets the PID of the command, and I will catch any errors in the next loop with a "wait" command.

So it basically works, but I realized that the "read" command does not actually replace the variable with the value of the array name properly, so the redirected command sends its output to that name to set the replaced VAR to the value. Or maybe the command is just wrong, so I am incorrectly redirecting the result of my command to the VAR name I am trying to set.

What's it worth when I run curl | python by hand (to pull the value and then parse the JSON output), it definitely succeeds, so I know I'm working, I just can't get a redirect to send the resulting result into a VAR name.

Here's an example of what I'm trying to do: In the parent script:

# Source the child script that has the functions I need
source functions.sh

# Create the array
VALUES=(
VALUE_A
VALUE_B
VALUE_C
)

# Call the function sourced from the script above, which will use the above defined array
function_getvalues

      

In the child (sourced) script:

function_getvalues()
{
  curl_pids=( )
  declare -A value_pids
  for value in "${VALUES[@]}"; do
    read ${value} < <(curl -f -s -X GET http://path/to/json/value | python3 -c "import sys, json; print(json.load(sys.stdin)['data']['value'])") & curl_pids+=( $! ) value_pids+=([$!]=${value})
  done
  for pid in "${curl_pids[@]}"; do
    wait "$pid" && echo "Successfully retrieved value ${value_pids[$pid]} from Webserver." || { echo "Something went wrong retrieving value ${value_pids[$pid]}, so we couldn't get the output data needed from Webserver. Exiting." ; exit 1 ; }
  done
}

      

+3


source to share


1 answer


The problem is that read

when run in the background it is not connected to the standard. [details] Consider this simplified working example, with a comment on how to screw it up:

VALUES=( VALUE_A VALUE_B )
for value in "${VALUES[@]}"; do
    read ${value} < <(echo ${RANDOM}) # add "&" and it stops working
done
echo "VALUE_A=${VALUE_A}"
echo "VALUE_B=${VALUE_B}"

      

You might be able to do this with coproc

or with the read -u

automatic allocation of the file descriptor
, but this is actually a job for temporary files:



tmpdir=$(mktemp -d)

VALUES=( VALUE_A VALUE_B )
for value in "${VALUES[@]}"; do
    (sleep 1; echo ${RANDOM} > "${tmpdir}"/"${value}") &
done
for value in "${VALUES[@]}"; do
    wait_file "${tmpdir}"/"${value}" && {
        read -r ${value} < "${tmpdir}"/"${value}";
    }
done
echo "VALUE_A=${VALUE_A}"
echo "VALUE_B=${VALUE_B}"

rm -r "${tmpdir}"

      

This example uses a wait_file

helper
, but you can use inotifywait

if you don't mind some OS dependencies.

+1


source







All Articles