How to run multiple curl commands in parallel over https
I am trying to make multiple requests to a server running on localhost in parallel from the command line using curl
but I am having a hard time getting them to work in parallel. One easy way to accomplish this is with something like the following:
for ((i = 0; i < 10; i++))
do
time curl -k https://localhost:$PORT/fooBar &
done
However, the result was the following:
"my output"
real 0m7.555s
user 0m0.007s
sys 0m0.002s
"my output"
real 0m14.628s
user 0m0.007s
sys 0m0.002s
"my output"
real 0m21.705s
user 0m0.007s
sys 0m0.002s
"my output"
...
This is despite the fact that the server is multithreaded and does not require locks (I know the responders will be suspicious of this, but consider the next option I tried). Since this came up post requests one at a time, I tried the following:
$ time curl -k https://localhost:$PORT/fooBar &
[1] 6780
$ time curl -k https://localhost:$PORT/fooBar &
[2] 6803
$ curl: (35) Unknown SSL protocol error in connection to localhost:<my port>
real 0m1.030s
user 0m0.003s
sys 0m0.002s
"my output"
real 0m7.128s
user 0m0.007s
sys 0m0.002s
Any ideas why these requests seem to be serialized when executed in a loop for
with the background, and why an SSL error occurs when I just use background lines directly from the command line?
EDIT: After further research, it seems that the lack of parallelism is due to the server using scala spray, and it seems that the requests are being handled by the actor in the same thread, even though the handler for the endpoints is flagged blocking
. This observation divides this question into two parts. The one related to bash
and curl
is that the loop for
and multiple one-liners of the shell give different results (one throws an SSL error). Eliminating the lack of concurrency on the server side seems to be a separate issue.
Unfortunately I'm not sure how to reproduce the problem all over the place since this is a private server, but I hope someone might have an idea why for loop and one-liners might give different results.
source to share
Just running your processes in the background is easy and really tempting, but unless you are iterating over a small set, you wouldn't be doing it. You can run CPU and / or Network as you have no control over the number of simultaneous boot processes.
I would suggest using xargs
.
CONCURRENT_PROCESSES=10
seq 1 9 |
xargs -P ${CONCURRENT_PROCESSES} \
--replace time curl -k https://localhost:{}/fooBar
xargs
--replace
automatically sets the number of arguments for each command call to 1 ( -n 1
) and replaces with {}
each argument specified in stdin.
If you need a more complex script (like logging the time of each individual request), you can use a wrapp script instead.
source to share