How do .Popen subprocesses work in Python?
I am fully aware of the existence of related questions, but they are all very fuzzy and cannot clearly explain what happens at each stage of the path. The examples provided are often untested and do not provide information on how to adapt them to different scenarios. Here are the questions as well as the Python documentation:
- How can I use subprocess.Popen to connect multiple processes through pipes?
- associate multiple Popen commands with channels
- https://docs.python.org/2/library/subprocess.html#popen-objects
This is essentially what I am trying to achieve, but in Python:
curl http://asdf.com/89asdf.gif | convert -resize 80x80 - - | icat -k -
Here's what I have after hours of frankensteining together the bits and pieces of the above answers:
import requests
import subprocess
from subprocess import Popen, PIPE, STDOUT
url = 'http://asdf.com/89asdf.gif'
img = requests.get(url)
p1 = subprocess.Popen(['convert', '-resize', '80x80', '-', '-'], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
p2 = subprocess.Popen(['icat', '-k', '-'], stdin=p1.stdout, stdout=subprocess.PIPE)
p1.communicate(img.content)
print p2.stdout
Here's my code again, this time improved based on your answers:
import requests
from subprocess import Popen, PIPE, STDOUT
url = 'http://asdf.com/89asdf.gif'
img = requests.get(url)
p1 = Popen(['convert', '-resize', '80x80', '-', '-'], stdin=PIPE, stdout=PIPE)
p2 = Popen(['icat', '-k', '-'], stdin=p1.stdout, stdout=PIPE)
p1.stdin.write(img.content)
p1.stdin.close()
p1.stdout.close()
output = p2.communicate()[0]
print output
Note: icat
Displays images in terminals with 256 colors. I have already managed to successfully print my result in another question . A typical image looks like this in a terminal, once rendered icat
:
Please correct me where I am wrong, but this is my real understanding:
-
p1.communicate(img.content)
: setsp1
STDIN. -
p1
STDIN -subprocess.PIPE
that providesp1.communicate(mg.content)
. -
p2
STDIN is thep1
STDOUT whose image has been modifiedconvert
. - Then I
print
p2
STDOUT which should be an image with the ASCII colors providedicat
.
Questions:
- I've seen both
stdin=subprocess.PIPE
andstdin=PIPE
. What is the difference? - Why is the first STDIN provided after defining other subprocesses?
Can someone please explain what is actually going on in the process and what is wrong with my code? This whole pipeline business should be simpler than it sounds, and I would like to understand how it works.
source to share
Q1. Either way subprocess.PIPE
or just PIPE
they refer to the same symbol that is PIPE
from the module subprocess
. The following values are identical:
# Version 1
import subprocess
proc = subprocess.Popen(['ls'], stdout=subprocess.PIPE)
output = proc.communicate()[0]
# Version 2
from subprocess import PIPE, Popen
proc = Popen(['ls'], stdout=PIPE)
output = proc.communicate()[0]
Q2. What does it really do communicate()
is send the input to the p1 STDIN stream. While subprocesses are indeed alive and running when constructors are called Popen
, in your particular case, the utility convert
seems to do nothing until it actually receives the content via STDIN. If it was a less interactive command (for example, ls
for example), it did not wait until communicate()
it did nothing.
UPDATED CONTENT In your case, instead of using communicate
to send input to p1, try this instead:
p1.stdin.write(img.content)
p1.stdin.close()
p1.stdout.close() # Protect against the busted pipe condition when p2 finishes before p1
output = p2.communicate()[0]
print output
See if you have more luck with this.
source to share
Q1. it makes no difference, both of them subprocess.PIPE
...
Q2. you don't need to do it the way you could just as easily do
p1.stdin.write(img.content)
(in fact, that's exactly what this part of communication does ....)
communicate
is used because it blocks until the process exits (in which case its stdout can be made available to p2.stdin (maybe?)) where as write
just writes to the pipe and goes to the next python line in the file
source to share