How do I write data to stdin of the first process in a Python shell pipeline?
I see that this piece of code gets mentioned a lot during discussions within Python subprocesses. Mandatory link: https://docs.python.org/3.4/library/subprocess.html#replacing-shell-pipeline
Slightly changed:
p1 = subprocess.Popen(['cat'],
stdin=subprocess.PIPE,
stdout=subprocess.PIPE)
p2 = subprocess.Popen(['head', '-n', '1'],
stdin=p1.stdout,
stdout=subprocess.PIPE)
# Allow p1 to receive a SIGPIPE if p2 exits.
p1.stdout.close()
output = p2.communicate()[0]
This shell pipeline is pointless except to briefly demonstrate the problem. Entrance "abc\ndef\nghi\n"
and only "abc\n"
needs to be fixed in output
.
What is the best way to write data to p1.stdin
? I know the argument input
for subprocess.Popen.communicate()
, but it won't work in a pipeline. In addition, the solution must handle the lock correctly.
My guess: Reverse code engineer communicate()
and create a different version for this particular problem. Before I do that, I want to ask if there is an easier solution that I am not aware of.
source to share
Write to p1.stdin
and then close it before calling p2.communicate()
:
In [1]: import subprocess
In [2]: %cpaste
Pasting code; enter '--' alone on the line to stop or use Ctrl-D.
:p1 = subprocess.Popen(['cat'],
: stdin=subprocess.PIPE,
: stdout=subprocess.PIPE)
:p2 = subprocess.Popen(['head', '-n', '1'],
: stdin=p1.stdout,
: stdout=subprocess.PIPE)
:p1.stdout.close()
:--
In [3]: p1.stdin.write(b'This is the first line.\n')
Out[3]: 24
In [4]: p1.stdin.write(b'And here is the second line.\n')
Out[4]: 29
In [5]: p1.stdin.close()
In [6]: p2.communicate()
Out[6]: (b'This is the first line.\n', None)
(Don't forget newlines in the data posted to cat
, or they won't work.)
source to share
As a working example:
import subprocess, threading
# Unmodified from original code
p1 = subprocess.Popen(['cat'],
stdin=subprocess.PIPE,
stdout=subprocess.PIPE)
p2 = subprocess.Popen(['head', '-n', '1'],
stdin=p1.stdout,
stdout=subprocess.PIPE)
# New code: Start a thread that writes stdin, and closes it when complete
def write_stdin():
p1.stdin.write("abc\ndef\nghi\n")
p1.stdin.close()
write_t = threading.Thread(target = write_stdin)
write_t.start()
# Unmodified from original code
p1.stdout.close()
output = p2.communicate()[0]
source to share