Why does Popen.stdout only contain part of the output?
I am executing two processes at the same time in python using the subprocess module:
p_topic = subprocess.Popen(['rostopic','echo','/msg/address'], stdout=PIPE)
p_play = subprocess.Popen(['rosbag','play',bagfile_path])
This is ROS : p_topic
listens on a file .bag
for playback and outputs certain information from that file .bag
to the stdout stream; Then I want to access this output using an object p_topic.stdout
(which behaves like a file).
However, what I find is that the object p_topic.stdout
only contains the first ~ 1/3 of the output lines it should have, that is, compared to two manual commands, simultaneously in two shells side.
I tried to wait many seconds for the output to complete, but that doesn't change anything, roughly the same ratio of the lines captured p_topic.stdout
each time. Any hints what this might be would be greatly appreciated!
EDIT:
Here's the reading code:
#wait for playing to stop
while p_play.poll() == None:
time.sleep(.1)
time.sleep(X)#wait for some time for the p_topic to finish
p_topic.terminate()
output=[]
for line in p_topic.stdout:
output.append(line)
Note that the X value in time.sleep (X) is irrelevant
source to share
By default, when a process is stdout
not connected to a terminal, output is blocked by a block. When connected to a terminal, it is buffered. You expect to get full lines, but you can't if rostopic
unbuffered or explicitly build buffers of your own stdout
(if it's a C program, you can use setvbuf
to do this automatically).
Another (possibly overlapping) possibility is that the buffer buffer itself fills up (pipe buffers are usually quite small), and since you never drain it, rostopic
fills the buffer buffer and then blocks indefinitely until you kill it, leaving only what was put into the pipe, which will be drained during the reading process stdout
. In this case, you either need to create a thread to pick up the pipe with Python, or use your main thread select
for the module components to monitor and drain the pipe (mixed with polling another process). Streaming is generally simpler, although you need to be careful to avoid thread safety issues.
source to share
Should I try to exchange data / wait? instead of sleeping and that will solve your problem?
I have it for general purpose, so not sure if you can take that and change it to whatever you need?
executable_Params = "{0} {1} {2} {3} {4}".format(my_Binary,
arg1,
arg2,
arg3,
arg4)
# execute the process
process = subprocess.Popen(shlex.split(executable_Params),
shell=False,
stderr=subprocess.PIPE,
stdout=subprocess.PIPE)
stdout, stderr = process.communicate()
ret_code = process.wait()
if ret_code == 0:
return 0
else:
#get the correct message from my enum method
error_msg = Process_Error_Codes(ret_code).name
raise subprocess.CalledProcessError(returncode=ret_code,
cmd=executable_Params)
source to share