Twisted, how can ProcessProtocol receive stdout without buffering?
I am using an external process that writes a short line of output for each chunk of data being processed. I would like to react after each of these lines without further delay. However, it seems that .outReceived()
of is ProcessProtocol
buffered. Status of documents:
.outReceived (data): Called with data received from process' stdout pipe. Pipes tend to provide data in larger chunks than sockets (one kilobyte is the total buffer size) , so you may not experience the "random dribbles and meshes" behavior typical of network sockets, but whatever you need to be ready to go unless you get all the data in one go. To do this correctly, outReceived should just accumulate data and defer nothing with it until the process finishes.
The result is that I get the output in one chunk after all processing is complete. How can I force ProcessProtocol
stdout to not buffer?
source to share
I am using an external process that writes a short line of output for each chunk of data being processed. I would like to react after each of these lines without further delay.
The result is that I get the output in one chunk after all processing is complete. How do I force ForceProtocol to disable stdout?
Buffering occurs in the producer's process, not in the consumer's. The C standard library is stdout
only buffered when connected to a terminal, otherwise it is fully buffered. This is what causes the producer process to output data in large chunks rather than line by line when not connected to a terminal.
Use stdbuf to force a producer process extension stdout
.
If the producer process is a python script, use the -u
python interpreter interpreter to disable standard stream buffering entirely. stdbuf
more useful though.
source to share