Ruby-Open3.popen3 / how to print output
I have a little ruby script that imports mysql
like this: mysql -u <user> -p<pass> -h <host> <db> < file.sql
but uses Open3.popen3
for this. This is what I have so far:
mysqlimp = "mysql -u #{mysqllocal['user']} "
mysqlimp << "-h #{mysqllocal['host']} "
mysqlimp << "-p#{mysqllocal['pass']} "
mysqlimp << "#{mysqllocal['db']}"
Open3.popen3(mysqlimp) do |stdin, stdout, stderr, wthr|
stdin.write "DROP DATABASE IF EXISTS #{mysqllocal['db']};\n"
stdin.write "CREATE DATABASE #{mysqllocal['db']};\n"
stdin.write "USE #{mysqllocal['db']};\n"
stdin.write mysqldump #a string containing the database data
stdin.close
stdout.each_line { |line| puts line }
stdout.close
stderr.each_line { |line| puts line }
stderr.close
end
Job actually does this, but there is one thing that worries me about the conclusion I would like to see.
If I change the first line to:
mysqlimp = "mysql -v -u #{mysqllocal['user']} " #note the -v
then the whole script hangs forever.
I am assuming this is due to the read and write thread blocking each other, and I am also assuming that it stdout
needs to be flushed regularly to stdin
keep being consumed. In other words, as long as the buffer is stdout
full, the process will wait until it turns red, but since this is executed at the very bottom of the first, this will never happen.
Hope someone can test my theory? How can I write code that outputs everything from stdout
and writes everything to stdin
?
Thanks go ahead!
source to share
- Since you're only on stdout, you can just use
Open3#popen2e
that brings togetherstdout
andstderr
in one stream. - To write lines ending with a line of string to a stream, you can use
puts
, as in$stdout
, in a simple hello program. - You must use
waith_thread.join
orwait_thread.value
to wait for the child process to complete. - In any case, you will need to start a separate stream to read from the stream if you want to see the results immediately.
Example:
require 'open3'
cmd = 'sh'
Open3.popen2e(cmd) do |stdin, stdout_stderr, wait_thread|
Thread.new do
stdout_stderr.each {|l| puts l }
end
stdin.puts 'ls'
stdin.close
wait_thread.value
end
Your code is fixed:
require 'open3'
mysqldump = # ...
mysqlimp = "mysql -u #{mysqllocal['user']} "
mysqlimp << "-h #{mysqllocal['host']} "
mysqlimp << "-p#{mysqllocal['pass']} "
mysqlimp << "#{mysqllocal['db']}"
Open3.popen2e(mysqlimp) do |stdin, stdout_stderr, wait_thread|
Thread.new do
stdout_stderr.each {|l| puts l }
end
stdin.puts "DROP DATABASE IF EXISTS #{mysqllocal['db']};"
stdin.puts "CREATE DATABASE #{mysqllocal['db']};"
stdin.puts "USE #{mysqllocal['db']};"
stdin.close
wait_thread.value
end
source to share
Whenever you start a process from the command line or via fork
, the process inherits stdin, stdout and stderr from the process's fathers. This means that if your command line is running on a terminal, then the stdin, stdout and stderr of the new process are connected to the terminal.
Open3.popen3
, on the other hand, doesn't bind stdin, stdout and stderr to the terminal because you don't want direct user interaction. Therefore, we need something else.
For stdin, we need something with two capabilities:
- The parent process requires something to insert data that the subprocess needs to receive when it reads from stdin.
- The subprocess needs to offer a function
read
like stdin.
For stdout and stderr, we need something like this:
- The sub-process needs something to write.
puts
andprint
must contain data that the father's process should read. - The dad needs something that offers a function
read
to get the stdout and stderr data of a subprocess.
This means that for stdin, stdout and stderr, we need three queues (FIFOs) for communication between the father process and the sub-process. These queues should act a bit like files, as they should provide read
, write
(for puts
and print
), close
and select
(is data available?). Thus, both Linux and Windows provide anonymous pipes . This is one of the traditional (local) mechanisms of interprocess communication. And, well, Open3.popen3
really wants to communicate between two different processes. This is why it Open3.popen3
links stdin, stdout, and stderr to anonymous pipes.
Each pipe, whether anonymous or named, has a buffer of limited size. This size depends on the operating system. Trap: If the buffer is full and processes try to write to the pipe, the operating system pauses the process until other processes have read from the pipe.
This could be your problem:
- You keep feeding data to your subprocess, but you don't read what your subprocess is writing to stdout.
- Hence, the output of our subprocess continues to accumulate in the buffer until the buffer is full.
- This is when the operating system suspends your subprocess (
puts
orprint
). - Now you can transfer data to an anonymous pipe that is connected to the stdin of your subprocesses until too much stdin data accumulates. The stdin pipe buffer is full. Then the operating system suspends the parent processes (
stdin.write
will be blocked).
I advise you to use Open3.capture2e
or similar wrapper around Open3.popen3
. You can pass data to a subprocess with a keyword argument :stdin_data
.
If you insist on communicating with your subprocess "interactively", you need to learn about IO.select
or use multithreading. Both are quite complex. Better to use Open3.capture*
.
source to share