Problem with subprocess.check_output ()
I am having some weird problems using subprocess.check_output()
. At first I just used subprocess.call()
and everything worked fine. However, when I just turn off call()
for check_output()
, I get a strange error.
Before the code (works great):
def execute(hosts):
''' Using psexec, execute the batch script on the list of hosts '''
successes = []
wd = r'c:\\'
file = r'c:\\script.exe'
for host in hosts:
res = subprocess.call(shlex.split(r'psexec \\\\%s -e -s -d -w %s %s' % (host,wd,file)))
if res.... # Want to check the output here
successes.append(host)
return successes
After code (doesn't work):
def execute(hosts):
''' Using psexec, execute the batch script on the list of hosts '''
successes = []
wd = r'c:\\'
file = r'c:\\script.exe'
for host in hosts:
res = subprocess.check_output(shlex.split(r'psexec \\\\%s -e -s -d -w %s %s' % (host,wd,file)))
if res.... # Want to check the output here
successes.append(host)
return successes
This gives an error:
I couldn't redirect this because the program is hanging here and I can't ctrl-c. Any idea why this is happening? What's the difference between subprocess.call () and check_output () that might be causing this?
Here's some additional code to include some of the multiprocessing:
PROCESSES = 2 host_sublists_execute = [.... list of hosts ... ] poolE = multiprocessing.Pool(processes=PROCESSES) success_executions = poolE.map(execute,host_sublists_execute) success_executions = [entry for sub in success_executions for entry in sub] poolE.close() poolE.join()
Thank!
source to share
You are facing Python Issue 9400 .
There is a key difference, you need to understand about subprocess.call()
vs subprocess.check_output()
. subprocess.call()
will execute the command you give it, then provide the return code. On the other hand, it subprocess.check_output()
returns you the program output in a string, but it tries to do you a favor and checks the return code of the program and raises an exception ( subprocess.CalledProcessError
) if the program was not successfully executed (returns a non-zero return code).
When you call pool.map()
with a multiprocessing pool, it will try to propagate exceptions in subprocesses to the main one and raise the exception there. There seems to be a problem with how the exception class is defined subprocess.CalledProcessError
, so the pickling fails when the multiprocessing library tries to propagate the exception to the main one.
The program you are calling returns a nonzero return code, which makes it raise an subprocess.check_output()
exception rather pool.map()
than handle it properly, which is why you end up TypeError
with a failed attempt to retrieve the exception.
As a side note, the definition subprocess.CalledProcessError
should be really confusing because if I open my 2.7.6 terminal, the import sub-process and explicitly raise the error, I still get TypeError
, so I don't think it's just an etching problem.
source to share