Clever Geek Handbook

How does subprocess.call () work with shell = False?

I am using a Python module subprocess

to call some Linux command line functions. The documentation explains the argument shell=True

as

If a shell True

, the specified command will be executed through the shell

There are two examples that seem the same to me from a descriptive point of view (i.e. they both invoke some command line command), but one is using shell=True

and the other is not

>>> subprocess.call(["ls", "-l"])
0

>>> subprocess.call("exit 1", shell=True)
1

My question is:

  • What does the command shell=False

    do with do, as opposed to shell=True

    ?
  • I got the impression that subprocess.call

    and check_call

    and check_output

    all must fulfill the argument through the shell. In other words, how can he not execute the argument through the shell?

It would also be helpful to get some examples:

  • Things that can be done with shell=True

    , that cannot be done with, shell=False

    and why they cannot be done.
  • And vice versa (although it seems that there are no such examples)
  • Things that don't matter, shell=True

    or False

    why it doesn't matter.
+3


source to share


2 answers


UNIX programs run against each other with the following three calls or derivatives / equivalents:

  • fork()

    - Create a new copy of yourself.
  • exec()

    - Replace yourself with another program (do this if you copy!).
  • wait()

    - Wait for another process to finish (optional if not running in the background).

So with help shell=False

you do exactly that (as the Python syntax pseudocode below - exclude wait()

if not call blocking like subprocess.call()

):

pid = fork()
if pid == 0: # we're the child process, not the parent
  execlp("ls", "ls", "-l", NUL);
else:
  retval = wait(pid) # we're the parent; wait for the child to exit & get its exit status

whereas shell=True

you do the following:

pid = fork()
if pid == 0:
  execlp("sh", "sh", "-c", "ls -l", NUL);
else:
  retval = wait(pid)

Note that with the shell=False

command we ran was ls

, whereas with the shell=True

command we ran was sh

.



I.e:

subprocess.Popen(foo, shell=True)

exactly the same as:

subprocess.Popen(
  ["sh", "-c"] + ([foo] if isinstance(foo, basestring) else foo),
  shell=False)

That is, you do a copy /bin/sh

and pipe that copy /bin/sh

to parse the string into the argument list and do it yourself ls -l

.



So why did you use shell=True

?

  • You are calling the inline wrapper.

    For example, the command is exit

    actually part of the shell itself, not an external command. However, this is a fairly small set of commands and is rarely useful in the context of a shell instance that only exists for the duration of a single call subprocess.call()

    .

  • You have code with wrapper constructs (i.e. redirects) that you can emulate without any problem.

    If, for example, your command cat one two >three

    , the syntax >three

    is a redirect: it is not an argument for cat

    , but an instruction for the shell to set stdout=open('three', 'w')

    when the command is run ['cat', 'one', 'two']

    . If you don't want to deal with redirects and pipelines yourself, you need a wrapper to do it.

    The case is somewhat more complicated cat foo bar | baz

    . To do this without the shell, you need to run both sides of the pipeline on their own: p1 = Popen(['cat', 'foo', 'bar'], stdout=PIPE), p2=Popen(['baz'], stdin=p1.stdout)

    .

  • You don't give a damn about security bugs.

    ... okay, it's a little too strong, but not strong. Use is shell=True

    dangerous. You can't do this: Popen('cat -- %s' % (filename,), shell=True)

    without the shell vulnerability: if your code was ever called with filename

    containing $(rm -rf ~)

    , you would have a very bad day. On the other hand, it ['cat', '--', filename]

    is safe with all possible filenames: the filename is purely data, not parsed as source by the shell or anything else.

    Safe scripts can be written in the shell, but you need to be careful. Consider the following:

    filenames = ['file1', 'file2'] # these can be user-provided
subprocess.Popen(['cat -- "$@" | baz', '_'] + filenames, shell=True)

    This code is safe (well - as safe as letting the user read any file they ever want) because it passes your out-of-range names from your script code, but it is only safe because the string passed to the shell , is fixed and hardcoded, and the parameterized content is external variables (list filenames

    ). And even then it is only "safe" to the point - a bug like Shellshock that triggers the initialization of the shell will affect it as much as anything else.

+5


source


I was under the impression that subprocess.call and check_call and check_output should all execute the argument through the shell.

No, a subprocess is perfectly capable of running a program directly (via a call to the operating system). It doesn't need a shell

Things that can be done with shell = True that cannot be done with shell = False

You can use shell=False

for any command that just runs some executables, possibly with some specified arguments.



You should use shell=True

if your command uses shell functions. This includes pipelines, |

or redirects, or that contain compound assertions combined with ;

or &&

or ||

, etc.

Therefore, for a command type grep string file

you can use shell=False

. But the command of the type grep string file | xargs something

, because of the |

demands shell=True

.

Since the shell has power functions that python programmers don't always find intuitive, it is considered best to use shell=False

if you don't really need a shell function. As an example, piping is really unnecessary because it can also be done using the PIPE subprocess function.

+1


source






More articles:


All Articles