Python Popen not using correct encoding in Windows PowerShell
I am running my Python script in Windows PowerShell, and the script has to run another program using Popen and then pipe the output of that program (Mercurial, in fact) for use in my script. I am getting a coding error when I try to execute my script in PowerShell.
I'm pretty sure this is happening because Python is not using the correct encoding that PowerShell is using when you get the output of the Popen call. The problem is, I don't know how to tell Python to use the correct encoding.
My script looks like
# -*- coding: utf-8 -*-
#... some imports
proc = Popen(["hg", "--cwd", self.path, "--encoding", "UTF-8"] + list(args), stdout=PIPE, stderr=PIPE)
#... other code
When I run this script on Linux, I have no problem. I can also run the script on Windows 7 Home Premium 64-bit using PowerShell without issue. PowerShell in this Windows 7 uses code page 850, which is the output chcp
- 850
("ibm850").
However , when I run the script on Windows 7 Starter 32-bits using PowerShell, which by default encodes cp437 ( chcp
= 437
), I get the following error from Python (version 2.7.2):
File "D:\Path\to\myscript.py", line 55, in hg_command
proc = Popen(["hg", "--cwd", self.path, "--encoding", "UTF-8"] + list(args), stdout=PIPE, stderr=PIPE)
File "C:\Program files\Python27\lib\subprocess.py", line 679, in __init__
errread, errwrite)
File "C:\Program files\Python27\lib\subprocess.py", line 852, in _execute_child
args = list2cmdline(args)
File "C:\Program files\Python27\lib\subprocess.py", line 615, in list2cmdline
return ''.join(result)
UnicodeDecodeError: 'utf8' codec cant decode byte 0xe3 in position 0: unexpected end of data
I have tried the following, with no success (i.e. the above error report remains the same):
- Remove the line
# -*- coding: utf-8 -*-
from my script. - Remove the option
-- encoding UTF-8
to run Mercurial via Popen in your script. - Change the encoding to
chcp 850
in PowerShell before executing my script. - Many other Python breaks I've found in other Stack Overflow answers.
For my specific details, all of my source code is available here on BitBucket . hgapi.py
is the script that is giving the error.
UPDATE: the script is called by this other script which sets the encoding as follows
sys.setdefaultencoding("utf-8")
This line looks important because if I comment it out I get a different error:
UnicodeDecoreError: 'ascii' codec cant decode byte 0xe3 in position 0: ordinal not in range(128)
source to share
After using it, from __future__ import unicode_literals
I started getting the same error, but in a different part of the code:
out, err = [x.decode("utf-8") for x in proc.communicate()]
An error occured
UnicodeDecodeError: 'utf8' codec cant decode byte 0xe3 in position 33 ....
Indeed, there x
was a byte string with \xe3
(which ã
in cp1252) is included. So x.decode('utf-8')
I used instead x.decode('windows-1252')
and it didn't give me any errors. To support any type of encoding I used instead x.decode(sys.stdout.encoding)
. The problem has been resolved.
And it was in Python 3.2.2 with a Windows 7 Starter machine, but Python 2.7 worked fine on the same machine as well.
source to share
Try changing the encoding to cp1252
. Popen on Windows requires shell commands encoded as cp1252
. This looks like a bug and it also seems to be fixed in Python 3.X via the module subprocess
: http://docs.python.org/library/subprocess.html
import subprocess
subprocess.Popen(["hg", "--cwd", self.path, "--encoding", "UTF-8"] + list(args), stdout=PIPE, stderr=PIPE)
update:
Your problem can be solved with the smart_str function of the Django module.
Use this code:
from django.utils.encoding import smart_str, smart_unicode
# the cmd should contain sthe string with the commsnd that you want to execute
smart_cmd = smart_str(cmd)
subprocess.Popen(smart_cmd)
You can find information on how to install Django on Windows here . You can install pip first , and then you can install Django by running a shell as administrator and run the following command:
pip install Django
This will install Django into the Python site-packages directory.
source to share