Sed command is run using os.system () or subprocess.call () leaves csv file without delimiter
I am running a Python script that takes a CSV dump from Postgres database, and then I want to escape the double quotes in all of those files. So I am using sed for this.
In my Python code:
sed_for_quotes = 'sed -i s/\\"//g /home/ubuntu/PTOR/csvdata1/'+table+'.csv'
subprocess.call(sed_for_quotes, shell=True)
The process exits without any error, but when I load these tables in Redshift I get an error No delimiter found
and after checking the CSV I find that one of the rows is only half loaded, for example if it is a timestamp column then only half of them and there is no data in the table after that (while the actual CSV has this data before running sed
). And this leads to an error No delimiter found
.
But when I run sed -i s/\"//g filename.csv
on those files in a shell it works fine and the csv after running sed has all lines. I have verified that there is no problem with the data in the files.
What is the reason for this not working in a Python program? I also tried using sed -i.bak
in python program, but it doesn't make any difference.
Note that I am using an extra backslash (\) in my Python code because I need to escape the other backslash.
Other methods tried :
- Using
subprocess.Popen
no buffer size and positive buffer size, but it didn't help - Using
subprocess.Popen(sed_for_quotes,bufsize=-4096)
(negative buffer size) worked for one of the files that were throwing an error but then ran into the same problem in another file.
source to share
Don't use a middleware when you don't need it. And check the return code of the subprocess to make sure it completed successfully ( check_call
does this for you)
path_to_file = ... # e.g. '/home/ubuntu/PTOR/csvdata1/' + table + '.csv'
subprocess.check_call(['sed', '-i', 's/"//g', path_to_file])
By "intermediate" shell, I mean that the shell process is running subprocess
, which parses the command (± is split into spaces, but not limited to) and runs it (runs in this example sed
). Since you know exactly what arguments to use for sed
, you don't need all of this, and it's best to avoid it.
source to share
Put yours sed
in shell script
for example
#!/bin/bash
# Parameter $1 = Filename
sed -i 's/"//g' "$1"
Call shell script
with subprocess
:
sed_for_quotes = 'my_sed_script /home/ubuntu/PTOR/csvdata1/'+table+'.csv'
Use docs.python.org/3.6: shlex.split
shlex.split (s, comments = False, posix = True)
Split the string s using shell-like syntax.
source to share