How to multi-user POST list of JSON / xml files using python requests
In python2.7 I am using requests
REST endpoint to communicate. I can load separate JSON and xml objects onto it. To speed things up, I want to load multiple json objects using multipart.
I have a curl command that shows how this can be done and it works. I need to translate what in python is asking for a POST command.
WORKING SPEED:
curl --anyauth --user admin:admin -X POST --data-binary \@sample-body \
-i -H "Content-type: multipart/mixed; boundary=BOUNDARY" \
"http://localhost:8058/v1/resources/sight-ingest?rs:transform=aireco-transform&rs:title=file1.xml&rs:title=file2.xml&rs:title=file3.xml"
Notes: I need to send a list of configurable parameters including the "title" parameter list, can't I do this by passing a dict? but we can get around this.
my python path:
import requests
files = {'file1': ('foo.txt', 'foo\ncontents\n','text/plain'),
'file2': ('bar.txt', 'bar contents', 'text/plain'),
'file3': ('baz.txt', 'baz contents', 'text/plain')}
headers = {'Content-Type': 'multipart/mixed','Content-Disposition': 'attachment','boundary': 'GRENS'}
params={'title':'file1','title':'file2','title':'file2'}
r = requests.Request('POST', 'http://example.com', files=files , headers=headers, params=params)
print r.prepare().url
print r.prepare().headers
print r.prepare().body
Gives me:
http://example.com/?title=file2
{'boundary': 'GRENS', 'Content-Type': 'multipart/mixed', 'Content-Length': '471', 'Content-Disposition': 'attachment'}
--7f18a6c1b09f42009228f600b0af35fd
Content-Disposition: form-data; name="file3"; filename="baz.txt"
Content-Type: text/plain
baz contents
--7f18a6c1b09f42009228f600b0af35fd
Content-Disposition: form-data; name="file2"; filename="bar.txt"
Content-Type: text/plain
bar contents
--7f18a6c1b09f42009228f600b0af35fd
Content-Disposition: form-data; name="file1"; filename="foo.txt"
Content-Type: text/plain
foo
contents
--7f18a6c1b09f42009228f600b0af35fd--
Questions:
- Seems like the material in the header isn't being used in the body?
- Can I set my own border? "GRENS" is not used in the body?
- Can I pass a list of header parameters (with the same key) as in the curl example?
source to share
Don't use a dictionary, use a list of tuples (key, value)
for your query parameters:
params = [('title', 'file1'), ('title', 'file2'), ('title', 'file3')]
otherwise you will only get one key.
You don't have to set a title Content-Type
; requests
will set it correctly for you when you use the parameter files
; this will also include the correct border. You should never set the border directly, really:
params = [('title', 'file1'), ('title', 'file2'), ('title', 'file3')]
r = requests.post('http://example.com',
files=files, headers=headers, params=params)
You can set headers for each part of the file by adding a fourth element to the root file for additional headers, but in this case you should not try to set the header yourself Content-Disposition
; it will still be overwritten.
Introspection of the prepared request object then gives you:
>>> import requests
>>> from pprint import pprint
>>> files = {'file1': ('foo.txt', 'foo\ncontents\n','text/plain'),
... 'file2': ('bar.txt', 'bar contents', 'text/plain'),
... 'file3': ('baz.txt', 'baz contents', 'text/plain')}
>>> headers = {'Content-Disposition': 'attachment'}
>>> params = [('title', 'file1'), ('title', 'file2'), ('title', 'file3')]
>>> r = requests.Request('POST', 'http://example.com',
... files=files, headers=headers, params=params)
>>> prepared = r.prepare()
>>> prepared.url
'http://example.com/?title=file1&title=file2&title=file3'
>>> pprint(dict(prepared.headers))
{'Content-Disposition': 'attachment',
'Content-Length': '471',
'Content-Type': 'multipart/form-data; boundary=7312ccd96db94419bf1d97f2c54bbad1'}
>>> print prepared.body
--7312ccd96db94419bf1d97f2c54bbad1
Content-Disposition: form-data; name="file3"; filename="baz.txt"
Content-Type: text/plain
baz contents
--7312ccd96db94419bf1d97f2c54bbad1
Content-Disposition: form-data; name="file2"; filename="bar.txt"
Content-Type: text/plain
bar contents
--7312ccd96db94419bf1d97f2c54bbad1
Content-Disposition: form-data; name="file1"; filename="foo.txt"
Content-Type: text/plain
foo
contents
--7312ccd96db94419bf1d97f2c54bbad1--
If you absolutely must have multipart/mixed
and not multipart/form-data
, you will need to build the POST body yourself and set the headers from it. The included urllib3
tools should be able to do this for you:
from requests.packages.urllib3.fields import RequestField
from requests.packages.urllib3.filepost import encode_multipart_formdata
fields = []
for name, (filename, contents, mimetype) in files.items():
rf = RequestField(name=name, data=contents,
filename=filename)
rf.make_multipart(content_disposition='attachment', content_type=mimetype)
fields.append(rf)
post_body, content_type = encode_multipart_formdata(fields)
content_type = ''.join(('multipart/mixed',) + content_type.partition(';')[1:])
headers = {'Content-Type': content_type}
requests.post('http://example.com', data=post_body, headers=headers, params=params)
or you can use a email
package to do the same:
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText
body = MIMEMultipart()
for name, (filename, contents, mimetype) in files.items():
part = MIMEText(contents, _subtype=mimetype.partition('/')[-1], _charset='utf8')
part.add_header('Content-Disposition', 'attachment', filename=filename)
body.attach(part)
post_body = body.as_string().partition('\n\n')[-1]
content_type = body['content-type']
headers = {'Content-Type': content_type}
requests.post('http://example.com', data=post_body, headers=headers, params=params)
but please note that this method assumes that you set the character set (I assumed UTF-8 for JSON and XML) and that it will most likely use Base64 encoding for the content:
>>> body = MIMEMultipart()
>>> for name, (filename, contents, mimetype) in files.items():
... part = MIMEText(contents, _subtype=mimetype.partition('/')[-1], _charset='utf8')
... part.add_header('Content-Disposition', 'attachment', filename=filename)
... body.attach(part)
...
>>> post_body = body.as_string().partition('\n\n')[-1]
>>> content_type = body['content-type']
>>> print post_body
--===============1364782689914852112==
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="baz.txt"
YmF6IGNvbnRlbnRz
--===============1364782689914852112==
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="bar.txt"
YmFyIGNvbnRlbnRz
--===============1364782689914852112==
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="foo.txt"
Zm9vCmNvbnRlbnRzCg==
--===============1364782689914852112==--
>>> print content_type
multipart/mixed; boundary="===============1364782689914852112=="
source to share