Mongodb and pymongo 16Mb document size limit
I am parsing includes in an assembly using '/ showIncludes' on Windows and '-H' on * nix, flags.
I am parsing this information using a python script. Each included file becomes an object listing its children (the files it contains) and ancestors (including the paths that include the file).
After creating these objects, I want to insert them into the mongodb database using pymongo.
This works fine for 99% of inclusions. But ~ 5 is very large. When I try to add them to mongodb it complains.
Traceback (most recent call last):
File "mongodb.py", line 94, in <module>
includes_collection.update({'id': include.include_id}, { 'ancestor_tree': ancestor_tree_ids } )
File "C:\Python27\lib\site-packages\pymongo-2.7.2-py2.7-win-amd64.egg\pymongo\collection.py", line 551, in update
docs, check_keys, self.uuid_subtype, client)
DocumentTooLarge: command document too large
Reading on a mango seems to be a design choice. By default, documents cannot exceed 16 MB. But this can be overridden using the --nssize command line option. Cm
http://docs.mongodb.org/manual/reference/program/mongod/#bin.mongod
So, I restarted mongod with --nssize 32/64/128. I don't think any of my included items are above 128MB. But the problem persisted.
So now I'm wondering if the pimongo is to blame. Does it mean this server setting?
My mongod version
Versiondb v2.6.3 2014-08-28T16: 56: 51.534 + 0100 git version: 255f67a66f9603c59380b2a389e386910bbb52cb
I am using pymongo-2.7.2-py2.7-win-amd64.
Is there a way to get around this limitation?
source to share
Unfortunately, you cannot exceed the BSON limit of 16MB per document. The mongod option you are using has nothing to do with document size. It specifies the default size of the mongodb namespace file - not at all related to the maximum document size. As suggested in the documentation, if you really need to store objects larger than 16MB, I would look at the GridFS API.
From the documentation:
BSON documents
BSON Document Size
The maximum size of a BSON document is 16 megabytes.
The maximum document size helps to ensure that a single document cannot use an excessive amount of RAM or, during transmission, an excessive amount of bandwidth. To store documents exceeding the maximum size, MongoDB provides the GridFS API. See Mongofiles and the documentation for your driver for more information on GridFS.
Namespace file size
Namespace files can be up to 2047 megabytes in size.
By default, namespace files are 16 megabytes. You can adjust the size using the nsSize parameter.
- nssize
Default: 16
Defines the default size for namespace files, which are files that end in .ns. Each collection and index counts as a namespace.
Use this parameter to control the size for newly created namespace files. This parameter does not affect existing files. The maximum size for a namespace file is 2047 megabytes. The default value of 16 megabytes provides approximately 24,000 namespaces.
http://docs.mongodb.org/manual/reference/limits/
http://docs.mongodb.org/manual/reference/program/mongod/#bin.mongod
source to share