Mass operation on mongoose
I want to store bulk data (over 1000 or 10000 records) in a single operation using mongoose. But Mongoose does not support bulk operations, so I will use my own driver (mongodb for insert). I know I will bypass all mongoose middleware, but that's ok. (Please correct me if I'm wrong! :))
I have the ability to store data by method inset
. But mongodb also provides Bulk class (ordering and unordered operation). Now I have the following question
- Difference between operations
insert
andbulk
(both can store bulk data)? - Any specific difference between
initializeUnorderedBulkOp()
(operation is performed sequentially) andinitializeOrderedBulkOp()
(operation is performed in parallel)? -
If I useinitializeUnorderedBulkOp
, will it affect the range search or any side effects? - Can I do this with Promisification (by bluebird)? (I am trying to do this.)
thank
EDIT :: I'm talking about insertion vs vs relative to multiple insertion. Which one is better. insert one by one with oily constructor OR insert in batches (1000) in insert mode. I hope it now cleans up the Mongoose batch insert (mongodb)? this link
source to share
If you call this from mongoose model you need .collection
accessor
var bulk = Model.collection.initializeOrderedBulkOp();
// examples
bulk.insert({ "a": 1 });
bulk.find({ "a": 1 }).updateOne({ "$set": { "a": 2 } });
bulk.execute(function(err,result) {
// result contains stats of the operations
});
That being said, you need to be careful. Apart from not being tied to the same validation and validation that can be attached to mongoose schemas when you call .collection
, you need to be sure that the database connection is already made. Mongoose's methods take this into account for you, but once you use the basic driver methods, you're all on your own.
As for diffferences, it's all there in the naming:
-
Order : means that batch instructions are executed in the same order in which they are added. They are executed one by one sequentially and one at a time. If an error occurs at any time, the batch is terminated and an error response is returned. Until then, all operations "go". This is not a rollback.
-
UnOrderred : means that batch operations can be performed in "any" sequence and often in parallel. This can lead to faster updates, but of course cannot be used where one bulk operation in a batch needs to happen before another (example above). Any errors that occur are simply "reported" as a result and the entire batch will be completed as sent to the server.
Of course, the main difference for any type of execution from standard methods is that a "whole batch" (in fact, 1000 in large quantities) is sent to the server and you only get one response. This saves network traffic and waits for each individual .insert()
or similar operation to complete.
As with "promise", anything with a callback that you can transform to return a promise follows the same rules as here. Remember that "callback / promise" refers to a method .execute()
, and what you return follows the rules for what is returned from bulk operations.
See Bulk in the main documentation for more information .
source to share