Stream query results using native mongoDB driver for node
I have 100,000 records in a mongoDB collection and am trying to find them in a node.js application using a native driver.
I am following the example in the MongoDB doc for CursorStream , but I get the error:
RangeError: Maximum call stack size exceeded
Before this error, I am getting a lot of them:
(node) warning: Recursive process.nextTick detected. This will break in the next version of node. Please use setImmediate for recursive deferral.
here is my code:
var query = {...};
var fields = {...};
var options = {
// "limit": 10
//"skip": 10,
//"sort": title
}
var stream = myCollection.find(query, fields, options).stream();
// stream.pause();
var results = [];
stream.on('data', function (item){
results.push(item);
stream.pause();
// Restart the stream after 1 miliscecond
setTimeout(function() {
stream.resume();
}, 1);
});
stream.on('close'.....
The error also occurs when I don't define a listener for the data event. but this does not happen if you suspend the thread immediately after it is created.
Mongod version - v2.4.1 node driver version: 1.2.x
Any help / hint would be appreciated.
+2
source to share
2 answers
You might want to try setImmediate instead of setTimeout in order to provide the ability to flush I / O operations. See also https://github.com/ZJONSSON/streamz/issues/1
0
source to share