Stream query results using native mongoDB driver for node

I have 100,000 records in a mongoDB collection and am trying to find them in a node.js application using a native driver.

I am following the example in the MongoDB doc for CursorStream , but I get the error:

RangeError: Maximum call stack size exceeded

      

Before this error, I am getting a lot of them:

(node) warning: Recursive process.nextTick detected. This will break in the next version of node. Please use setImmediate for recursive deferral.

      

here is my code:

var query = {...};
var fields = {...};
var options = {
    // "limit": 10
    //"skip": 10,
    //"sort": title
}

var stream = myCollection.find(query, fields, options).stream();
//  stream.pause();
var results = [];
stream.on('data', function (item){
    results.push(item);
    stream.pause();
    // Restart the stream after 1 miliscecond
    setTimeout(function() {
        stream.resume();
    }, 1);
});

stream.on('close'.....

      

The error also occurs when I don't define a listener for the data event. but this does not happen if you suspend the thread immediately after it is created.

Mongod version - v2.4.1 node driver version: 1.2.x

Any help / hint would be appreciated.

+2


source to share


2 answers


It looks like the problem is solved by setting the batch size on the cursor stream:



var stream = myCollection.find(query, fields, options).batchSize(10000).stream();

      

+6


source


You might want to try setImmediate instead of setTimeout in order to provide the ability to flush I / O operations. See also https://github.com/ZJONSSON/streamz/issues/1



0


source







All Articles