How do I iterate through a large dataset using Firebase / Node.js?

This question is really about the performance of iterations in Node and not something Firebase related if I understand correctly. The question is, when you have this array of 20,000+ objects, what is the best way to execute them and perform an operation on each of them?

There are several possible performance optimizations you could make. Firstly, you can use a loop for

instead Array.prototype.forEach()

, which tends to be faster for very large arrays.

...
var len = questionsSnapshot.length;
for (var i = 0; i < len; i++) {
  console.log(questionsSnapshot[i].val());
}

      

        
        
        
      

    

Or, using ES6 for ... of syntax:

for (let question of questionsSnapshot) {
  console.log(question.val());
}

      

        
        
        
      

    

However, when I did a quick test using console.time () in Node on an array of 20,000 objects, I didn't notice the constant performance gain. In fact, Array.prototype.forEach()

often faster than using a loop for

.

Another optimization you can do is move the nested callback in forEach()

to the module scope. This way, it will only be created once, instead of being recreated every time its parent function is called.

fbase.child("questions").once("value").then(function(questionsSnapshot) {
  return questionsSnapshot.forEach(logValue);
});

function logValue(questionSnapshot) {
  return console.log(questionSnapshot.val());
}

      

        
        
        
      

    

However, this will lead to a significant performance increase if you make many calls to the parent function, that is, if you make your Firebase request frequently. It seems to me that the question has more to do with repeating a large array if you already have that, rather than repeating the same request.

So, if you want to perform an operation on every element in your array, I think you are doing it ok. 20,000 objects are not that great; it takes about 5ms to iterate through such an array on my unimpressive computer. In fact, no matter what you do for each element, it will likely take longer than iterating through the array.

If you notice a certain performance lag, you can clarify by editing the post. Otherwise, I would advise not to worry about it.