Are promises and locks that use all my memory?

I am currently working on a project that uses node.js as a driving system for doing relatively large-scale machine learning with images. I end up running out pretty quickly trying to do this, although I'm trying to optimize usage as much as possible and my data shouldn't take up too much space. My code relies heavily on promises and anonymous functions to control the pipeline, and I'm wondering why I'm seeing insane usage in my test case.

For context only, I'm using the mnist dataset for my testing, which can be found here . The training kit consists of 60,000 20x20 images. From these I extract the overflow functions, a description of this can be found here . What it boils down to is an array of 4096 elements for each image, so 60,000 of them. I am caching all images and function data in redis.

A quick calculation tells me that the full set of functions here should be 4096 * 8 * 60000 = 1966080000

bytes or appx 1.82GB of memory, assuming each element in the array is a 64-bit javascript number. The images themselves should take up very little space and I don't store them in memory. However, when I run my code, I see more than 8-10 GB of memory used after fetch / load. When trying to do more operations on this data (for example, print it all out in a JSON file so I can make sure the fetch is working correctly), I quickly wipe out 16GB of available memory on my machine, node script crashing.

So my general question is, why am I seeing such large memory usage? Is this because of my heavy use of promises / closures? Can I refactor my code to use less memory and allow more variables to be garbage collected?

The code is available for review here . Keep in mind that this is a little rough as far as the organization goes.