Clever Geek Handbook

Mongo error when using aggregation: sort exceeded memory limit

I am getting mongo error exceeded memory limit

with error code 16819

when I use aggregation sort.

I am using mongo 2.6.

The request looks like this:

db.BASE_TABLE_CREATION_ExecuteHiveScript_26_V0.aggregate([
     { "$project" : { "visitor_localdate" : 1 , "_id" : 0}}, 
     { "$sort" : { "visitor_localdate" : -1}}
])
+3


source to share


2 answers


By default, aggregation in MongoDB happens in memory, and pipeline stages are limited to 100MB of RAM. It looks like you have exceeded this threshold. To process a large dataset, you must activate the steps in the aggregation pipeline to write the data to temporary files. Use allowDiskUse

for this:



db.BASE_TABLE_CREATION_ExecuteHiveScript_26_V0.aggregate([
    { "$project" : { "visitor_localdate" : 1 , "_id" : 0}},
    { "$sort" : { "visitor_localdate" : -1}}
], { "allowDiskUse" : true })
+3


source


You don't need aggregation for this at all. Use query

db.BASE_TABLE_CREATION_ExecuteHiveScript_26_V0.find({}, { "_id" : 0, "visitor_localdate" : 1 }).sort({ "visitor_localdate" : -1 })


and put the index on visitor_localdate

. It is simpler and faster than aggregation.

+1


source






More articles: