How to insert 5 million records into MongoDB?

I am trying to insert 5 million documents into MongoDB and it is taking too long. Please suggest me a more efficient way. This is where I posted the db structure and associated query (code). Please take a look at this and point out improvements that can be made to speed up insertion.

Database structure:

    { 
        "_id" : {
            "msisdn" : "919899587091", 
            "op" : "idea", 
            "eid" : "547c0a0ccbbc64ce2b773488", 
            "cid" : "547c0a8ecbbc64cf2b773488", 
            "d" : ISODate("2015-05-26T04:30:00.000+0000")
        }, 
        "value" : {
            "unq" : NumberInt(1), 
            "ut" : "NNN"
        }
    }

      

Up to 5 million.

I am using aggregate like below:

    var v = db.collection(collectionname).aggregate([{$match:{'@timestamp':             {'$gte':new Date(starttime), '$lt':new Date(endtime)},'eid':{$ne:'-'},ut:{$ne:'-'}}},
        {$project:{'msisdn':"$msisdn",'op':'$op','eid':'$eid','cid':'$cid','ut':'$ut','mi':{'$millisecond':'$@tim`enter code here`estamp'},'m':{'$minute':'$@timestamp'},'s':{'$second':'$@timestamp'},d:'$@timestamp'}},
        {$project:{'msisdn':"$msisdn",'op':'$op','eid':'$eid','cid':'$cid','ut':'$ut','d':{'$subtract':['$d',{'$add':[{'$multiply':['$m',60,1000]},{'$multiply':['$s',1000]},'$mi',1800000]}]}}},
        {$group : {'_id' : {'msisdn':"$msisdn",'op':'$op','eid':'$eid','cid':'$cid','d':'$d'},'unq':{$sum:1},'uts':{$addToSet:'$ut'}}},
        {$project:{'_id':'$_id','value':{'unq':'$unq',
      'ut':{ $cond: {if: {$setIsSubset:[['NNN'],'$uts']} ,then: 'NNN',
         else: { $cond: {if: {$setIsSubset:[['RNN'],'$uts']} ,then: 'RNN',
           else:{ $cond: {if: {$setIsSubset:[['RRN'],'$uts']} ,then: 'RRN',else:'RRR'}}
                }}
          }}} }
        }],{ allowDiskUse: true, cursor: {batchSize: 1000}});

    var bulk = db.collection(outcollectionname).initializeUnorderedBulkOp();

    v.on('data', function(data){ 
            bulk.insert(data);  
        });

    v.on('end', function(){ 
    bulk.execute(function(err, result) {

            });
        });

      

+3


source to share


1 answer


I do it manually what is the $ out stage of the aggregation pipeline.

http://docs.mongodb.org/manual/reference/operator/aggregation/out/



Saves the results to another collection.

This allows you to copy the object to the nodejs context and copy it back to mongodb.

+3


source







All Articles