What's the fastest way to delete MongoDB documents by date?
At our company, we store eight days of data (with a million aprox records.), So we have a cronjob that deletes documents older than eight days every day. Now we are using the Published field and this field is not indexed .
It takes 15 minutes to get rid of 100,000 records, and we found the operation to be too long.
This is a query where "docs" is a variable with an array of documents that we do not want to delete. The variable 'theDate' is the date eight days ago.
records.remove( { "Published" : { $lte : theDate }, "_id" : { $nin : docs } }
Would it be better to use the _id field that is indexed in the order in which this operation is performed? How can we use the _id field to perform the same operation?
source to share
Cancel Cron Job Completely: This is a job for TTL indexes. http://docs.mongodb.org/manual/core/index-ttl/
Create a TTL index in the field Published
with expireAfterSeconds: 691200
and watch as your documents are automatically deleted 8 days after publication.
And if you don't want to indiscriminately delete all documents 8 days after they were posted, save your Cron job and just create a simple index on the field Published
.
source to share