Failed to get more than 10 thousand records
I am developing an application where I have over 10,000 records added to a class in parsing. Now I am trying to fetch these records using PFQuery (I am using the skip property ). But I cannot get records beyond 10k and I get the following error
"Passes exceeding 10,000 are not allowed "
This is a big problem for me as I need all the data.
Has anyone faced such a problem. Share your opinions.
thank
source to share
The problem is really related to the cost of mongo skip operations. You can frame your request in such a way that you don't need a skip operator. My preferred method is orderBy objectId and then add the condition that objectId> last yielded objectId. This type of query can be indexed and remain fast, as opposed to skipping paging, which has a cost of O (N ^ 2) lookups.
source to share
My guess would be that it was based on performance issues with MongoDB skip .
The cursor.skip () method is often expensive because it requires the server to go from the beginning of the collection or index to get an offset or skip position before returning the result. As the offset increases (like pageNumber above) the .skip () cursor will run slower and more intensively. With large collections, cursor.skip () can become IO bound.
source to share