Django Haystack - ElasticSearch result window is too large

I am working on a web application developed with Django that uses Postgres and ElasticSearch. It uses ElasticSearch as the database stores a lot of data that grows by the hour. To get things up and running quickly, I used django-haystack 2.6.0, which allows me to easily access Elastic Search. So far so good.

The problem came when I made a request to ElasticSearch and the result was really great. The error I received was the following:

TransportError: TransportError (500, u'search_phase_execution_exception), u'Result window is too large, from + size should be less than or equal to: [10000], but was [12602]. See scrolling api for a more efficient way to query large datasets. This limit can be set by modifying the index level parameter index.ax_result_window. ')

After some research, I tried to increase the size of the result windows, but this will not work in the long run as the queryset continues to grow in size. Also, django-haystack does not implement the scroll api, so I cannot follow the guidelines shown in the stack trace.

My question is, is there anyway I can fix this error and get a large set of queries with django-haystack, perhaps paginating the results in some way? Or is there a better approach without using a haystack?

Thanks in advance!

+3


source to share





All Articles