What does Elasticsearch auto-slicing do?

Auto slicing is a way to parallelize work for several different endpoints like reindex , update on demand and remove on demand .

All three APIs work in the same way, creating a scroll request for the target index. Scroll queries provide a more efficient way to make queries that yield larger result sets than the normal requested pages. Scroll queries can be further improved by using slice them .

It is clear that if a query needs to return a large number of hits, you can make a normal query and page through the results with from

/ size

, but this will fail due to deep lookup . To work around this issue, ES allows scrolling queries to get results in batches of N strokes. These scroll requests can be improved by cutting them, i.e. Split scrolling into multiple chunks that can be used independently by your client application.

So, let's say you have a query that needs to return 1,000,000 hits, and you want to loop through that result set in batches of 50,000 hits using a normal scroll request (i.e. no slicing), your client application would have to make the first scroll call and then 20 more synchronous calls (i.e. one after the other) to get each batch of 50k calls.

Using slicing, you can parallelize 20 scroll calls. If your client application is multithreaded, you can force each scroll call to use 5 (for example) chunks, and thus you will get 5 chunks of ~ 10K hits that can be consumed by 5 different threads in your application, instead of having one thread consume 50 thousand hits. This way you can use the full processing power of your client application to take advantage of these views.

The ideal number of slices should be a multiple of the number of shards in the original index. For best performance, you should select the same number of fragments as there are shards in the original index. For this reason, you can use automatic slicing instead of manual slicing, as ES will pick that number for you.