ElasticSearch multi-scroll API
I want to get all data from the index. Since the number of elements is too large for memory, I use Scroll (nice function):
client.prepareSearch(index)
.setTypes(myType).setSearchType(SearchType.SCAN)
.setScroll(new TimeValue(60000))
.setSize(amountPerCall)
.setQuery(MatchAll())
.execute().actionGet();
What works when called:
client.prepareSearchScroll(scrollId)
.setScroll(new TimeValue(600000))
.execute().actionGet()
But, when I call the first method multiple times, I get the same thing scrollId
multiple times, so I can't scroll multiple times - in parallel.
I found http://elasticsearch-users.115913.n3.nabble.com/Multiple-scrolls-simultanious-td4024191.html which says it's possible - although I don't know if it's ES.
Am I doing something wrong?
source to share
After searching for a few more, I realized it was (the same scrollId
) by design. When the timeout expires (which is reset after every call to Scan and Scroll Elasticsearch - add new index ).
This way you can only get one open scroll per index.
https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-scroll.html :
Scrolling is not intended for real-time user queries, but for processing large amounts of data, for example. to override the contents of one index into a new index with a different configuration.
So it looks like what I wanted is not an option, on purpose - perhaps because of optimization.
Update
As stated, creating multiple scrolls cannot be done, but this is only true when the query you are using to scroll is the same. If you are scroll
for, for example, another type
, index
or just query
, you can have severalscrolls
source to share
You can loop through the same index in the same amount of time, this is what elasticsearch-hadoop does.
Just remember that under the hood, the index is made up of multiple shards that have their own data, so you can loop through each fragment in parallel using:
.setPreference("_shards:1")
source to share