How to iterate over a huge number of records using scala sorm
I want to iterate over all the records of a specific table in a sorm, but I want to do it in a way that is memory efficient.
The code I'm using today is:
Db.query[Items].whereEqual("title", someTitle).fetch.foreach { webCount =>
//do something
}
The problem is that this code first loads all records before entering each element. Is there a way to write records?
+3
source to share
1 answer
Ideally, this functionality would require support for database cursors, but this is not implemented.
However, this is solvable with manual dosing:
val results : Stream[ Items ] = {
val batchSize = 256
Stream
.from(0)
.map(Db.query[Items].whereEqual.limit(batchSize).offset(_ * batchSize).fetch)
.takeWhile(_.nonEmpty)
.flatten
}
You can of course wrap this pattern in a utility function or implicit conversion.
0
source to share