What to do when the queue size is reached?
I am using Slick 3.0 with HikariCP 2.3.8 (also game 2.4)
I am doing a lot of database I / O and getting queued all the time. Is there a way to get the current queue size and how can I increase it? Or is it even recommended to do it, or would a wrapper around a database with its own queue be better?
Exception I mean:
java.util.concurrent.RejectedExecutionException: Task slick.backend.DatabaseComponent$DatabaseDef$$anon$2@39496345 rejected from java.util.concurrent.ThreadPoolExecutor@21ee20b4[Running, pool size = 20, active threads = 20, queued tasks = 1000, completed tasks = 7021]
source to share
1000 tasks in the queue seem very important to me. Obviously slick is using a fixed size queue (1000 items) executor, and you are working at that limit because tasks are not removed quickly enough.
The most obvious reason is the SQL runtime. If you can shorten your SQL execution time, you will buy yourself a lot of extra queue space.
Typically, this is done on the database side by checking query execution plans and, depending on the database, asking the database what long queries are running.
On the HikariCP side, you can enable DropWizard metrics (not sure how to do it with slick, tho) and enable DropWizard Log Log to log pool statistics every 10 seconds or so.
Probably the most interesting metric would be usage, as it will show you how many long connections are pooled between getConnection () and close (). When you tune your database and / or queries, you want this number to start dropping.
One of the most important points is that if the database cannot handle the load on the application, increasing the queue from 1000 to 5000 (or even 10000) will buy you nothing but a small amount of time before it reaches this limit. You have to find the source of the bottleneck and fix it so the queues come out faster than your application generates them (except for temporary bursts, of course).
source to share