Reusing ThreadPoolExecutor versus creating and deleting Ad Hoc?
I am creating a multi-threaded process that has multiple stages, each stage iterating through an unknown number of objects (hundreds of thousands from a buffered request or text file). Each stage runs a runnable or callable for every object, but all runnables / callables must be filled before moving on to the next stage.
I dont want to use a latch or any kind of synchronizer because I dont want to hurt the bandwidth. I suspect the inside of the latch will slow things down with a synchronized counter. I also don't want to use a list of futures with invokeAll () either because I want to run the runnables right away when I iterate over them.
However, creating a ThreadPoolExecutor for each stage, overflowing and dispatching all runnables, and then closing it for each stage seems to be a functional solution ...
public void runProcess() {
ResultSet rs = someDbConnection.executeQuery(someSQL);
ExecutorService stage1Executor = Executors.newFixedThreadPool(9);
while (rs.next()) {
//SUBMIT UNKNOWN # OF RUNNABLES FOR STAGE 1
}
rs.close();
stage1Executor.shutdown();
rs = someDbConnection.executeQuery(moreSQL);
ExecutorService stage2Executor = Executors.newFixedThreadPool(9);
while (rs.next()) {
//SUBMIT UNKNOWN # OF RUNNABLES FOR STAGE 2
}
rs.close();
stage2Executor.shutdown();
}
However, I know that setting up threads, threadpools, and everything that involves concurrency is expensive to build and destroy. Or maybe it's not such a big deal and I'm just too careful about performance because concurrency has an expensive overhead no matter what. Is there a more efficient way to do this? By using some kind of wait-to-complete operation that I'm not aware of?
source to share
If you destroy the thread pool and restart a new one, it will probably cost you a lot more than using CountDownLatch!
In addition, the call stage1Executor.shutdown();
does not guarantee that all current threads will complete their execution before starting and starting a new ExecutorService. Even a challenge shutdownNow()
cannot guarantee this! (and you probably don't want to call shutdownNow()
because you want your threads to finish executing).
Donald Knuth once said:
premature optimization is the root of all evil.
so even if you don't convince me, you better listen to it :)
source to share
Setting up and disabling multiple thread pools is minor. Try it in a loop in a test.
Using a countdown latch is fine, but maybe it just duplicates the work that ThreadPoolExecutor is doing and ties your task to your execution map. Not a fan of this approach.
As far as source code is concerned, ExecutorService has a method awaitTermination
, so you can wait until your work is done before moving on to the next step.
With my money, your pseudocode is fine. Just replace executor.shutdown () with shutdownAndAwaitTermination(ExecutorService)
which source is here: http://docs.oracle.com/javase/7/docs/api/java/util/concurrent/ExecutorService.html
source to share