ExecutorService.shutdown call on dynamic thread number
Good morning in my time zone.
I am using thread pool to develop a little Http robot that moves from link to link on every page. When I found a new link, I create a new thread that will explore this new page. Pseudocode.
pool = Executors.newFixedThreadPool(40); pool.execute(new Exploit(tree.getRoot()));
In this case Exploit is an inner class that implements the Runnable interface and has access to the pool, so every time one thread finds a link, it will use the pool to add a new "thread" like this:
for(Link n : links){
pool.execute(new Exploit(n));
}
I've seen many examples using the ExecutorService class, but they all use the same code:
ExecutorService executor = Executors.newFixedThreadPool(NTHREDS);
for (int i = 0; i < 500; i++) {
Runnable worker = new MyRunnable(10000000L + i);
executor.execute(worker);
}
// This will make the executor accept no new threads
// and finish all existing threads in the queue
executor.shutdown();
In the above code, the number of threads is static, so when the code triggers shutdown, all threads have already been added to the pool. I cannot follow this code because in my case I don't have a static number of threads to add. My stop condition to add more threads to the pool is when I have reached the deep search level. So my question is, how can I call executor.shutdown on the main thread? Is there some kind of connection I can use on the main thread?
Thanks in advance. Regards
source to share
You can take a look at Phaser . You can still use a fixed number of threads, but every time you find a link, you can register the other side and send an executable based on that link.
Phaser phaser = new Phaser(1);
ExecutorService e = Executors.newFixedThreadPool(n);
public void crawl(final String url){
visit(url);
phaser.arriveAndAwaitAdvance();
e.shutdown();
}
private void visit(String url){
phaser.register();
e.submit(new Runnable(){
public void run(){
//visit link maybe another visit(url)
phaser.arrive();
}
});
}
At this point, e.shutdown () will never happen until all links have been visited.
source to share
You need to keep track of how many tasks are currently in the pool. Increment the counter before each execute () call. Then decrement the counter at the end of each task, make sure you do this even when there is an exception.
Then the code that turns off the executor (one message of the first task) must wait in a while loop to see if the counter is 0.
The decreasing code needs to use notification to wake up the main thread.
class TaskCounter {
private final Object lock = new Object();
private long count;
public void taskStart() {
synchronize (lock) {
count++;
}
}
public void taskEnd() {
synchronize (lock) {
count--;
if (count == 0) {
lock.notify();
}
}
}
public void waitForAllTasksToComplete() throws InterruptedException {
synchronize (lock) {
while (count != 0) {
lock.wait();
}
}
}
}
source to share
In the code you are showing you have a static number of threads. newFixedThreadPool
creates a thread pool with a fixed number of threads.
When you call pool.execute
, you are not creating a new thread. You are creating a new task that will be executed by one of the existing threads. This is the whole thread pool.
source to share
newFixedThreadPool
will only set the number of threads running at the same time. It does not define the number of threads you can put into the executor service. So you can add as many threads as you want in your main stream start execute()
and shutdown()
the ExecutorService, when you consider that you can no longer add
source to share