Python multiprocessing. Can workers start at slightly different times?
import multiprocessing as mp
from datetime import datetime
def worker(a):
print(str(a)+": "+str(datetime.now()))
time.sleep(1)
pool=mp.Pool(3)
if __name__ == '__main__':
pool.map(worker,range(10))
output:
0: 2017-04-18 23:37:31.399574
1: 2017-04-18 23:37:31.400422
2: 2017-04-18 23:37:31.400571
3: 2017-04-18 23:37:32.401644
4: 2017-04-18 23:37:32.401765
5: 2017-04-18 23:37:32.401904
6: 2017-04-18 23:37:33.403168
7: 2017-04-18 23:37:33.403250
8: 2017-04-18 23:37:33.403370
9: 2017-04-18 23:37:34.405025
3 workers start at the same time.
I really want to know that it is possible to run 3 workers at (slightly) different times?
Thank you very much in advance!
PS. As in the comment, any insignificant timing would be good (it would be better if I could control). However, I do not want workers to sleep before every job they do. I just want 3 workers to start at different times, but then never sleep between jobs. So I'm not sure if using the time.sleep function in a worker function works.
+3
source to share
1 answer
Pools can run an initializer in a child process. Use this for waiting.
import multiprocessing as mp
from datetime import datetime
import time
import random
import os
def sleepy():
nap = random.randint(1,10)
print(os.getpid(), 'sleeps', nap)
time.sleep(nap)
def worker(a):
print(os.getpid(), str(a)+": "+str(datetime.now()))
time.sleep(1)
pool=mp.Pool(3, initializer=sleepy)
if __name__ == '__main__':
pool.map(worker,range(10))
You can control your sleep time by creating a queue over time. Each initializer reads one value
import multiprocessing as mp
from datetime import datetime
import time
import random
import os
import threading
def sleepy(time_q):
timeout = time_q.get()
print(os.getpid(), 'sleeps', timeout, datetime.now())
time.sleep(timeout)
def worker(a):
print(os.getpid(), 'worker', str(a)+": "+str(datetime.now()))
time.sleep(1)
if __name__ == '__main__':
pool_size = 3
time_q = mp.Queue()
for sleep_time in range(1, pool_size+1):
time_q.put(sleep_time)
pool=mp.Pool(pool_size, initializer=sleepy, initargs=(time_q,))
pool.map(worker,range(10))
time_q.close()
time_q.join()
+2
source to share