Filling Python in a loop but with maximum flows
Right now I have a loop for
that goes through a list, usually this list is 100-500 units long. In the loop, a for
new stream is opened for each item. So now my code looks like this:
threads = []
for item in items:
t = threading.Thread(target=myfunction, args=(item,))
threads.append(t)
t.start()
But I don't want to start a new thread because it only takes a few MAX seconds for each thread to execute my function. I want to make my loop motionless by accessing my function with every argument. But, to close the stream as soon as it is done, and allow another to take charge. The maximum number of threads that I want to open is at least 3, no more than 20. Although if it's easier this range may change. I just don't want to open a new stream of every item in the loop.
For the curious, and if it matters. myfunction is a function I defined that uses urllib to send a post request to the site.
I am new to python, but I am not new to coding. Sorry for the noob question.
source to share
I think you are looking for a thread pool to solve your problem.
Some possible solutions are described in the answers to this question .
One of the simplest (assuming python3 or backport in pypi ):
from concurrent.futures import ThreadPoolExecutor
executor = ThreadPoolExecutor(max_workers=10)
futures = []
for item in items:
a = executor.submit(myfunction, item)
futures.append(a)
This will do myfunction on all elements using 10 threads. You can wait for the calls to end later using the futures list.
source to share
I believe your problem lies in missing features. this could be a number of problems, i recommend you visit the pythons homepage: https://goo.gl/iAZuNX
#!/usr/bin/python
import thread
import time
# Define a function for the thread
def print_time( threadName, delay):
count = 0
while count < 5:
time.sleep(delay)
count += 1
print "%s: %s" % ( threadName, time.ctime(time.time()) )
# Create two threads as follows
try:
thread.start_new_thread( print_time, ("Thread-1", 2, ) )
thread.start_new_thread( print_time, ("Thread-2", 4, ) )
except:
print "Error: unable to start thread"
source to share
slightly modify your code to enable checking for the number of active threads at any given time:
threads = []
consumed_by_threads = 0
consumed_by_main = 0
for item in items:
at = threading.activeCount()
if at <= 20:
t = threading.Thread(target=myfunction, args=(item,))
threads.append(t)
consumed_by_threads += 1
t.start()
else:
print "active threads:", at
consumed_by_main += 1
myfunction(item)
print "consumed_by_threads: ", consumed_by_threads
print "consumed_by_main: ", consumed_by_main
# here the rest of your code, thread join, etc
Note. I only check the maximum number of threads. BTW: should be 21 since the main thread is included in the counter (see here and follow the link enumerate
)
Nota Bene: As usual, double check the benefits of multithreading on your specific application, depending on the python implementation you are using and whether streams are thread-bound or I / O bound.
source to share