Iterating through a huge loop using python
I have 100000 images and I need to get vectors for each image
imageVectors = []
for i in range(100000):
fileName = "Images/" + str(i) + '.jpg'
imageVectors.append(getvector(fileName).reshape((1,2048)))
cPickle.dump( imageVectors, open( 'imageVectors.pkl', "w+b" ), cPickle.HIGHEST_PROTOCOL )
getVector is a function that takes 1 image at a time and takes about 1 second to process it. So basically my problem boils down to
for i in range(100000):
A = callFunction(i) //a complex function that takes 1 sec for each call
What I already tried: ( only pseduo code is shown here )
1) Using a numpy vector:
def callFunction1(i): return callFunction2(i) vfunc = np.vectorize(callFunction1) imageVectors = vfunc(list(range(100000))
2) Using a python map:
def callFunction1(i):
return callFunction2(i)
imageVectors = map(callFunction1, list(range(100000))
3) Using python multiprocessing:
import multiprocessing
try:
cpus = multiprocessing.cpu_count()
except NotImplementedError:
cpus = 4 # arbitrary default
pool = multiprocessing.Pool(processes=cpus)
result = pool.map(callFunction, xrange(100000000))
4) Using multiprocessing in a different way:
from multiprocessing import Process, Queue q = Queue() N = 100000000 p1 = Process(target=callFunction, args=(N/4,q)) p1.start() p2 = Process(target=callFunction, args=(N/4,q)) p2.start() p3 = Process(target=callFunction, args=(N/4,q)) p3.start() p4 = Process(target=callFunction, args=(N/4,q)) p4.start() results = [] for i in range(4): results.append(q.get(True)) p1.join() p2.join() p3.join() p4.join()
All of the above methods are time consuming. Is there any other way more efficient than this so that perhaps I could loop through many items at the same time and not sequentially or in any other faster way.
Time is mostly done by the function itself getvector
. While working, I split my data into 8 different batches and ran the same program for different parts of the loop and ran eight separate python instances on an octa-main VM in the google cloud. Can anyone suggest if there is a good option to reduce the map or use the GPU using PyCuda?
source to share
The solution multiprocessing.Pool
is good in the sense that it uses all of your kernels. Therefore, it should be about N times faster than using a regular old card, where N is the number of cores you have.
By the way, you can skip determining the number of cores. By default, it multiprocessing.Pool
uses as many processes as your processor has cores.
Instead of a simple one map
(which blocks until everything is processed), I would suggest using imap_unordered
. This is an iterator that will start returning results as soon as they become available so that your parent process can start further processing, if any. If ordering is important, you might want to return a tuple (number, array) to identify the result.
Your function returns a numpy array of 2048 values, which I believe numpy.float64
Using standard display functions transports the results back to the parent process using IPC. On a 4-core computer, that would result in 4 IPC transponders 2048 * 8 = 16384 bytes, so 65536 bytes per second. It doesn't sound so bad. But I don't know how much overhead the IPC will have (including etching and queues).
If you have a lot of overhead, you may need to create a shared memory area to store the results. To store 100,000 results, 2048 8-byte floats will require approximately 1.5 Gib. This is a significant amount of memory, but not relevant for current machines.
For 100,000 images and 4 cores, and each image takes about one second, your program's execution time will be 8 hours.
Your most important optimization challenge is to learn how to reduce the execution time of a function getvector
. For example, would it work just as well if you halve the size of your images? Assuming the execution time is linear with the number of pixels, which should cut the execution time to 0.25s.
source to share