Python: using List List instead of loop to improve performance
I have a type from a dictionary (example)
l =('1037_97',["a","b","c","d","e"])
I want to save a file (las format), but Liblas can only write one dot.
for l in Groups.iteritems():
for p in xrange(len(l[1])):
file_out.write(l[1][p])
I am trying to use if possible with a List List to save the code and speed up the loop
source to share
If you want a shorter solution, consider using map()
for an inner loop, or even both. But this is unlikely to lead to significant performance gains. However, it for p in l[1]:
may be faster than this c xrange
. The following example should do what you wanted in one line:
map(lambda g: map(file_out.write, g), groups.itervalues())
Now compare the performance of different implementations. Here I tried to measure time from some test data:
import timeit
groups = dict(('1037_%d' % i, ["a","b","c","d","e"]) for i in xrange(100))
class FOut(object):
def write(self, v):
#print v
pass
file_out = FOut()
def using_map():
map(lambda g: map(file_out.write, g), groups.itervalues())
def initial_version():
for l in groups.iteritems():
for p in xrange(len(l[1])):
file_out.write(l[1][p])
def seq_iteration():
for l in groups.iteritems():
for p in l[1]:
file_out.write(p)
def seq_iteration_values():
for l in groups.itervalues():
for p in l:
file_out.write(p)
def list_compr():
[[file_out.write(v) for v in g] for g in groups.itervalues()]
tests = ('initial_version', 'using_map', 'seq_iteration', 'list_compr', 'seq_iteration_values')
for test in tests:
print test, timeit.timeit('%s()'%test, 'from __main__ import %s'%test, number=10000)
And the result:
initial_version 0.862531900406
using_map 0.703296899796
seq_iteration 0.541372060776
list_compr 0.632550954819
seq_iteration_values 0.493131160736
As you can see, your initial version is the slowest, fixing the iterations helps a lot, the version map()
is short but not as fast as the c version itervalues()
. List accounting that creates unnecessary lists is nice, but still slower than a regular loop.
source to share
Cycle concepts do not necessarily speed up the cycle. They only speed up the loop if the end result is to be a list. Enumerating lists is faster than creating an empty list and adding to it one at a time.
In your case, you want to write items to a file rather than create a new list. The cost of creating the list is then lost.
You don't need a call xrange()
, but just go through l[1]
. You don't need .iteritems()
to either, since you are ignoring the keys. Use instead .itervalues()
:
for lst in Groups.itervalues():
for p in lst:
file_out.write(p)
I used lst
as a loop variable; l
easy to confuse for i
in many fonts.
source to share