Clever Geek Handbook

Iterating over a large dictionary (over 2 billion entries) using python

I am very new to python and I want to loop through a dictionary containing about 2 billion entries. through:

for key,value in edge_dict.items():

However, I am getting an out of memory exception because it seems like it is trying to load the whole thing and then iterating over them. I had this problem with trivial things like:

for i in range (2000000000)

but i could easily replace them

while (i < 2000000000):
    i++

which solves the problem. However, with a dictionary, I don't know how to deal with keys without using a for-in iterator. (Note that keys are strings and values ​​are int).

+3


source to share


1 answer 


for key,value in edge_dict.iteritems():

I think what you want

similarly if

for i in range (2000000000)


raises a memory error, you can use an iterator

for i in xrange(2000000000)

iterators (and / or generators) only load one element at a time and are consumed as they iterate ... this removes many of the problems that large lists in memory can have

+8


source






More articles:


All Articles