Storage capacity and ability to store huge data in a dictionary in python
I want to store huge data in a dictionary in python. Huge data can be around 21 GB . I wrote a snippet to do this. Storing integer values inside a dictionary.
Code:
import timeit
import sys
dicts = {}
print "\n***Building dict..."
start = timeit.default_timer()
for j in range(0,5):
for i in range(0,1000000):
dicts[''+str(j)+str(i)] = i
print str(i) + '-' + str(j)
print "Size : ", sys.getsizeof(dicts)/1024/1024, " MB"
print "Total time of build dict", timeit.default_timer() - start
At runtime, when I reached the size using **getsizeof(dicts)**
around 1.2GB , it cannot store the values in the dictionary, but no error is displayed . Does the dictionary have some storage capacity ?
So the question is, how can I store huge data in a dictionary?
NOTE. No need to store data in files or databases . because I want to get a key , a pair of values very quickly .
source to share
The Python python size limit depends on the free memory available to the OS. The problem is that as the dict grows (resizes), it needs to be copied itself in order to reallocate the keys, so when the dictionary gets really huge this process can start to require much more memory than is actually available.
source to share