How can I work with a large memory array?
I have a program that creates an array:
List1 = zeros((x, y), dtype=complex_)
I am currently using x = 500
and y = 1000000
. I will initialize the first column of the list with some kind of formula. Subsequent columns will then calculate their own values based on the previous column. After the list is completely filled, I show this multidimensional array with imshow()
.
The size of each value (item) in the list is 24
bytes. Approximate value from code:4.63829355451e-32
When I run the code with y = 10000000
, it takes up too much RAM and the system stops execution. How to solve this problem? Is there a way to save my RAM while still being able to easily process the list with imshow()
? Also, how big can the list be displayed imshow()
?
source to share
There is no way to fix this problem (in any way).
Computers ( as commonly understood ) have a limited amount of RAM and require elements in RAM to run.
The array size complex128
10000000x500
will require about 74GiB for storage. You need to somehow reduce the amount of data being processed if you want to use a regular computer for this (as opposed to a supercomputer ).
A common technique is to separate your data and process each section separately (possibly on multiple computers). Depending on the problem you are trying to solve, there may be special data structures that you can use to reduce the amount of memory required to represent the data. A sparse matrix is a good example .
It is very unusual that a lot of memory is needed . Be sure to do a thorough research to see if this is really necessary before you move on to extremely complex workarounds.
source to share