Python uses CPU system time when consuming memory
I noticed that when python (maybe just numpy
?) Takes up memory, the system cpu time becomes significant.
For example, when running the following code ( numpy
multithreading is disabled):
import numpy as np
a = np.random.rand(50, 50, 1000000)
About 88% of a running kernel is devoted to user time and 12% to system time.
However, when running the following code:
for i in range(1000000):
a = np.random.rand(50, 50)
All processor time is devoted to user time.
I want to understand:
(1) why is this happening?
(2) Do I need to profile my code to use less memory? Note that when I talk about profiling, I don't care about memory, and the wall is the only thing I care about. I'm just worried that too much system time is slowing down my program.
source to share
Test code:
N=10000000
M=10
# Method 1
a=[]
for i in range(N):
a.append(np.random.rand(M,M))
# Method 2
a = np.random.rand(M,M,N)
# Method 3
for i in range(N):
a = np.random.rand(M,M)
Results using the linux command time
:
Fig. 1
and:
Method 3 at M=10 and N=10000000
real 0m15.221s
user 0m15.169s
sys 0m0.016s
The results show that the system call time decreases with increasing block size in method 1 and finally goes to the same level as in method 2. Meanwhile, the fraction of the time cost of the system call decreases as the block size decreases due to the faster growth cost of time in user space.
As for method 3, it did not account for the overhead of allocating large amounts of memory, resulting in extremely negligible system call costs, which in your program is mostly unrealistic (I know what it does!).
source to share