Pandas Python: writing output to specific cells in CSV
I am trying to write pandas df to csv. However, I need to loop a lot to get my output in chunks. I want to output the output vertically in csv. So after each iteration, I want to write a specific set of lines (1:10, then 11:20, etc.) and clear the memory so as not to create a giant object. Can df.to_csv be used for this?
+3
source to share
1 answer
to_csv
takes a mode argument to add:
import numpy as np
# df.shape returns the dimensions in a tuple, the first dimension is the number of rows
df_list = np.array_split(df, df.shape[0]/10)
for d in df_list:
d.to_csv(csv_path, mode='a')
You can use numpy array splitting to create a list of df sections of 10 lines each and then write them or whatever you need to do to save memory
+5
source to share