Dask, create dataframe from multiple dask arrays
Suppose I have a set of dask arrays like:
c1 = da.from_array(np.arange(100000, 190000), chunks=1000)
c2 = da.from_array(np.arange(200000, 290000), chunks=1000)
c3 = da.from_array(np.arange(300000, 390000), chunks=1000)
Is it possible to create a dask frame out of them? In pandas, I could say:
data = {} data['c1'] = c1 data['c2'] = c2 data['c3'] = c3 df = pd.DataFrame(data)
Is there a similar way to do this using dask?
+3
source to share
1 answer
The following should work:
import pandas as pd, numpy as np
import dask.array as da, dask.dataframe as dd
c1 = da.from_array(np.arange(100000, 190000), chunks=1000)
c2 = da.from_array(np.arange(200000, 290000), chunks=1000)
c3 = da.from_array(np.arange(300000, 390000), chunks=1000)
# generate dask dataframe
ddf = dd.concat([dd.from_dask_array(c) for c in [c1,c2,c3]], axis = 1)
# name columns
ddf.columns = ['c1', 'c2', 'c3']
+3
source to share