Clever Geek Handbook

Best way to mark (split?) The dataset in each row

I have a dataset that contains 485k (1.1GB) rows. Each line contains about 700 characters containing about 250 variables (1-16 characters per variable), but it has no sections. The lengths of each variable are known. What is the best way to change and label data with a symbol ,

?

For example: I have lines like:

0123456789012...
1234567890123...

and an array of lengths: 5,3,1,4,...

then I have to do the following:

01234,567,8,9012,...
12345,678,9,0123,...

Can anyone help me with this? Python or R tools are mostly preferred for me ...

+2


source to share


4 answers


Pandas can load this with read_fwf

:

In [321]:

t="""0123456789012..."""
pd.read_fwf(io.StringIO(t), widths=[5,3,1,4], header=None)
Out[321]:
      0    1  2     3
0  1234  567  8  9012


This will give you a dataframe that allows you to access every single column for whatever purpose you require

+1


source


In R read.fwf

will work:

# inputs
x <- c("0123456789012...", "1234567890123... ")
widths <- c(5,3,1,4)

read.fwf(textConnection(x), widths, colClasses = "character")

giving:



     V1  V2 V3   V4
1 01234 567  8 9012
2 12345 678  9 0123

If you want numeric rather than character columns, drop the argument colClasses

.

+1


source


Try this in R:

x <- "0123456789012"

y <- c(5,3,1,4)

output <- paste(substring(x,c(1,cumsum(y)+1),cumsum(y)),sep=",")
output <- output[-length(output)]
+1


source


One option in R

indx1 <- c(1, cumsum(len)[-length(len)]+1)
indx2 <- cumsum(len)
toString(vapply(seq_along(len), function(i)
         substr(str1, indx1[i], indx2[i]), character(1)))
#[1] "01234, 567, 8, 9012"

data

str1 <- '0123456789012'
len <- c(5,3,1,4)
0


source






More articles:


All Articles