Reaching memory allocation in R
Trying to view an RData file that is 650,000KB and the machine has 8GB of RAM but keeps getting this error:
Achieved total distribution of 7758Mb: see help (memory.size)
Error in view: cannot allocate a 54.6 MB vector
I originally imported this file (2.5GB) using Read.table with a lot of difficulty due to the size. Used code:
A <- read.table(file.choose(),header = TRUE, sep= "|", fill = T, nrows = 9000000, stringsAsFactors = T)
save(A, file ="A.RData")
load("A.RData")
View(A)
source to share
Given the size of your file, I would suggest using packages ff
and ffbase
and reading directly from a text file using read.table.ffdf
. This bypasses memory requirements by storing data on the hard drive, but performing operations on it as if it were being read into RAM. Unfortunately, there are ffdf
not many functions for currently implemented for ffdf
, so depending on the modeling you would like to do, you may need to write your own implementations of the tools you need.
Other than that, you can try setting up a hadoop cluster and using rhadoop
or something like that, but I'm not really an authority.
source to share