R: Unable to allocate more than x MB
I have a main function in R that calls other files to run my program. I am calling the main file via a bat (.exe) file. When I run it in turn, it runs without memory error, but when I call the bat file to run it, it stops and gives me the following error:
Memory over 51 MB cannot be allocated.
How can I avoid this?
source to share
Memory constraints in R, such as this, are a recurring nightmare for many of us.
Very often the problem is limited by the limitations of your OS (which can usually be changed in the Bash or PowerShell command line), architecture (32 v. 64 bit) or the availability of contiguous free RAM, regardless of the total available memory.
It's hard to say why something didn't cause memory issues when executed line by line, but would hit the memory limit when run as .bat.
Which version of R are you using? Have you both installed? Is the 32-bit call Rscript
when you run your .bat file, then how do you run the 64-bit in turn? You can check the R version that is running with R.Version()
.
You can test this by running the command memory.limit()
in your R IDE / terminal and in your .bat file (be sure print
or save the result as an object in your .bat file). You may also need to set memory.limit()
to your .bat file as it may be smaller by default, possibly due to differences in your R profile that is called in your IDE or terminal compared to the .bat file.
If the architecture is not the cause of the error in your memory, then you have a few more troubleshooting steps to try:
- Check memory usage in both environments (directly in R and through your .bat process) using this:
sort( sapply(ls(),function(x){object.size(get(x))}))
- Run the garbage collector explicitly in your scripts so that the command
gc()
- Check all object sizes to make sure there are no unexpected results in your .bat process:
sort( sapply(ls(),function(x){format(object.size(get(x)), units = "Mb")}))
-
Try memory profiling:
Rprof(tf <- "rprof.log", memory.profiling=TRUE) Rprof(NULL) summaryRprof(tf)
-
While this is a RAM issue, for good measure you might need to check that the available computing power is sufficient and does not vary between these two ways of running your code:
parallel::detectCores()
-
Examine your performance with the Hadley Wikham tool
lineprof
(warning: requireddevtools
and does not work in lines of code that invoke the C programming language)Links . While I'm pulling these snippets from my own code, most of them were originally linked from other related StackOverflow posts such as:
source to share