Clever Geek Handbook

RCURL memory leak in getURL method

Looks like we got into a bug in RCurl. The getURL method seems to be causing a memory leak. A simple test case to reproduce the error is provided here:

library(RCurl)
handle<-getCurlHandle()
range<-1:100
for (r in range) {x<-getURL(url="news.google.com.au",curl=handle)}

If I run this code, the memory allocated for the R session is never reclaimed.

We are using RCURL for some lengthy experimentation and we are running out of memory on our test system.

Our test system specifications:

OS: Ubuntu 14.04 (64 bit)

Memory: 24 GB

RCurl version: 1.95-4.3

Any ideas on how to get around this issue?

thank

+4


source to share


2 answers


See if there is getURLContent()

also a getURLContent()

problem, that is, replace getURL()

with getURLContent()

. The function getURLContent()

is a richer version getURL()

and getURLContent()

more focus.



+4


source


I just hit this too and made the following code change to work around it:

LEAK (old code)

h = basicHeaderGatherer()
tmp = tryCatch(getURL(url = url,
                      headerfunction = h$update,
                      useragent = R.version.string,
                      timeout = timeout_secs),
               error = function(x) { .__curlError <<- TRUE; __curlErrorMessage <<- x$message })


NO LEAK (New code)

method <- "GET"
h <- basicHeaderGatherer()
t <- basicTextGatherer()
tmp <- tryCatch(curlPerform(url = url,
                            customrequest = method,
                            writefunction = t$update,
                            headerfunction = h$update,
                            useragent=R.version.string,
                            verbose = FALSE,
                            timeout = timeout_secs),
                error = function(x) { .__curlError <<- TRUE; .__curlErrorMessage <<- x$message })
+1


source






More articles:


All Articles