How to track progress in mclapply in R in a parallel package
My question is related to this question . However, the above question is using a package multicore
that has been replaced with parallel
. Most of the functionality in the answer cannot be replicated in a package parallel
. Is there a way to track progress in mclapply
. When looking at the documentation, mclapply
there is a parameter named mc.silent
, I'm not sure if this can track the progress and if so how and where can we see the log file? I am working on ubuntu
Linux OS. Below is an example of a reproducible example for which I would like to note progress.
require(parallel)
wait.then.square <- function(xx){
# Wait for one second
Sys.sleep(2);
# Square the argument
xx^2 }
output <- mclapply( 1:10, wait.then.square, mc.cores=4,mc.silent=FALSE)
Any help would be greatly appreciated.
source to share
This is an update to my linked answer .
library(parallel)
finalResult <- local({
f <- fifo(tempfile(), open="w+b", blocking=T)
if (inherits(parallel:::mcfork(), "masterProcess")) {
# Child
progress <- 0.0
while (progress < 1 && !isIncomplete(f)) {
msg <- readBin(f, "double")
progress <- progress + as.numeric(msg)
cat(sprintf("Progress: %.2f%%\n", progress * 100))
}
parallel:::mcexit()
}
numJobs <- 100
result <- mclapply(1:numJobs, function(...) {
# Do something fancy here... For this example, just sleep
Sys.sleep(0.05)
# Send progress update
writeBin(1/numJobs, f)
# Some arbitrary result
sample(1000, 1)
})
close(f)
result
})
cat("Done\n")
source to share
Thanks to the package, pbmcapply
you can now easily track the progress of tasks mclapply
and mcmapply
. Just replace mclapply
with pbmclapply
:
wait.then.square <- function(xx) {
Sys.sleep(2)
xx^2
}
library(pbmcapply)
output <- pbmclapply(1:10, wait.then.square, mc.cores = 4)
... which will display a pretty progress bar.
The author has a nice blog post about tech details and benchmarks here .
source to share