How do I use Rmpi in R on a linux Cluster to grow the kernel available with DEoptim?
I am using code developed in R to calibrate an 8 parameter hydrological model using DEoptim (a function that aims to minimize an objective function). DEoptim code uses the "parallel" package to determine the number of cores available with "DetectCores ()". On my PC, I have 4 cores with 2 threads each, so it detects 8 cores and then sends the hydrological model to the core with different parameter values and the results are returned to the center. He does this hundreds or thousands of times and repeats the parameters to try to find the optimal set. Therefore, the more cores there are, the faster it will work.
I am at university and have access to a Linux compute cluster. They have servers with up to 12 cores (i.e. not threads) and if I were to use that it would run two to three times faster than my computer. Fine. Ideally, however, I would distribute the code to other servers so that I have access to more cores and all information is sent back to the master.
So my question is how to incorporate Rmpi into my code to effectively increase the available kernels. As you can probably tell, I am completely new to using clusters.
Thanks a lot, Antony
source to share
If you want to run DEoptim
on multiple nodes of a Linux cluster, I believe you need to use foreach
by specifying parallelType=2
in the argument control
. You can use either a parallel server doMPI
or a backend doParallel
with an MPI cluster object. For example:
library(doParallel)
library(Rmpi)
cl <- makeCluster(mpi.universe.size()-1, type='MPI')
registerDoParallel(cl)
# and eventually...
DEoptim(fn=Genrose, lower=rep(-25, n), upper=rep(25, n),
control=list(NP=10*n, itermax=maxIt, parallelType=2))
You need to install the package snow
in addition to the others. Also, make sure you execute your script with mpirun using the option -np 1
. If you don't use mpirun, all workers will be created on the local machine.
source to share