I read you had to use stopCluster() after running parallel function: foreach() in R. However, I can get away with registerDoParallel() and then running foreach() as many times as I want without ever using stopCluster(). So do I need stopCluster() or not?
Does not using stopCluster() mean your cores are occupied with your current task? So if I am using parallel programming with only a little bit of single core sequential tasks in between, then I don't need to stopCluster()? I understand there are also significant overhead time consumption with setting up parallel.
1 Answer
parallel::makeCluster() and doParallel::registerDoParallel() create a set of copies of R running in parallel. The copies are called workers.
parallel::stopCluster() and doParallel::stopImplicitCluster() are safe ways of shutting down the workers. From the help page ?stopCluster:
It is good practice to shut down the workers by calling ‘stopCluster’: however the workers will terminate themselves once the socket on which they are listening for commands becomes unavailable, which it should if the master R session is completed (or its process dies).
Indeed, CPU usage of unused workers is often negligible. However, if the workers load large R objects, e.g., large datasets, they may use large parts of the memory and, as a consequence, slow down computations. In that case, it is more efficient to shut down the unused workers.
1
