Tutorial: Introduction to High-Performance Computing with R

Dirk Eddelbuettel, Debian Project, Chicago, USA.

Abstract

R users are often limited by available memory, cpu power or both. The tutorial will cover a number of available options to enhance or accelerate R processing. Compared to the UseR 2008 tutorial, we will focus more on parallel computing (Rmpi, snow) and interfacing with compiled code (Rcpp, inline) and less on 'large memory'. Profiling, scripting and 'big memory' approaches with also be covered.

All code will be made available on a 'ready-to-run' live-cdrom for booting or virtual machine use.

Outline

The tutorial will cover the following topics:

Measuring performance using profiling tools from 'base R', add-on R package and external profiling tools.
Extending R using compiled code:
- using the 'inline' package
- using the Rcpp package for C/C++ interfaces
- going the other way and embedding R
Parallel R using explicit methods:
- using Rmpi directly
- using NWS with the python-nwsserver backend
- using snow on top of Rmpi, NWS, sockets or pvm
- using the slurm resource management software with Rmpi / snow
- using 'implicit parallelism' using pnmath/pnmath0
Extending memory limits: biglm, ff, bigmemory and other packages for 'out of memory' processing
Automation and scripting using littler and Rscript

Intended Audience

R users wishing to learn about measuring / profiling performance, running R in parallel or extending R by means of compiled code.

Required knowledge

Basic R programming and computing knowledge; some C / C++ knowledge may be helpful.