Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.


Excerpt

This page describes how to run RStudio in a container on Pawsey systems with Slurm.

...

For this example we're going to be using the rocker/tidyverse (external site) Docker image. At the time of writing the latest available R version is 3.6.1. It provides an R installation, the RStudio server, the R devtools and the Tidyverse collection for data science. This image ships with a startup script that allows for a number of runtime options to be specified: see the USE menu in the Rocker homepage (external site).

Setting up the job script

...

Column
width900px


Tip
titleSet the working directory within RStudio

To read data from and write data to this directory, first manually change directory to this location from inside the RStudio session (see figure 2).


The following script launches an RStudio server on the compute node (download the template batch script). The first step in the script creates a working directory before launching Rstudio.

Column
width900px


Code Block
languagebash
themeEmacs
titleListing 1. Slurm script for running RStudio in a container
collapsetrue
#!/bin/bash -l
# Allocate slurm resources, edit as necessary 
#SBATCH --account=$PAWSEY_PROJECT
#SBATCH --ntasks=1
#SBATCH --nodes=1
#SBATCH --cpus-per-task=1
#SBATCH --mem=58G
#SBATCH --time=01:00:00
#SBATCH --job-name=rstudio_server
#SBATCH --partition=work 
#SBATCH --export=NONE

# Set our working directory
# Should be in a writable path with some space, like /scratch
# You'll need to manually change dir to this one through the RStudio interface
# Session -> Set Working Directory -> Choose Directory -> ...
dir="${MYSCRATCH}/rstudio-dir"

# Set the image and tag we want to use
image="docker://rocker/tidyverse:4.3.1"

# Load Singularity. The version may change over time. 
module load singularity/3.811.64-nompi    

# You should not need to edit the lines below

# Prepare the working directory
mkdir -p $dir
cd ${dir}

# Get the image filename
imagename=${image##*/}
imagename=${imagename/:/_}.sif

# Create a user-specific tmp directory to avoid clashes between users
tmp_dir="/tmp/tmp_$USER"
mkdir -p $tmp_dir

# Get the hostname of the Zeus node
# We'll set up an SSH tunnel to connect to the RStudio server
host=$(hostname)

# Set the port for the SSH tunnel
# This part of the script uses a loop to search for available ports on the node;
# this will allow multiple instances of GUI servers to be run from the same host node
port="8787"
pfound="0"
while [ $port -lt 65535 ] ; do
  check=$( ss -tuna | awk '{print $4}' | grep ":$port *" )
  if [ "$check" == "" ] ; then
    pfound="1"
    break
  fi
  : $((++port))
done
if [ $pfound -eq 0 ] ; then
  echo "No available communication port found to establish the SSH tunnel."
  echo "Try again later. Exiting."
  exit
fi

# Generate a random password for the session
export PASSWORD=$(openssl rand -base64 15)

# Pull our Docker image in a folder
singularity pull $imagename $image

echo "*****************************************************"
echo "Setup - from your laptop do:"
echo "ssh -N -f -L ${port}:${host}:${port} $USER@setonix.pawsey.org.au"
echo "*****"
echo "The launch directory is: $dir"
echo "*****"
echo "Secret for this session is: $PASSWORD"
echo "*****************************************************"
echo ""
echo "*****************************************************"
echo "Terminate - from your laptop do:"
echo "kill \$( ps x | grep 'ssh.*-L *${port}:${host}:${port}' | awk '{print \$1}' )"
echo "*****************************************************"
echo ""

# Launch our container
# Note that content of /home will be lost after runtime
# You'll need to manually change dir to the working dir through the RStudio interface
# Session -> Set Working Directory -> Choose Directory -> ... 
srun -N $SLURM_JOB_NUM_NODES -n $SLURM_NTASKS -c $SLURM_CPUS_PER_TASK \
  singularity exec -c \
  -B ${tmp_dir}:/tmp \
  -B ${dir}:$HOME \
  -B ${tmp_dir}:/var \
  ${imagename} \
  rserver --www-port ${port} --www-address 0.0.0.0 --auth-none=0 --auth-pam-helper-path=pam-helper --server-user=$(whoami) 


...