Infrastructure hierarchy

HPC

Login node

"Master" of the slaves

Don't run things here :)

Taurus

128 GB of RAM

64 CPUs ("cores")

Great for batch and interactive jobs

Mammoth

384 GB of RAM

16 CPUs ("cores")

Great for high-memory jobs

Data gravity

a.k.a. why are there so many people on this bridge?

Paying the price

Data goes from storage to application...

New data goes back to storage from application...

Using local storage in batch jobs

taurus and mammoth have a local "scratch" space we can use:

/var/scratch

Scratch space is a local, temporary place to put your data where you don't incur penalties for going over the network

Example batch script

#!/bin/env bash
#SBATCH -p batch
#SBATCH -n 4
#SBATCH -J test

WORKDIR=/var/scratch/$SLURM_JOBID
mkdir -p $WORKDIR

echo "Using $WORKDIR on $SLURMD_NODENAME"
echo

Trinity.pl --seqType fq --JM 10G --single AR1960BN.fastq --output $WORKDIR/AR1960BN.out --CPU 4 --inchworm_cpu 4 --bflyCPU 4

Creates a unique temp directory for you during runtime, ie:

/var/scratch/10287

Make sure to clean up when you're done!

Environment Modules

a.k.a magic!

Not Magic

modules modify your environment to dynamically load support for programs/packages

[aorth@hpc: ~]$ R
-bash: R: command not found
[aorth@hpc: ~]$ module load R
[aorth@hpc: ~]$ R

R version 3.0.0 (2013-04-03) -- "Masked Marvel"
...

In this case, the R module modified your $PATH to include the path to R

Ok, maybe it's magic

The $PATH is where your shell looks for commands when you type them...

[aorth@hpc: ~]$ echo $PATH
/usr/local/bin:/bin:/usr/bin:/usr/sbin:/sbin
[aorth@hpc: ~]$ module load R
[aorth@hpc: ~]$ echo $PATH
/export/apps/R/3.0.0/bin:/usr/local/bin:/bin:/usr/bin:/usr/sbin:/sbin

Every time you load a module, it prepends directories to your $PATH

The Man Behind the Curtain

Here's the "modulefile" for R:

set    appname       R
set    version       3.0.0
set    prefix        /export/apps/${appname}/${version}
set    exec_prefix   ${prefix}/bin
set    datarootdir   ${prefix}/share

conflict    R

prepend-path    PATH        ${exec_prefix}
prepend-path    MANPATH     ${datarootdir}/man

Simple enough, right? Notice it also overrides the $MANPATH

Other tricks

modules can also handle dependencies, for example:

[aorth@hpc: ~]$ module list
Currently Loaded Modulefiles:
  1) ilri/latest
[aorth@hpc: ~]$ module load cufflinks
[aorth@hpc: ~]$ module list
Currently Loaded Modulefiles:
  1) ilri/latest        2) samtools/latest    3) cufflinks/latest

cufflinks isn't much use without samtools...

More modules commands

module unload module -- Unload module

module avail -- List all available modules

SLURM stats

Powered by UBMoD

Top SLURM users (# jobs)

August's users who love science the most... ;)

1. ekamau (3,512)

2. mwamalwa (715)

3. dmujibi (573)

Top SLURM users (wall time)

1. tamuge (159.6)

2. ekamau (70.5)

3. dmujibi (61.8)

... in days!

More stats

9,402 jobs since April

6,396 jobs in the last 30 days

1,300 jobs in the last week

457 days of wall clock time in the last month!