System Guides: SHC

1. How to Get An Account

CACR Accounts

2. How to Access the Machine

Connect to the front end, shc.cacr.caltech.edu:

ssh -l username shc.cacr.caltech.edu

Edits, compiles, builds, and job submissions are done on the front end.

NOTE: password entry into the head node is not allowed, but rather access is permitted via ssh public keys. If you do not have a public key, please see SSH key generation instructions.

3. System Configuration Information / Technical Summary

The Shared Heterogenous Cluster provides computing capabilities specifically configured to meet the needs of applications from Caltech’s Predictive Science Academic Alliance Program, Turbulent Mixing, Applied and Computational Mathematics, and Numerical Relativity communities.

The configuration, integrated by Hewlett-Packard and CACR’s technical support team, consists of 203 AMD dual processor nodes, connected via Voltaire’s Infiniband and an Extreme Networks BlackDiamond 8810 GigEswitch.

The typical programming model used for SHC jobs is MPI, with flexible queue policies allowing for development and production runs.

Cluster Configuration Overview Diagram (coming soon)

Technical Summary:

Architecture	Opteron RHEL Linux Cluster
Head Nodes	8 processors, single core 16 GB ECC SDRAM memory 2 nodes
Compute Nodes	dual-processor, dual & quad core 2 GB ECC SDRAM memory per core 229 nodes (1180 cores)
Processor	Dual-processor, dual-core 86 AMD Opteron [275, 2214 HE], 2.2 GHz 77 AMD Opteron [280, 2216 HE], 2.4 GHz Dual-processor, quad-core 66 AMD Opteron 2380, 2.5 GHz
Network Interconnect	PCI-x InfiniBand Copper Gigabit Ethernet
Disk	~84 TB raw, RAID6 nfs project work area/nfs/ds[01,02,03,04]/project-id 180 GB local scratch/node
Operating System	Linux 2.6.18 (RHEL 5.3)
Compilers	GNU 4.3.3 PGI 8,9 Pathscale 3.[1,2]
Batch System	Torque with Maui
MPI	Open MPI

4. Available File Systems and Descriptions/Intended Usage

• nfs mounted project work area, /nfs/ds[01,02,03,04]/project-id, seen globally. RAID6, not backed up
• psaap project users have a dedicated nfs mounted project area, /nfs/ds02/psaap/username
• sxs project users have their own dedicated nfs mounted project space: /nfs/ds0[3,4]/sxs/username and /nfs/ds0[3,4]/sxs_bbhdata for group readable data sets
• a local home – seen globally and backed up daily.
• shc-storage01, hosting /nfs/ds01/, is a dual processor, dual core AMD Opteron 870 (2 GHz) system with 16 GB of memory. A 16 port Areca controller hosts an 8 TB (raw) RAID6 disk array. A 24-port Areca controller hosts 24×750GB disks.
• shc-storage02 hosting /nfs/ds02 is dual core AMD Opteron 875 (2.2 Ghz), 16 GB memory, 40×750 GB disks
• shc-storage03, hosting /nfs/ds03 is a dual processor, quad core AMD Opteron 8350 (2 Ghz), 16 GB memory, 40×750 GB SATA disks
• Each compute node has a local scratch disk (~180 GB) which is freshly purged when the scheduler gives you access to the nodes. Local scratch is visible only to the compute node controlling the disk.
• Local scratch is not backed up.

5. Compiling/Preparing to Run

Available Compilers:

• PGI: pgcc, pgCC, pgf90, pgf77
• PathScale: pathcc, pathCC, pathf90
• Gnu: gcc, g++, g77
  • install root = /usr/bin

Available MPIs:

• OpenMPI-1.3.3, Open MPI

/usr/local/bin/pkgs lists available packages on shc. Packages are items on the software stack which are commonly used, and necessarily need to have environment variables and paths updated in order to be used properly. Available packages are:

In order to set your environment to use a particular package, do use pkgname. For example, to use the PathScale compilers do

use pathscale

which will update your PATH, LD_LIBRARY_PATH, MANPATH and set the PathScale enviroment variables appropriately. To drop a package from your environment, do drop pkgname. For example, to drop the PGI compilers from your environment do drop pgi

6. Supported Debuggers and Debugging Tips

• pathdb (PathScale Debugger)
• pgdebug (PGI Debugger)
• libefence.a – electric fence library for boundary violation reporting
• valgrind – To use valgrind with MPI jobs, do [l,h]mpirun -n X /usr/bin/valgrind –log-file=memlog a.out where memlog is a random name, and you’ll see memlog.pid for each task with info about memory and pointer usage.
• TotalView Documentation
  • Examples are located in /usr/local/TotalView/linux-x86-64/examples
• 32 processor, single user license
• To run your MPI application under TotalView:
  • compile and link your code with -g
  • add -tv to the mpirun argument list
  • Example requesting 4 interactive nodes for 2 hrs, including X11 forwarding, running a 16 way a.out executable qsub -I -l nodes=4 -l walltime=2:0:0 -X mpirun -tv -np 16 ./a.out

7. How to Launch and Manage Parallel Jobs

Interactive jobs on the compute nodes are initiated from the head node via

 qsub -I -l nodes=X:core4 -l walltime=HH:MM:SS

where X is the number of nodes you want for the specified wall time. Batch jobs are submitted via,

qsub sample.pbs

where sample.pbs looks like:

#!/bin/csh -f

# ask for 2 core4 nodes for 1 hour
#PBS -l nodes=2:core4
#PBS -l walltime=01:00:00
#
# Direct stdout/err as desired
#PBS -o /home/sharon/examples/hello_out
#PBS -e /home/sharon/examples/hello_err
#
# Let's make sure each node can load the executable
/usr/bin/ldd $HOME/examples/hello
# cd to the exeuctable area if desired or launch with absolute path
# Do an 8 way hello run
/usr/local/openmpi/bin/mpirun -np 8 $HOME/examples/hello

SHC has three types of compute nodes, Opteron 275, 280, and 2380. There are eighty-six 275s, running at 2.2GHz, and seventy-seven 280s running at 2.4GHz, and sixty-six 2380s running at 2.5 GHz.

http://www.amd.com/us-en/Processors/ProductInformation/0,,30_118_8796_9240,00.html

• Short (< 2 hour) jobs are given scheduling priority on 6 “core4″ nodes Mon-Fri, 8am to 6pm.
• Available queues on shc are “productionQ”, “weekdayQ”, “weekendQ”, and “dedicatedQ”. The philosophy is to generally not have to specify a queue. Specify instead, the resources you need and your job will be routed to the proper queue.
• To accommodate dual processor, quad core nodes as well as dual processor, dual core nodes in the shc environment, resource specification is as follows:

-l nodes=NN[:<type<][:ppnval][+MM[:<type>][:ppnval]]...

NN, MM	Number of nodes of the specified typerequired by the job; default is 1.
type	Node type: “core8″ for eight-core nodesand “core4″ for four-core nodes; default value is “core4″.
ppnval	Number of processes that will be run oneach node, specified as “ppn=K”, where “K” is a value not exceeding the core count; default value is the number of cores the allocated nodes have.

Jobs submitted with a runtime of > 12 hours will automatically be routed to a weekendQ.

dedicatedQ runs need special approval

WeekdayQ runs are M-F, 0800 to 1700

WeekendQ runs are F-M, 1700 to 0800. Caltech holidays extend the active window for the WeekendQ

Job submission on shc examples:

A standard, single-node, 4 way MPI executable which needs to run

for 30 minutes on a four-core node:

 qsub -l walltime=00:30:00 -l nodes=1:core4:ppn=4 jobA
 qsub -l walltime=00:30:00 -l nodes=1:core4 jobA
 qsub jobA

*all of the above are equivalent

Equivalent ways to submit a six-node, 48 way, jobB which needs to

run for 1.5 hours on eight-core nodes:

 qsub -l walltime=1:30:00 -l nodes=6:core8:ppn=8 -q productionQ jobB
 qsub -l walltime=1:30:00,nodes=6:core8 jobB

Equivalent way to submit a job that should run on a mixture of

node types (three eight-core nodes plus six four-core nodes)

for four hours:

 qsub -l walltime=4:00:00 -l nodes=3:core8+6:core4 jobC
 qsub -l walltime=4:00:00,nodes=3:core8+6:core4 jobC