System Guides: Shared Heterogeneous Cluster
1. How to Get An Account
http://www.cacr.caltech.edu/resources/accounts/
2. How to Access the Machine
Connect to the front end, shc.cacr.caltech.edu:
ssh -l username shc.cacr.caltech.edu
You should edit, compile, build and submit your compute node jobs on the front end.
NOTE: password entry into the head node of the Opteron cluster is not allowed, but rather access is permitted via ssh public keys. If you do not have a public key, please see http://www.cacr.caltech.edu/resources/sshkey-instructions.cfm.
3. System Configuration Information / Technical Summary
The Shared Heterogenous Cluster provides computing capabilities specifically configured to meet the needs of applications from Caltech's ASC, Turbulent Mixing, and Numerical Relativity communities. The configuration, integrated by Hewlett-Packard and CACR's technical support team, consists of 163 AMD Opteron dual core dual processor nodes, connected via Voltaire's Infiniband and an Extreme Networks BlackDiamond 8810 GigEswitch. The typical programming model used for SHC jobs is MPI, with flexible queue policies allowing for development and production runs.
| Architecture |
|
| Head Nodes |
- 8 processors, single core
- 16 GB ECC SDRAM memory
- 2 nodes
|
| Compute Nodes |
- dual-processor, dual core
- 8 GB ECC SDRAM memory
- 162 nodes (648 cores)
|
| Processor |
- 86 AMD Opteron 275, 2.2 GHz
- 77 AMD Opteron 280, 2.4 GHz
|
| Network Interconnect |
- PCI-x InfiniBand
- Copper Gigabit Ethernet
|
| Disk |
- ~84 TB raw, RAID6 nfs project work area /shc/datastore-[01,02,03]
- 180 GB local scratch/node
|
| Operating System |
- Linux 2.6.15.6 (SuSE SLES 9.0)
|
| Compilers |
- GNU 3.4.5, 4.2.[1,2]: Fortran77 C
- C++
- PGI 6.2, 7.[0,1]
- Pathscale 2.[4,5]
|
| Batch System |
|
| MPI |
|
4. Available File Systems and Descriptions/Intended Usage
- nfs mounted project work area, /nfs/ds[01,02,03]/projectid, seen globally. RAID6, not backed up
- psaap project users have a dedicated nfs mounted project area /nfs/ds02/psaap/username
- sxs project users have their own dedicated nfs mounted project space: /nfs/ds03/sxs/username and /nfs/ds03/sxs_bbhdata for group readable data sets
- local home - seen globally.
- Each project has a quota'd partition under this mount point, with subdirectories for each user within the project.
- shc-storage01, hosting /nfs/ds01/projectid, is a dual processor, dual core AMD Opteron 870 (2 GHz) system with 16 GB of memory. A 16 port Areca controller hosts an 8 TB (raw) RAID6 disk array. A 24-port Areca controller hosts 24x750GB disks.
- shc-storage02 hosting /nfs/ds02 is dual core AMD Opteron 875 (2.2 Ghz), 16 GB memory, 40x750 GB disks
- shc-storage03, hosting /nfs/ds03 is a dual processor, quad core AMD Opteron 8350 (2 Ghz), 16 GB memory, 40x750 GB SAS-RAID6, 10 GigE ada
- Each compute node has a local scratch disk (~180 GB) which is freshly purged when the job scheduler gives you access to the nodes. Local scratch is visible only to the compute node controlling the disk.
- Local scratch is not backed up.
5. Compiling/Preparing to Run
Available Compilers:
- PGI: pgcc, pgCC, pgf90, pgf77
- install root = /usr/local/PGI
- PathScale: pathcc, pathCC, pathf90
- install root = /usr/local/PathScale
- Gnu: gcc, g++, g77
Available MPIs:
Useful "home grown" commands for accessing the compilers and MPI utilities:
/usr/local/bin/pkgs lists available packages on shc. Packages are items on the software stack which are commonly used, and necessarily need to have environment variables and pathes updated in order to be used properly. Available packages are:
- mpich - Portable MPI from ANL
- pathscale - The PathScale compiler suite - default compilers (v3.1)
- pathscale_3.1
- python - [2.4.3, 2.5, 2.5.2, 2.6]
- pgi - The Portland Group compilers (v7.1)
- pgi_6.2, pgi_7.[0,1]
- openmpi - openmpi (v1.2.1) default MPI
- openmpi_[1.2...]
- totalview - The Totalview Debugger
- tecplot - Tecplot-360-r2
- tecplot_10
- use --list (shows available packages)
- use --what (shows what packages you are using )
In order to set your environment to use a particular package, do use pkgname. For example, to use the PathScale compilers do
use pathscale
which will update your PATH, LD_LIBRARY_PATH, MANPATH and set the PathScale enviroment variable to the appropriate installation base directory. To drop a package from your environment, do drop pkgname. For example, to drop the PGI compilers from your environment do drop pgi
6. Supported Debuggers and Debugging Tips
- pathdb (PathScale Debugger)
- pgdebug (PGI Debugger)
- libefence.a - electric fence library for boundary viloation reporting
- valgrind - To use valgrind with MPI jobs, do [l,h]mpirun -n X /usr/bin/valgrind --log-file=memlog a.out where memlog is a random name, and you'll see memlog.pid for each task with info about memory and pointer usage.
- TotalView Documentation
- Examples are located in /usr/local/TotalView/linux-x86-64/examples
- 32 processor, single user license
- To run your MPI application under TotalView:
- compile and link your code with -g
- add -tv to the mpirun argument list
- Example requesting 4 interactive nodes for 2 hrs, including X11 forwarding, running a 16 way a.out executable
qsub -I -l nodes=4 -l walltime=2:0:0 -X
mpirun -tv -np 16 ./a.out
7. How to Launch and Manage Parallel Jobs
Interactive jobs on the compute nodes are initiated from the head node via qsub -I -l nodes=X -l walltime=HH:MM:SS where X is the number of nodes you want for the specified wall time. Batch jobs are submitted via, qsub sample.pbs where sample.pbs looks like:
Submitting a job to the weekend queue (24 hour runtime max) requires -q weekend as an option to qsub or a #PSUB -q weekend line in your batch script.
SHC has two types of compute nodes, opeteron 275 and 280. There are eighty-six 275s, running at 2.2GHz, and seventy-seven 280s running at 2.4GHz.
http://www.amd.com/us-en/Processors/ProductInformation/0,,30_118_8796_9240,00.html
Submitting a run to use the 2.4 GHz nodes requires the tag, operon280, in the resource spec. For example, to request four 2.4GHz nodes, do
#PBS -l nodes=4:opteron280
Specifically requesting the slower 2.2GHz nodes is done via
#PBS -l nodes=4:opteron275
qstat -Q -f gives a detailed description of all the shc queues.
Wrappers for compiling MPI codes
- /usr/local/openmpi-1.2.1/bin/mpi[cc,CC,f77,f90] uses PathScale compilers and openmpi-1.2.1. Codes created with these wrappers are launched with /user/local/openmpi/bin/mpirun. These are the default MPI wrappers.
7a. Commonly Used Job Scheduler Commands
- Display the status of a batch job, qstat -a
When monitoring jobs with qstat, look at "Elap Time" (elapsed time) rather than "Time Use". This is because "Elap Time" is the time since the job started, while "Time Use" is the CPU time used by the user process; this number is usually zero or close to it, since it counts the script that actually launches the MPI job, not the job itself.
- Delete (cancel) a job, qdel PBS_JOBID
- Show all running jobs on the system, qstat -r
- Show detailed information for a specified job, qstat -f PBS_JOBID
- Show all queues, qstat -q
- Show queue limits for all queues, qstat -Q
- Show quick information of the server, qstat -B
- Show compute node status, pbsnodes -a
Jobs are routed and prioritized depending on the walltime you request, and fairshare valve. A system queue exists for special jobs. Just send mail to shc-support at cacr.caltech.edu and request this queue be activated for your benchmark runs, long runs, special needs runs, etc.
- Show "earliest" startime for <JOB> , showstart <jobid>
- Show summaries of running, idle, blocked jobs - showq
8. Getting Help/Communicationg with Other Users/Staff
Send mail to shc-support at cacr.caltech.edu. CACR technical support is available during standard business hours (M-F, 8am to 5pm); after hours responses are as time permits.
9. Expectations About System Down Times/News
Scheduled Preventive Maintenance (PM) is Monday from 0900 to 1300. CACR Operations will not always take the full PM for systems work. If you would like to schedule benchmarking or test runs with dedicated access to the entire system, asking shc-support for a portion of PM time is fine and encouraged.
News about operational changes (e.g. system software upgrades, file system policy changes, dedicated runs) will be posted using news. New news items since last ssh'ing to the head node will be displayed automatically.
Using news:
- /usr/local/bin/news -help
- Usage:
- news prints all new news items
- news X prints news item "X"
- news -a prints all news items
- news -l prints names of all news items
- news -h prints this message
10. Known Problems
pbsnodes -a believes nodes are either free or down. Allocated nodes appear free, so look at the job field and note there's a job id assigned to the allocated nodes and not to the free nodes.
qstat -a allows you to monitor your job in a basic way by looking at the Elapsed Time field grow with time. qstat with no arguments doesn't report useful walltime/cputime for jobs in the Time Use field because the time reported is the negligible time consumed by the script which launches the mpi job, not the mpi job itself.
11. Miscellaneous System Software
- xmgrace - a WYSIWYG tool to make two-dimensional plots of scientific data
- xemacs - new version of Emacs with improvements over GNU Emacs
- sxemacs - spin-off of xemacs
- Petsc version 2.3.0 (patchlevel >= 41 )
- PETSc 2.3.2-p1
- HDF 4.2r1, 5.5, 5-1.6.5, 5-1.8.1
- tecplot -[360, 10] (tecplot tutorials)
- python v 2.4.3, 2.5, 2.5.2, 2.6
- perl5 (revision 5 version 8 subversion 7)
- /usr/lib/libgoto.so
- /usr/lib/xerbla.o
- /usr/lib64/libblas.a
- /usr/lib64/libatlas
- Matlab v 7.3.0.298 (R2006b)
11a. System Monitoring
Monitoring activity on shc (as a whole or on a somewhat fine grain level) can be done using Ganglia, an opensource distributed monitoring system for clusters. Just point your browser at http://shc.cacr.caltech.edu/ganglia/ to see an overview of system wide and node specific activity.
12. Performance
Single core Linpack, 3.811 Gflops (86% peak)
13. Policies
File system policies will regularly be reviewed. When 70% utilization is realized on pvfs, users will be warned to clean up.
All users are expected to adhere to the CACR Computing Policies.
14. Accounting and Job Priority Policies
Jobs are scheduled according to "weight." Many factors are taken into consideration when determining a job's weight, including cpu time consumed recently by user/group, time spent waiting in the queue, priority of the group, runtime and node count being requested, etc.
The intent of FairShare job scheduling is to prevent a user or group from dominating compute resources - striking a balance between utilization of cpu resources, job throughput, and fair scheduling between projects and project members.
Current fair share reporting outputs can be otained via executing diagnose -f from an shc head node
Notice that there are 8 columns, numbered 0 to 7. If entries associated with a user in these columns show a value other than "---", this means there was cpu consumption X, where X ranges from 0 to 7 days ago. Consumption a week ago (column 7) is weighted less heavily (0.4783) than consumption today (column 0). The value listed reflects job duration, job size and when the job ran.
Notice the column with the "%" heading. This shows the "penalty" or weight associated with a given user and/or group. If a value > 8.00 (Target value) appears by a user, this indicates a new jobs submitted will get be "penalized", due to recent consumption by himself or his group, allowing queued jobs from users with lower "%" values to run sooner.
Groups have different priorities - note that vtf3d and tmx have 2x priority over sxs, but as consumption by tmx or vtf3d reaches the target, jobs from these group decrease in priority so group domination of the cpu resources does not generally occur.
Queues have different priorities and policies. When special runs are requested, these jobs are placed in the system queue and given highest priority. The system queue is typically used for long jobs prior to a conference with an upcoming deadline, etc. Jobs submitted to the weekend queue have priority on weekends, over the dque, but the weekend queue is only active (running jobs) Friday 1700 to Monday 0800. Holidays extend weekend queue active hours.
[ last updated 3 November 2008]