DANSE Resource – foxtrot – Getting Started
1. How to Get An Account
CACR Account Request Forms (must be DANSE Project Affiliate). All users are expected to adhere to the CACR Computing Policies.
2. How to Access the Machine
Password entry into the cluster is not allowed, but rather access is permitted via ssh public keys:
ssh -l username foxtrot.danse.us
This brings you into the head node, behind which sits 34 compute nodes (8 with 64Gigs of RAM and the rest with 32Gigs). These nodes have 2 x 6core processors, ie 12 cpus.
3. System Configuration Information
Foxtrot provides computing capabilities and an environment specifically configured to meet the production needs of applications from Caltech’s DANSE VNF lab, simulating neutron scattering experiments. The configuration, integrated by Penguin Computing and CACR’s technical support team, consists of 408 AMD Opteron 2431 core @ 2.4GHz with 672GB of total main memory. See the Technical Summary for further information.
The compute nodes DO NOT have a full linux installation, instead they run a specialized “bproc” kernel. Access to running programs on these nodes can be done via “bpsh” as in
bpsh n0 “whoami”
To prevent user conflicts it is encouraged to reserve a node (see PBS below).
Nodes are named n0…n33. These nodes are also connected to each other via Infiniband; the IP interface to the Infiniband network for each of these nodes is:
n<[0-33]-ipmi
e.g.
n0-ipmi
4. Software
The system comes with software such as OpenMPI, Mpich, BLAS libraries, LAPAC libraries pre-installed. For a list of all CACR installed software do:
module avail
(another useful command is rpm -q –filesbypkg <rpmName>)
To set up your environment to use a particular piece of software that is not installed in the default locations, do:
module add <moduleName>
Currently only OpenMPI is setup for module control. In the future CACR software will be and the rest of the sytem software shouldn’t need any special treatment.
To set up your environment to compile OpenMPI programms with the gcc compiler located in /usr/bin/gcc, do
module add openmpi/gnu
To get more info on the module environment for OpenMPI do:
module whatis openmpi
You may want to start by compiling a small mpich program as noted below.
PBS, otherwise known as Torque, is currently configured on this system. PBS provides a FIFO access into the nodes. Accessing nodes only allocated to you via PBS prevents two users using the node at that same time.
To get a node for interactive use, do
qsub -I -l “nodes=1:ppn=12″
To get a node with 64 gigs do:
qsub -I -l ‘nodes=1:ppn=12:laregemem’ -l walltime=1:00:00
and with 32 Gigs of memory do:
qsub -I -l ‘nodes=1:ppn=12:smallmem’ -l walltime=1:00:00
To get 2 64G nodes to use for interactive work do:
qsub -I -l ‘nodes=1:ppn=12:laregemem’ -l walltime=1:00:00
Once you are on one of the nodes if you want to see what the other node allocated to you is, do:
cat $PBS_NODEFILE
This node file is created in such a way that it will work as input to your OpenMPI jobs. The output has to be modified to work appropriately with mpich (hooks will be made to create a mpich appropriate node file) It will be put in the $PBS_NODEFILE_MPICH environment variable
See /usr/local/share/CACR/mpich/HowTo for how to copy and try out the sample mpich programs provided by the mpich developers in /usr/share/mpich/examples. This will be fine-tuned and an mpich-specific node file will be created for you eventually by us.
5. Documentation
Access to Penguin’s documentation for the system can be seen by going to:
Currently access to these docs are restricted to the .caltech.edu domain.
6. Communicating with other users/staff
- User community email:
- User support email:
