Caltech Center for Advanced Computing Research » clusters

CACR-hosted Theoretical AstroPhysics Cluster expands

cacrweb — Tue, 02 Aug 2011 18:09:11 +0000

The MRI2 cluster, meeting the application needs of Caltech’s Theoretical AstroPhysics Including Relativity (TAPIR) group, has expanded to include an additional 40 compute nodes. The configuration, integrated by Hewlett-Packard and CACR’s Operations team, now consists of 2016 Intel X5650 compute cores, in 168 dual Westmere hex-core nodes, equipped with ~ 4 TB of memory (2 GB/core). The cluster is connected by a 2:1 fat tree QDR InfiniBand network, with high speed access to 80 TB (usable) of high performance Panasas storage and 48 TB of archival storage. For more information about the system, see the CACR Facilities & Operations page.

The MRI2 cluster, called “zwicky”, provides compute and storage resources for research codes investigating core-collapse supernovae, gamma-ray bursts, black holes and neutron stars.

New Cluster for Theoretical AstroPhysics Installed

cacrweb — Wed, 11 Aug 2010 22:53:27 +0000

Rendering of a rapidly spinning, gravitational-wave emitting newborn neutron star. Simulation: Ott et al. 2007 Rendering: Ralf Kaehler ZIB/AEI/KIPAC 2007

This month CACR has installed and configured a new cluster in the Powell-Booth Laboratory for Computational Science. This system is specifically configured to meet the applications needs of Caltech’s Theoretical AstroPhysics Including Relativity (TAPIR) group in the Physics, Mathematics, and Astronomy Division.

The MRI2 cluster is funded by an NSF MRI-R2 award with matching funds from the Sherman Fairchild Foundation.The configuration, integrated by Hewlett-Packard and CACR’s operations team, consists of 1536 Intel X5650 compute cores in 128 dual Westmere hex-core nodes equipped with a total of ~3 TB of memory, connected via QDR InfiniBand (IB). It includes 100 TB of high-performance, high-reliability disk space access via IB through a Panasas rack.

The research project using the new cluster, Simulating eXtreme Spacetimes: Facilitating LIGO and Enabling Multi-Messenger Astronomy, is led by Professor Christian Ott. The co-Investigators on the MRI award are Dr. Mark Scheel of TAPIR and CACR’s director, Dr. Mark Stalzer. The research will explore the dynamics of spacetime curvature, matter, and radiation at high energies and densities. Central project aspects are the simulation of black hole binary coalescence, neutron-star — black hole inspiral and merger, and the collapse of massive stars leading to core-collapse supernovae or gamma-ray bursts. Key results will be the prediction of gravitational waveforms from these phenomena to enable LIGO gravitational wave searches and to facilitate the extraction of (astro-)physics from observed signals.

The MRI2 cluster is named Zwicky, in honor of Caltech Astrophysics Professor Fritz Zwicky (1898-1974), who discovered supernovae and who was the first to explain how supernovae can be powered by the collapse of a massive star into a neutron star. Zwicky also discovered the first evidence for dark matter in our universe, proposed to use supernovae as standard candles to measure distances in the universe, and suggested that galaxy clusters could act as gravitational lenses.

Cellcenter & SHC users: Compiler Changes

cacrweb — Thu, 28 Jan 2010 00:24:40 +0000

Cellcenter and SHC users now have access to the Intel Cluster Toolkit 3.2, including the compiler suite v 11.1. We have a two-seat license. “use intel” will this package into your environment. More Information about the Intel Cluster Toolkit

Pathscale compilers are being retired Feb 25, 2010. Users are encouraged to rebuild their codes using Intel, PGI, or gnu compilers. Specifically, these pathscale based packages will retire Feb 25:

pathscale (alias for pathscale 3.2)
pathscale_[3.1,3.2]
openmpi_1.3.3_pathscale
openmpi_1.3.3_pathscale_3.2
openmpi_1.3.4_pathscale
openmpi_1.4_pathscale
openmpi_pathscale (alias for openmpi_1.3.3_pathscale_3.2)

Additional packages that will be retiring on Feb 25, due to bug fixes and newer release compatible offerings:

openmpi_1.3.2 (alias for openmpi_1.3.2_gcc)
openmpi_1.3.2_gcc
openmpi_1.3.4_gcc
openmpi_1.3.4_intel
openmpi_1.3.4_pgi
pgi_8.0
tecplot-2006r2
tecplot-2009r1
totalview_8.6

CACR system support over the holidays

cacrweb — Thu, 24 Dec 2009 23:33:42 +0000

CACR’s support staff will be observing Institute holidays and special release days, Dec 25 to Jan 3. During the break, we’ll periodically check e-mail but primarily be watching for support issues of critical nature. Jan 4, we are back to full staff/regular support operations. Wishing you all a peaceful Holiday season!

SHC Software Stack Upgrade – Update

cacrweb — Wed, 09 Sep 2009 17:37:58 +0000

Important Information for SHC Users

As of Sept 8, 2009, SHC has been transitioned to the new sw stack (RHEL+OpenIB). There are currently 115 core4 nodes and 65 core8 nodes, in production. For more information, please visit the SHC Getting Started / System Guide.

SHC Software Stack Upgrade

cacrweb — Thu, 03 Sep 2009 16:58:54 +0000

Important Information for SHC Users

Over the next couple of days, more backend nodes from shc-a will be transitioned to shc-[c,new]’s cluster of backend nodes, running the new software stack. By Sept 4, there will be just 24 shc-a backend nodes, all the rest of the compute nodes will be running the new software stack, seen from shc-[new,c].

Please port your codes to the new software environment if you’ve not already done so!
Please report any porting problems you’re having; we’ll help asap.
Details on how to rebuild your code for the new SHC environment can be found here
Your MPI based code must be rebuilt for the new and improved shc software stack.

Preventive Maintenance on Sept 8 from 0800 to 1400 will encompass testing the complete transition of SHC compute and head node resources to the upgraded software stack environment. The fully upgraded production SHC cluster configuration will be two head nodes (shc-[a,b]) and 1180 Opteron compute node cores (163 dual cpu/dual core + 66 dual cpu/quad core).

Questions or concerns about the upgrade? Just let us know.

SHC Cluster Expansion

cacrweb — Thu, 12 Feb 2009 21:30:47 +0000

CACR’s 163 node Shared Heterogeneous Cluster (SHC) has recently expanded by an additional 20 nodes. Each of these new nodes contains 16 GB of memory and have two quad-core, 2.5 GHz AMD Opteron Processors (model 2380). As with the existing SHC nodes, each of the new nodes is connected via Infiniband to CACR’s Infiniband Switch.

The SHC provides computing capabilities specifically configured to meet the needs of applications from Caltech’s PSAAP, Turbulent Mixing, Applied and Computational Mathematics, and Numerical Relativity communities. For more information about the SHC, including information for test users of the new nodes, see this page.

CACR’s Shared Heterogeneous Cluster (SHC) Now Online

cacrweb — Fri, 24 Mar 2006 21:15:59 +0000

The nature of financial support for high-end computing resources has evolved given the widespread adoption of Beowulf clusters. Research groups that need computing often obtain funds for clusters as part of their grants. CACR participates in some of these efforts, and supports significant dedicated resources for high-energy physics, astronomy, geophysics, physics-based simulation, and others. Unfortunately, the balkanization of computation by this model has created inefficiencies. The clusters do not take advantage of economies of scale, can be underutilized, and poorly administered. CACR has developed a shared cluster model, and Professors Paul Dimotakis, Dan Meiron, and Kip Thorne have agreed to be pioneer partners in this effort. CACR has purchased a machine optimized for parallel numerical codes that can sustain over 1 trillion floating point operations per second. It consists of 352 2.2Ghz AMD Opteron cores, 700+ Gigabytes of memory, all interconnected by an Infiniband networking fabric that can move 160+ Gigabytes/s between the compute nodes. The cluster is administered by CACR with funds from the partner groups, and each group has an allocation of time on the machine proportionate to its contribution. By sharing, the groups get better pricing from vendors, professional systems administration by experienced CACR staff, and the ability to use a much larger machine than each group could afford separately. Some of the partners are also supporting efforts at CACR in visualization and code tuning. The shared cluster model is extremely scalable, and CACR is interested in expanding the machine to increase simulation capability and add support for data intensive science. Please contact CACR’s Executive Director, Mark Stalzer (stalzer at caltech.edu) for more information.