Rendering of a rapidly spinning, gravitational-wave emitting newborn neutron star. Simulation: Ott et al. 2007 Rendering: Ralf Kaehler ZIB/AEI/KIPAC 2007

This month CACR has installed and configured a new cluster in the Powell-Booth Laboratory for Computational Science. This system is specifically configured to meet the applications needs of Caltech’s Theoretical AstroPhysics Including Relativity (TAPIR) group in the Physics, Mathematics, and Astronomy Division.

The MRI2 cluster is funded by an NSF MRI-R2 award with matching funds from the Sherman Fairchild Foundation.The configuration, integrated by Hewlett-Packard and CACR’s operations team, consists of 1536 Intel X5650 compute cores in 128 dual Westmere hex-core nodes equipped with a total of ~3 TB of memory, connected via QDR InfiniBand (IB). It includes 100 TB of high-performance, high-reliability disk space access via IB through a Panasas rack.

The research project using the new cluster, Simulating eXtreme Spacetimes: Facilitating LIGO and Enabling Multi-Messenger Astronomy, is led by Professor Christian Ott. The co-Investigators on the MRI award are Dr. Mark Scheel of TAPIR and CACR’s director, Dr. Mark Stalzer. The research will explore the dynamics of spacetime curvature, matter, and radiation at high energies and densities. Central project aspects are the simulation of black hole binary coalescence, neutron-star — black hole inspiral and merger, and the collapse of massive stars leading to core-collapse supernovae or gamma-ray bursts. Key results will be the prediction of gravitational waveforms from these phenomena to enable LIGO gravitational wave searches and to facilitate the extraction of (astro-)physics from observed signals.

The MRI2 cluster is named Zwicky, in honor of Caltech Astrophysics Professor Fritz Zwicky (1898-1974), who discovered supernovae and who was the first to explain how supernovae can be powered by the collapse of a massive star into a neutron star. Zwicky also discovered the first evidence for dark matter in our universe, proposed to use supernovae as standard candles to measure distances in the universe, and suggested that galaxy clusters could act as gravitational lenses.

As of Sept 8, 2009, SHC has been transitioned to the new sw stack (RHEL+OpenIB). There are currently 115 core4 nodes and 65 core8 nodes, in production. For more information, please visit the SHC Getting Started / System Guide.

Over the next couple of days, more backend nodes from shc-a will be transitioned to shc-[c,new]’s cluster of backend nodes, running the new software stack. By Sept 4, there will be just 24 shc-a backend nodes, all the rest of the compute nodes will be running the new software stack, seen from shc-[new,c].

• Please port your codes to the new software environment if you’ve not already done so!
• Please report any porting problems you’re having; we’ll help asap.
• Details on how to rebuild your code for the new SHC environment can be found here
• Your MPI based code must be rebuilt for the new and improved shc software stack.

Preventive Maintenance on Sept 8 from 0800 to 1400 will encompass testing the complete transition of SHC compute and head node resources to the upgraded software stack environment. The fully upgraded production SHC cluster configuration will be two head nodes (shc-[a,b]) and 1180 Opteron compute node cores (163 dual cpu/dual core + 66 dual cpu/quad core).

Questions or concerns about the upgrade? Just let us know.

The SHC provides computing capabilities specifically configured to meet the needs of applications from Caltech’s PSAAP, Turbulent Mixing, Applied and Computational Mathematics, and Numerical Relativity communities. For more information about the SHC, including information for test users of the new nodes, see this page.

The TeraGrid is a multi-year effort to build and deploy the world’s largest, most comprehensive distributed infrastructure for open scientific research. The TeraGrid also offers storage, visualization, database, and data collection capabilities. Hardware at multiple sites across the country is networked through a 40-gigabit per second backplane — the fastest research network on the planet.

The systems currently in production represent the first of two deployments, with the completed TeraGrid scheduled to provide over 20 T eraflops of capability. The phase two hardware, which will add more than 11 T eraflops of capacity, was installed in December 2003 and is scheduled to be available to the research community this spring.

“We are pleased to see scientific research being conducted on the initial production TeraGrid system,” said Peter Freeman, head of NSF’s Computer and Information Sciences and Engineering directorate. “Leading-edge supercomputing capabilities are essential to the emerging cyberinfrastructure, and the TeraGrid represents NSF’s commitment to providing high-end, innovative resources.”

The TeraGrid sites are: Argonne National Laboratory; the Center for Advanced Computing Research (CACR) at the California Institute of Technology; Indiana University; the National Center for Supercomputing Applications (NCSA) at the University of Illinois, Urbana-Champaign; Oak Ridge National Laboratory; the Pittsburgh Supercomputing Center (PSC); Purdue University; the San Diego Supercomputer Center (SDSC) at the University of California, San Diego; and the Texas Advanced Computing Center at The University of Texas at Austin.

“This is an exciting milestone for scientific computing — the TeraGrid is a new concept and there has never been a distributed computing system of its size and scope,” said NCSA interim director Rob Pennington, the TeraGrid site lead for NCSA. “In addition to its immediate value in enabling new science, the TeraGrid project is a tool for the development of a national cyberinfrastructure, and the cooperative relationships forged through this effort provide a framework for future innovation and collaboration.”

“The TeraGrid partners have worked extremely hard during the two-year construction phase of this project and are delighted that this initial phase of what will be an unprecedented level of computing and data resources is now online for the nation’s researchers to use,” said Fran Berman, SDSC director and co-principal investigator of the TeraGrid project. “The TeraGrid is one of the foundations of cyberinfrastructure that will provide even more computing resources later this year.”

The computing systems that entered production this month consist of more than 800 Itanium-family IBM processors running Linux. NCSA maintains a 2.7-teraflop cluster, which was installed in spring 2003, and SDSC has a 1.3-teraflop cluster. The 6-teraflop, 3,000-processor HP AlphaServerSC Terascale Computing System (TCS) at PSC is also a component of the TeraGrid infrastructure.

“The launch of the National Science Foundation’s TeraGrid project provides scientists and researchers across the nation with access to unprecedented computational power,” said David Turek, vice president of Deep Computing with IBM.”Working with the NSF, IBM is committed to the continued development of breakthrough Grid technologies that benefit our scientific/technical and commercial customers.”

Allocations for use of the TeraGrid were awarded by the NSF’s Partnerships for Advanced Computational Infrastructure (PACI) last October. Among the first wave of researchers to use the TeraGrid are scientists studying the evolution of the universe and the cleanup of contaminated groundwater, simulating seismic events, and analyzing biomolecular dynamics.

Among the allocations awarded included one for Caltech physicist Harvey Newman . Newman leads a team of investigators who are developing codes for CERN’s Compact Muon Solenoid (CMS) collaboration. The CMS experiment will begin operation at the Large Hadron Collider (LHC) in 2007. The Caltech team’s planned use of the TeraGrid will be a valuable and possibly critical factor in the success of several planned “Data Challenges” for CMS. These Challenges are designed to test the readiness of the global Grid-enabled computing system being developed for the experiment, in collaboration with partner projects such as PPDG, GriPhyN, iVDGL, DataTAG, LCG, and others. The TeraGrid will further a program of developing optimized search strategies for the Higgs particles, thought to be responsible for mass in the Universe, for super-symmetry, and for investigating new physics processes beyond the Standard Model of particle physics.

To learn more about the TeraGrid, go to www.teragrid.org

Cray Inc. today announced that it, together with New Technology Endeavors, Inc. , have signed an agreement with the Defense Advanced Research Projects Agency (DARPA) to participate in the second phase of DARPA’s High Productivity Computing Systems Program. The program will provide Cray and its university research partners with $49.9 million in additional funding over the next three years to support an advanced research program aimed at developing a commercially available system capable of sustained performance in excess of one petaflops (a million billion calculations per second).

Ed Upchurch as Caltech/JPL ‘ s Principal Investigator leads the team and Thomas Sterling is Chief Technologist . Other research partners include Stanford University  and The University of Notre Dame , and the Principal Investigator of the project is Cray’s chief scientist, Burton Smith . “The DARPA HPCS program in general and the Cray Cascade Petaflops computer project in particular is an important opportunity for Caltech and JPL to directly, significantly, and substantively influence the direction of future supercomputer systems architecture and software ,” Sterling says. ” Through this program, innovative concepts developed by scientists at Caltech and JPL, in collaboration with colleagues at the University of Notre Dame, in the field of advanced PIM architecture will be developed and transferred to real world end users from academic research through this industrial partnership. The result may be little less than the next revolution in supercomputing.”

DARPA formed the High Productivity Computing Systems Program to foster development of the next generation of high productivity computing systems for both the national security and industrial user communities.  Program goals are for these systems to be more broadly applicable, much easier to program and more resistant to failure than currently available high performance computing systems.

Five computer-makers, including Cray, were selected for the first phase concept study that was initiated in mid-2002, and all five firms submitted proposals for the second phase. Cray, along with IBM and Sun Microsystems, were selected to continue to the second phase, where further definition and validation of the proposed systems will occur. In mid-2006, DARPA plans to select up to two vendors for the final phase, a full-scale development phase with initial prototype deliveries scheduled for 2010.

More information about the Cascade project can be found at http://www.cray.com/cascade/ .