One of the first proton proton collisions seen in the CMS Detector, displayed using the collaboration's software tool "Fireworks"

First beam circulated in the world’s most powerful particle accelerator the Large Hadron Collider (LHC) at CERN on 20 November 2009 – a clockwise circulating beam was established at ten o’clock that evening, followed by a circulating beam in the other direction a few hours later. When the proton beams are made collide at the centres of each of the four LHC experiments, the electronic data captured from the detectors will flow at rates ranging from a few hundred MBytes/sec to over one GByte/sec.

Global transport and analysis of this imminent stream of physics data is one of the major computing and networking challenges facing particle physics experiments. Leading edge explorations such as these require advances in all system components, from detectors to remote data analysis. CACR research staff have been involved with demonstrating technologies that reliably deliver over 100 Gb/s sustained from worldwide sources to a single analysis point. CACR also hosts a major “Tier2″ computing center, which is dedicated to receiving LHC datasets over twin 10Gbps networks from CERN, and running applications that analyse the events they contain.

• More Information on LHC First Physics here: http://press.web.cern.ch/press/lhc-first-physics/)
• More information on High Energy Physics projects at CACR: http://www.cacr.caltech.edu/main/?page_id=133)
• More CACR news items tagged “LHC”: http://www.cacr.caltech.edu/main/?tag=lhc)

Caltech’s exhibit at SC08 by the CACR and the High Energy Physics (HEP) group demonstrated new applications and systems for globally distributed data analysis for the Large Hadron Collider (LHC) at CERN, along with Caltech global monitoring system MonALISA and its collaboration system EVO (Enabling Virtual Organizations), together with near real-time simulations of earthquakes in the Southern California region, experiences in time-domain astronomy with VOEventNet and Google Sky, and recent results in multiphysics multiscale modeling with the PSAAP project.

A highlight of the exhibit was the HEP team record-breaking demonstration of storage-to-storage data transfers over wide area networks from a single rack of servers on the exhibit floor. The team’s demonstration of “High Speed LHC Data Gathering, Distribution and Analysis Using Next Generation Networks” achieved a bidirectional peak throughput of 114 gigabits per second (Gbps) and a sustained data flow of more than 110 Gbps among clusters of servers on the show floor and at Caltech, Michigan, CERN (Geneva), Fermilab (Batavia), Brazil (Rio de Janiero, Sao Paulo), Korea (Daegu), Estonia, and locations in the US LHCNet network in Chicago, New York, Geneva, and Amsterdam.

The image shows a sample of the results obtained at the Caltech booth, monitored by MonALISA, flowing in and out of the servers at the booth. The feature in the middle of the graph is the result of briefly losing the local session at SC08 driving some of the flows.

Read more in the Caltech Press Release

The first beam in the Large Hadron Collider at CERN was successfully steered around the full 27 kilometers of the world’s most powerful particle accelerator at 10h28 September 10. This historic event marks a key moment in the transition from over two decades of preparation to a new era of scientific discovery. (Read the full story on the Caltech website)

The LHC experiments are using a globally distributed “Tiered” grid of computational resources to process the proton collision data. This multi-Tier design was proposed and prototyped by Caltech in the late 90s, and is now universally adopted. The prototype cluster in the CACR machine room has been continually expanded and enhanced since then, and it is now one of the major Tier2 centres in the world, with over 1M SPECInt2k of compute power, several hundred Terabytes of storage space, and multiple 10 Gigabit network connections to other LHC sites in the US, and to CERN. First data from the initial LHC tests has already arrived for processing and analysis on the Caltech Tier2.

Local Press, featuring commentary and an interview with CACR Principal Computational Scientist Julian Bunn:

• Audio: AirTalk (KPCC)
• Video: ABC News (KABC)

For further information on the Large Hadron Collider, see the CERN website.

Based on exciting recent developments, the Caltech award is for the project called UltraLight, Bunn says. UltraLight was developed in 2004 in large part to support the decades of research that will emerge from the Large Hadron Collider (LHC) at CERN in Geneva, Switzerland. The project provides advanced global systems and networks, and this summer will start transferring data as the LHC becomes operational.

UltraLight exhibited its capabilities in a showroom demonstration for CENIC during a supercomputing conference in November 2007, sustaining disk-to-disk data transfers of up to 88 gigabits per second (Gbps) between Caltech and Reno, Nevada, for more than a day. But data flows from the LHC experiments will be the first time that UltraLight will strut its stuff for scientists hungry for data.

The CENIC Innovations in Networking awards are split into four categories, and this year for the first time CENIC declared a tie in Experimental/Developmental Applications between UltraLight and another contender, CineGrid, which facilitates the exchange of digital media over a network. Bunn will accept the trophy and present the group’s project at the CENIC 2008: Lightpath to the Stars conference in Oakland on Tuesday, March 11.

> Read more at the Caltech Press Release.

Click here for a full-sized version of the thumbnail above. Image shows a MonALISA plot of the aggregated network traffic to the Caltech booth, during and after the Bandwidth Challenge. (The initial blue region at the left of the graph is the BWC entry.)

An international team of physicists, computer scientists, and network engineers led by the California Institute of Technology, CERN, and the University of Michigan with partners at the University of Florida and Vanderbilt, as well as participants from Brazil and Korea, joined forces to set new records for sustained data transfer between storage systems during the SC06 Bandwidth Challenge.

The high-energy physics team’s demonstration of “High Speed Data Gathering, Distribution and Analysis for Physics Discoveries at the Large Hadron Collider” achieved a peak throughput of 17.77 gigabits per second (Gbps) between clusters of servers at the show floor and at Caltech. Following the rules set for the SC06 Bandwidth Challenge, the team used a single 10-Gbps link provided by National Lambda Rail that carried data in both directions. Sustained throughput throughout the night prior to the bandwidth challenge exceeded 16 Gbps (or two gigabytes per second) using just 10 pairs of small servers sending data at 9 Gbps to Caltech from Tampa, and eight pairs of servers sending 7 Gbps of data in the reverse direction. (Read more in the Caltech Press Release)

PITTSBURGH, Pa. — For the second consecutive year, the High Energy Physics team of physicists, computer scientists and network engineers led by the California Institute of Technology and their partners at the Stanford Linear Accelerator Center (SLAC), Fermilab, CERN and the University of Florida, as well as international participants from the UK (University of Manchester, UCL and UKLight), Brazil (Rio de Janeiro State University, UERJ, and the State Universities of Sao Paulo, USP and UNESP) and Korea (Kyungpook National University, KISTI) joined forces at the Supercomputing 2004 (SC04) Bandwidth Challenge to capture the Sustained Bandwidth Award. Their demonstration of High Speed TeraByte Transfers for Physics achieved a throughput of 101 gigabits per second (Gbps) to and from the show floor, which exceeds the previous year’s mark of 23.2 Gbps, set by the same team, by a factor of more than four. The record data transfer speed is equivalent to downloading three full DVD movies per second, or transmitting all of the content of the Library of Congress in 15 minutes. It also has been estimated to be approximately 5% of the total rate of production of new content on Earth during the test.

The new mark, according to Bandwidth Challenge (BWC) sponsor Dr. Wesley Kaplow, V.P. of Engineering and Operations for Qwest Government Services, exceeded the sum of all the throughput marks submitted in the present and previous years by other BWC entrants. The extraordinary achieved bandwidth was made possible in part through the use of the FAST TCP protocol developed by Professor Steven Low and his Caltech Netlab team. It was achieved through the use of seven 10 Gbps links to Cisco 7600 and 6500 series switch routers provided by Cisco Systems at the Caltech Center for Advanced Computing (CACR) booth, and three 10 Gbps links to the SLAC/Fermilab booth. The external network connections included four dedicated wavelengths of National LambdaRail, between the SC2004 show floor in Pittsburgh and Los Angeles (two waves), Chicago, and Jacksonville, as well as three 10 Gbps connections across the Scinet network infrastructure at SC2004 with Qwest-provided wavelengths to the Internet2 Abilene Network (two 10 Gbps links), the TeraGrid (three 10 Gbps links) and ESnet. 10 Gigabit Ethernet (10 GbE) interfaces provided by S2io were used on servers running FAST at the Caltech/CACR booth, and interfaces from Chelsio equipped with transport offload engines (TOE) running standard TCP were used at the SLAC/FNAL booth. During the test, the network links over both the Abilene and National Lambda Rail networks were shown to operate successfully up to 99 percent of full capacity.

The Bandwidth Challenge allowed the scientists and engineers involved to preview the globally distributed Grid system that is now being developed in the US and Europe in preparation for the next generation of high energy physics experiments at CERNs Large Hadron Collider (LHC), scheduled to begin operation in 2007. Physicists at the LHC will search for the Higgs particles thought to be responsible for mass in the universe, supersymmetry, and other fundamentally new phenomena bearing on the nature of matter and spacetime, in an energy range made accessible by the LHC for the first time.

The largest physics collaborations at the LHC, CMS and ATLAS, each encompass more than 2000 physicists and engineers from 160 universities and laboratories spread around the globe. In order to fully exploit the potential for scientific discoveries, many Petabytes of data will have to be processed, distributed and analyzed. The key to discovery is the analysis phase, where individual physicists and small groups repeatedly access, and sometimes extract and transport Terabyte-scale data samples on demand, in order to optimally select the rare signals of new physics from potentially overwhelming backgrounds from already-understood particle interactions. This data will be drawn from major facilities at CERN in Switzerland, at Fermilab and the Brookhaven lab in the U.S., and at other laboratories and computing centers around the world, where the accumulated stored data will amount to many tens of Petabytes in the early years of LHC operation, rising to the Exabyte range within the coming decade.

Future optical networks, incorporating multiple 10 Gbps links are the foundation of the Grid system that will drive the scientific discoveries. A hybrid network integrating both traditionally switching and routing of packets, and dynamically constructed optical paths to support the largest data flows, is a central part of the near-term future vision that the scientific community has adopted to meet the challenges of data intensive science in many fields. By demonstrating that many 10 Gbps wavelengths can be used efficiently over continental and transoceanic distances (often in both directions simultaneously), the high energy physics team showed that this vision of a worldwide dynamic Grid supporting many Terabyte and larger data transactions is practical.

While the SC2004 100+ Gbps demonstration required a major effort by the teams involved and their sponsors, in partnership with major research and education network organizations in the U.S., Europe, Latin America and Asia Pacific, it is expected that networking on this scale in support of the largest science projects (such as the LHC), will be commonplace within the next three to five years.

The network has been deployed through exceptional support by Cisco Systems, Hewlett Packard, Newisys, S2io, Chelsio, Sun Microsystems and Boston Ltd., as well as the staffs of National LambdaRail, Qwest, the Internet2 Abilene Network, CENIC, ESnet, TeraGrid, AMPATH, RNP and the GIGA project, as well as ANSP/FAPESP in Brazil, KAIST in Korea, UKERNA in the UK, and the Starlight international peering point in Chicago. The international connections included the LHCNet OC-192 link between Chicago and CERN at Geneva, the CHEPREO OC-48 link between Abilene ( Atlanta), FIU ( Miami) and Sao Paulo, as well as an OC-12 link between Rio de Janeiro, Madrid, Geant, and Abilene ( New York). The APII-TransPAC links to Korea also were used with good occupancy. The throughputs to and from Latin America and Korea represented a significant step up in scale, that the team members hope will be the beginning of a trend towards the widespread use of 10 Gbps-scale network links on DWDM optical networks interlinking different world regions in support of science, by the time the LHC begins operation in 2007. The demonstration and the developments leading up to it, were made possible through the strong support of the U.S. Department of Energy and the National Science Foundation, in cooperation with the agencies of the international partners.

As part of the demonstration, a distributed analysis of simulated LHC physics data was done using the Grid-enabled Analysis Environment (GAE) developed at Caltech for the LHC and many other major particle physics experiments, as part of the Particle Physics Data Grid (PPDG), GriPhyN/iVDGL and Open Science Grid projects. This involved the transfer of data to CERN, Florida, Fermilab, Caltech, UC San Diego, and Brazil for processing by clusters of computers, and finally aggregating the results back to the show floor to create a dynamic visual display of quantities of interest to the physicists. In another part of the demonstration, file servers at the SLAC/FNAL booth, in London and Manchester also were used for disk to disk transfers from Pittsburgh to the UK. This gave physicists valuable experience in the use of the large distributed datasets and computational resources connected by fast networks, on the scale required at the start of the LHC physics program.

The team used the MonALISA (MONtoring Agents using a Large Integrated Services Architecture) system developed at Caltech to monitor and display the real-time data for all the network links used in the demonstration, as illustrated in the figure. MonALISA ( http://monalisa.caltech.edu ) is a highly scalable set of autonomous self-describing agent-based subsystems which are able to collaborate and cooperate in performing a wide range of monitoring tasks for networks and Grid systems, as well as the scientific applications themselves. Detailed results for the network traffic on all the links used are available at : http://boson.cacr.caltech.edu:8888/ .

The team hopes this new demonstration will encourage scientists and engineers in many sectors of society to develop and plan to deploy a new generation of revolutionary Internet applications. Multi-gigabit/s end-to-end network performance will lead to new models for how research and business is performed. Scientists will be empowered to form “virtual organizations” on a planetary scale, sharing in a flexible way their collective computing and data resources. In particular, this is vital for projects on the frontiers of science and engineering, in data intensive fields such as particle physics, astronomy, bioinformatics, global climate modeling, geosciences, fusion, and neutron science.

Harvey Newman, Professor of Physics at Caltech and head of the team said, “This is a breakthrough for the development of global networks and Grids, as well as inter-regional cooperation in science projects at the high energy frontier. We demonstrated that multiple links of various bandwidths, up to the 10 Gbps range can be used effectively over long distances. This is a common theme that will drive many fields of data intensive science, where the network needs are foreseen to rise from tens of Gbps to the Terabit/sec range within the next 5-10 years. In a broader sense, this demonstration paves the way for more flexible, efficient sharing of data and collaborative work by scientists in many countries, which could be a key factor enabling the next round of physics discoveries at the high energy frontier. There are also profound implications for how we could integrate information sharing and on-demand audiovisual collaboration in our daily lives, with a scale and quality previously unimaginable.

Les Cottrell, assistant director of SLAC’s computer services, said: The smooth interworking of 10GE interfaces from multiple vendors, the ability to successfully fill 10Gbits/s paths both on local area networks (LANs), cross country and inter-continentally, the ability to transmit greater than 10Gbits/s from a single host, and the ability of TCP Offload Engines to (TOE) to reduce CPU utilization, all illustrate the emerging maturity of the 10Gigabit/second Ethernet market. The current limitations are not in the network but rather in the servers at the ends of the links, and their buses.

Further information about the demonstration may be found at:
http://ultralight.caltech.edu/sc2004 and http://www-iepm.slac.stanford.edu/monitoring/bulk/sc2004/hiperf.html

Scientists at the California Institute of Technology (Caltech) and the European Organization for Nuclear Research (CERN), along with colleagues at AMD, Cisco, Microsoft Research, Newisys, and S2io have set a new Internet2 land-speed record. The team transferred 859 gigabytes of data in less than 17 minutes at a rate of 6.63 gigabits per second between the CERN facility in Geneva, Switzerland, and Caltech in Pasadena, California, a distance of more than 15,766 kilometers. The speed is equivalent to transferring a full-length DVD movie in just four seconds.

The technology used in setting this record included S2io’s Xframe 10 GbE server adapter, Cisco 7600 Series Routers, Newisys 4300 servers utilizing AMD Opteron processors, Itanium servers, and the 64-bit version of Windows Server 2003.

The performance is also remarkable because it is the first record to break the 100 petabit meter per second mark. One petabit is 1,000,000,000,000,000 bits.

This latest record by Caltech and CERN is a further step in an ongoing research-and-development program to create high-speed global networks as the foundation of next-generation data-intensive grids.

Multi-gigabit-per-second IPv4 and IPv6 end-to-end network performance will lead to new research and business models. People will be able to form “virtual organizations” of planetary scale, sharing in a flexible way their collective computing and data resources. In particular, this is vital for projects on the frontiers of science and engineering, projects such as particle physics, astronomy, bioinformatics, global climate modeling, and seismology.

Harvey Newman, professor of physics at Caltech, said, “This is a major milestone towards our dynamic vision of globally distributed analysis in data-intensive, next-generation high-energy physics (HEP) experiments. Terabyte-scale data transfers on demand, by hundreds of small groups and thousands of scientists and students spread around the world, is a basic element of this vision; one that our recent records show is realistic.” Olivier Martin, head of external networking at CERN and manager of the DataTAG project said, “As of 2007, when the Large Hadron Collider, currently being built at CERN, is switched on, this huge facility will produce some 15 petabytes of data a year, which will be stored and analyzed on a global grid of computer centers. This new record is a major step on the way to providing the sort of networking solutions that can deal with this much data.”

The team used the optical networking capabilities of the LHCnet, DataTAG, and StarLight and gratefully acknowledges support from the DataTAG project sponsored by the European Commission (EU Grant IST-2001-32459), the DOE Office of Science, High Energy and Nuclear Physics Division (DOE Grants DE-FG03-92-ER40701 and DE-FC02-01ER25459), and the National Science Foundation (Grants ANI 9730202, ANI-0230967, and PHY-0122557).

About Caltech:

With an outstanding faculty, including three Nobel laureates, and such off-campus facilities as Palomar Observatory, and the W. M. Keck Observatory, the California Institute of Technology is one of the world’s major research centers. The Institute also conducts instruction in science and engineering for a student body of approximately 900 undergraduates and 1,000 graduate students who maintain a high level of scholarship and intellectual achievement. Caltech’s 124-acre campus is situated in Pasadena, California, a city of 135,000 at the foot of the San Gabriel Mountains, about 10 miles northeast of the Los Angeles Civic Center. Caltech is an independent, privately supported university. More information is available at http://www.caltech.edu .

About CERN:

CERN, the European Organization for Nuclear Research, has its headquarters in Geneva, Switzerland. At present, its member states are Austria, Belgium, Bulgaria, the Czech Republic, Denmark, Finland, France, Germany, Greece, Hungary, Italy, the Netherlands, Norway, Poland, Portugal, Slovakia, Spain, Sweden, Switzerland, and the United Kingdom. Israel, Japan, the Russian Federation, the United States of America, Turkey, the European Commission, and UNESCO have observer status. For more information, see http://www.cern.ch .

About the European Union DataTAG project:

The DataTAG is a project co-funded by the European Union, the U.S. Department of Energy, and the National Science Foundation. It is led by CERN together with four other partners. The project brings together the following European leading research agencies: Italy’s Istituto Nazionale di Fisica Nucleare (INFN), France’s Institut National de Recherche en Informatique et en Automatique (INRIA), the UK’s Particle Physics and Astronomy Research Council (PPARC), and Holland’s University of Amsterdam (UvA). The DataTAG project is very closely associated with the European Union DataGrid project, the largest grid project in Europe also led by CERN. For more information, see http://www.datatag.org .

Contact: Robert Tindol (626) 395-3631 tindol@caltech.edu