The Caltech Tier2 Center located at the Center for Advanced Computing Research (CACR) is an important medium-scale facility for research in grid computing, high performance networking and data analysis needed for the highly data-intensive LHC physics programme. The center is a fully participating component of various grid projects that include OSG, GriPhyN, iVDGL, and PPDG. At present, the Tier2 Center consists of 4 clusters: OSG production grid clusters (for large-scale simulation and reconstruction batch use, two integration testbed (ITB) clusters for testing and developing the grid infrastructure, and an interactive cluster for CMS Monte Carlo production and Caltech-based analysis.
Center for Advanced Computing Research at the Powell-Booth Laboratory
The Caltech Tier2 is running a variety of services for monitoring, data publishing, mapping virtual organization users to local accounts and others: MonALISA, GUMS, PHeDEX+PubDB, dCache/PNFS, IEPM, Clarens+IEPM, OSG-Discovery, and LambdaStation. It is actively used by researchers from several virtual organizations (USATLAS, USCMS, SDSS, LIGO and others), Caltech physicists, MonALISA and Clarens developers, and the IEPM group at SLAC and FNAL.
MonALISA monitoring quantities in the display next to the Tier2 cluster
Networking components include two LAN connections to CACR, another LAN connection to Caltech ITS (Information Technology Services), two 10GbE waves to Los Angeles, a Cisco 6509E switch for Ultralight high performance network research, and a Force10 E600 switch for connecting nodes of production cluster to Ultralight network. All the data transfer operation in the PHeDEx topology takes place through Ultralight network.
Physically, the Tier2 clusters are housed in eight 45U tall racks.
The following sections provide more detail about the individual sub-clusters in the Caltech Tier2.
OSG Production Cluster (CIT_CMS_T2)
This is our main cluster that is one of the core OSG sites and is fully engaged in serving grid community for their simulation and analysis tasks on 24x7 basis.This cluster is used for remote data operation for grid based analysis with tools like CRAB. A dCache storage element has been deployed in this cluster with SRM interface. Since most of the nodes also run GridFTP servers, data transfers takes place through least loaded nodes decided by SRM door. This cluster will be used for upcoming CMS Service Challege 4 (SC4) and CSA06. We will gradually ramp up the resources on this cluster and be fullly ready for stable operation in the time of LHC startup. Currently, there are 66 nodes in this cluster constituing approximately 254 KSPECInt2000 computing capabilty. It has 34TB resilient dCache storage space for persistent databases. There is approximately 2TB of Network Attached Storage for various OSG related directories and 45GB on each node's local partitions for serving local I/O need by running jobs. This cluster is the combination of dual and single core Opteron nodes and dual cpu hyperthreded Xeon nodes.A dual cpu, dual-core 2.2GHz Opteron node has been used for cluster head node. Two Foundry 48-port gigabit switches interconnected with 10GBE link have been used for cluster private network. We used customized Rocks3.3 cluster management software for contructing this clusters based on SLC3 compatible OS. All the nodes are running 2.4.21-32.0.1.EL.cernsmp optimized kernel. Since most of the jobs submissions and collection of output takes place through Grid, a very few system/user accounts are created on this cluster.
Rear view of some racks of the Tier2 cluster with the power and network connections
Integration Testbed Clusters (CIT_ITB_1 and CIT_ITB_2)
The Tier2 Center also hosts two small clusters, CIT_ITB_1 (Xeon) and CIT_ITB_2 (Opteron). Each cluster consists of one head node and one worker node. As part of the OSG release integration activity, these clusters are used to test new releases prior to their installation on the production clusters.
Analysis cluster (tier2c)
This cluster is dedicated to CMS Monte Carlo production and interactive analysis by Caltech users. No grid services are provided. The cluster consists of 7 nodes, one headnode and 6 compute nodes, interconnected by an Asante Gigabit switch. The headnode is a 4U system with dual Xeon 3.06 GHz CPUs and 4 GB RAM. The 1U Compute nodes are 2.4 GHz Xeon processors with 1 GB RAM and dual Gigabit ethernet ports. The cluster has 2.5 TB NAS storage space.
