Expanding HPC & Research Computing - The Sustainable Way

• By Mary Grush
• 09/01/09

How one university is creating cost-effective growth models for data center and resource expansion that are also environmentally and community-friendly.

INCREASED DEMANDS FOR RESEARCH and high-performance computing-- along with growing expectations for cost and environmental savings-- are putting new strains on the campus data center. More and more, CIOs like the University of Notre Dame's (IN) Gordon Wishon are seeking creative ways to build more sustainable models for data center and resource expansion. Wishon is not only the CIO, but he's also associate vice president and associate provost at Notre Dame, indicating the central role he plays in supporting the data and resource needs of both administrative and educational programs, as well as, increasingly, the university's strategic effort to grow the size and scale of its research initiatives.

Recently, CT spoke with Wishon about efforts underway at Notre Dame to respond to the upswing in HPC and research computing at the university while reducing costs and the environmental impact of program growth.

Campus Technology: As CIO at Notre Dame, are you seeing increasing demands on your data center, especially from HPC and research computing?
Gordon Wishon: The strategic objectives of the institution [to grow its research programs] are changing the nature of the demands on the IT organization, and we are facing these challenges in an era of significant economic challenges. At the same time, we're trying to focus on sustainability and preserving the environment. This all places a significantly different and sometimes new set of challenges on the IT organization, as it does at most educational institutions.

As our research program grows we are anticipating significant growth and demand for data center-provided services. Even if we didn't have such a focus on expanding our research programs, we would continue to see growth in that demand, driven by the need to continually update and improve the productivity of our administrative processes and the services that we provide to our students and our faculty.

[Bottom line,] we are anticipating an increase in demand on our data centers. Handling the growth in a way that is both cost-effective and sensitive to environmental concerns is front and center for us right now.

CT: Is IT support centralized at Notre Dame?
Wishon: Historically we have been much more centralized in our IT support model and strategy than most research universities, such as my previous institution, Georgia Tech, where the responsibility for IT service and support was very highly decentralized. In recent years there has been a growing trend [in higher education] back toward centralization, even at those institutions that are highly decentralized. There's a recognition now by the deans and the heads of academic units of the true cost of providing IT services for their faculty, particularly services that are increasingly required to be administered in a more professional way so as to address concerns about security and risk to the institution.

At Notre Dame we are hopeful about developing a support posture that effectively captures the best of both centralized and decentralized environments. I'm currently collaborating with our vice president for research, as well as the deans, to determine the best support model going forward.

CT: And how does server virtualization fit into the picture?
Wishon: Virtualization, of course, is a key technology that we are using to address some of [our challenges]. We've done some work to estimate the growth of demand for services, and the growth of the number of server platforms within our data center. Our projections suggest that at the current rate of growth over the past two years-- even without factoring in a significant increase in demand driven by research-- by 2010 we will see a growth in the number of servers on our data center floor from the current 558 to more than 700. If we were to meet all these growth requirements with the former architectural approach of standing up individual servers and application suites to support each new application, we would have 700 individual platforms or clusters of platforms, with the concomitant increase in power consumption and cooling requirements.

Two years ago, we began to experiment with virtualization in our data center, and in fact, of the 558 servers that we have on our data center floor today, over the course of the past two years we've actually reduced the number of physical servers to about 360. So, today, the 558 servers are distributed across the 360-or-so physical platforms, which represents about a 34 percent virtualization rate.

CT: What will you gain from doing that?
Wishon: What we're hoping to achieve as we grow to the projected 700 servers by 2010 is a 50 percent virtualization rate. If we can get more aggressive with our virtualization as we continue to grow, we hope to meet this growth demand of 700 servers without substantially increasing the number of physical platforms on the floor beyond the number we have in place today.

That has some obvious effects on reducing power consumption and the demands for air handling and cooling. If we can achieve a 50 percent rate of virtualization by 2010, we will effectively flatline our growth in consumption of power and cooling, with the effect of avoiding over a million dollars in annual costs.

CT: So you will not only reduce growth of the number of physical servers on your data center floor through virtualization, but you'll also achieve some real net savings over time. Do you also have plans for using cloud services toward similar ends?
Wishon: In the longer term-- and I know some institutions are already heading in this direction-- rather than achieving our net savings through virtualization, we'll begin to promote and increasingly rely upon cloud solutions, or if you will, outsourced solutions that take advantage of much greater economies of scale. I happened to be talking to my counterpart at Arizona State recently, Adrian Sannier, and ASU is in fact hoping to reduce the need for a data center, or perhaps eliminate the need for a data center on campus altogether, by leveraging the cloud and cloud providers, especially for commodity services such as cycles and storage. Companies like Google, Amazon, Microsoft, EVault, and others are able to take advantage of economies of scale and ultimately reduce the demand for net power consumption and cooling requirements.

CT: Are there other strategies similar to virtualization that you are leveraging within the data center?
Wishon: Another opportunity in the data center is the use of technologies such as Oracle RAC [Real Application Clusters]. While Oracle RAC does not initially reduce costs or environmental impact, we believe that over 10 years it will significantly reduce the cost of ownership, thanks to reduced maintenance costs and higher availability-- because it's a much more scalable and manageable architecture. It also will help us to avoid adding more platforms in our storage environment, and in our database environment. So that's very similar to the way virtualization is allowing us to avoid adding more hardware in the data center.

CT: What are some examples of things you're doing to reduce environmental impact?
Wishon: We've attracted public interest about an effort where we're looking at ways to recapture waste heat. Historically, of course, the heat generated by computing platforms and servers in our data centers has just been exhausted to the atmosphere, thrown away. As server platforms are growing in use of blades and tacking on more chips and processing power into the same space, that's generating not only increased power consumption and cooling requirements, it's also generating additional heat. If we could find a way to capture waste heat, perhaps recycle it in some way, then once again we'd have a method of reducing the environmental impact and reducing our carbon footprint.

“If we achieve a 50 percent virtualization rate by 2010, we will effectively flatline our growth in power and cooling consumption, avoiding over a million dollars in annual costs.”

CT: So what are you doing to move in this direction?
Wishon:We have a couple of successful initiatives that have been coordinated by and were the brainchild of our Chief Technology Officer Dewitt Latimer and a scientist in our Center for Research Computing, Paul Brenner, who have helped the city of South Bend, [IN], to reduce some of its energy requirements by recycling waste heat.

In the first example, the city runs an arboretum/greenhouse facility that contains a display of southwestern American flora, that was a donation from a biologist here at Notre Dame a number of years ago. The winters in South Bend are particularly harsh, and the city was finding it increasingly difficult to heat that greenhouse display facility through the long winters, especially in the face of rising energy costs. So, since we have collaborated with the city to build out a metropolitan area network, it turns out that the arboretum facility sits within close proximity of some fiber that connects the university to parts of the city. We were able to simply take a rack of high-performance computing equipment, particularly dense computers that generated a lot of waste heat, and sit it right down in the greenhouse facility. So what was waste heat is now being used to help heat that greenhouse facility. That not only lowers the costs for the city of South Bend to heat the greenhouse, it also allows us to move that rack off of our data center floor, so we don't have the burden of cooling that particular suite of equipment.

We're looking at other ways to leverage this notion. We have another pilot project now with the city of South Bend and its sewage treatment facility, [also] coordinated by Latimer. Sewage treatment facilities try to raise the temperature of solid waste material to a certain point in order to generate bacterial activity that breaks the waste down into byproducts, one of which can be fertilizer. So, rather than hauling the material away to the landfill, the treatment facility can generate a revenue source-- provided it is able to raise the temperatures long enough and to the right levels to produce useful fertilizer. So we are engaged in an effort to capture waste heat from the data center, direct it over to the sewage treatment facility, and help the city to solve the waste treatment problem and offset some of its costs.

CT: It seems like IT's role is not only changing within the university, it's changing within the community as well.
Wishon: As we go forward, we're going to focus on awareness and education as well as our technology initiatives. I think we can all do much better than we have done in the past in addressing environmental concerns and our impact on the environment.

::WEBEXTRAS ::
Head to our Green Technologies solution center for the latest news, case studies, research, features, and more.