Poltergeist

The problem may not rival the movies Poltergeist or The Amityville Horror for sheer terror, but CIOs and data center managers are still well advised to deal decisively with so-called ghost servers. Like celluloid zombies, these forgotten pieces of equipment are dead when it comes to improving the bottom line, but they are very much alive when it comes to eating up IT budgets.The unproductive -- and usually undocumented -- servers take up valuable real estate, consume increasingly expensive electricity and, in some cases, absorb ongoing maintenance and lease payments. "You can find ghost servers in a lot of enterprises," says John Phelps, an analyst at Gartner Inc. "And the larger and more diverse the company, the harder it can be to have a single group or technology platform that provides control over all corporate assets." Sun Microsystems Inc., through case studies of two large corporate data center operations and anecdotal analysis of efforts with many customers, believes that 8% to 10% of all servers in large corporations have no identifiable function. In the two data center studies, 150 ghost servers were found in an installation of 1,800 servers, and 354 ghost servers were found in an installation of 3,500 servers. One of the companies studied was Sun itself. Sun used system performance tools to monitor CPU utilization and I/O and network traffic, collecting the data over the course of a month, and sorted out machines with zero utilization. Sun removed the questionable servers from operation for 90 days to determine any impact. At the end of the period, it found that 60% of the servers could be permanently decommissioned, says Mark Monroe, director of sustainable computing at Sun. The company now conducts quarterly reviews of utilization rates. "It's hard to get people to admit they have unused infrastructure," Monroe says. "It's expensive, wasteful, and having a CIO admit he's got millions of dollars of idle assets lying around could get a guy fired. I think we can remove some of the stigma by talking about the facts, and having people realize it's worse just to leave them lying around." Corporations need to admit "they are like everyone else" and try to reduce the number of idle machines to 3% or less, "which is three times better than the industry average," he says. The cost of running a server for three years exceeds its original acquisition cost, so keeping the ghost servers around has an easily measured effect in energy savings. Identifying and eliminating wasted resources are key components of green or "eco-computing," Monroe says. The good news: Advances in asset management and configuration software can help businesses find and shut down these useless machines. And more companies are being proactive when it comes to adopting policies in this area. A survey of more than 300 businesses in the U.S. and Europe conducted by Gartner in 2006 found that 74% of respondents said they have a formal software and hardware asset management program. More advanced autodiscovery capabilities and more efficient, effective and accurate search engines mean "there has recently been more focus on discovering the lost and unidentified equipment, especially in industries where security concerns are growing," says Richard Ptak, analyst at Ptak, Noel & Associates. "Every major company I've ever dealt with has had the problem to some level."
When BlueCross BlueShield of Florida (BCBSF), which provides insurance services for nearly 9 million people, decided to build a new data center in 2005, the company also felt it was time to get a comprehensive look at its asset inventory, says Paul Stallings, senior manager for provisioning services. In addition to the new and existing data centers, BCBSF also has about 10 smaller server rooms in various locations around the state. The server room equipment was being managed by different departments within the insurance provider, and each had a different way of tracking and documenting assets, Stallings says. Using Aperture Technologies Inc.'s VISTA tool beginning in late 2005, BCBSF began documenting its IT resources under a single umbrella. "There were definitely undocumented assets," Stallings says. "When you have seven or eight teams managing the hardware, each using a different process, there was really no way to get a systemwide understanding of what we had in our various data centers." By getting a complete view of its asset inventory, BCBSF has been able to create a formal decommissioning process. When a system needs to be retired, a workflow ticket is automatically generated. The IT department can then start to reclaim floor space, remove cabling and place old servers in pool of equipment that can be either redeployed or completely removed from the environment. As one of largest IT services providers in the world, Fujitsu Services operates seven Fujitsu-owned data centers in the U.K. with more than 200,000 sq. ft. of floor space, and manages 11 customer-owned facilities. As a result of its outsourcing business, the company has acquired data center facilities and IT equipment from a variety of sources, and much of the equipment came with no asset or configuration management process in place, says Mark Scott, global data center delivery manager at Fujitsu Services. The company had previously used a configuration cataloging system that recorded only the location of equipment on the floor, but it did not provide any insight into specific use of the equipment. In attempt to get a comprehensive inventory and create an asset management strategy that would allow it to maximize existing data center space and continue to grow, Fujitsu Services began working with Aperture to deploy its VISTA data center resource management system in 2005, and the company began to see a variety of issues that were wasting resources. "We found we had IT equipment on the floor that people within the company had thought we had gotten rid of years ago," he says. "We looked deeper and found out we were also still paying lease and maintenance on the equipment. We even found that we were paying lease and maintenance on equipment that been removed from our data centers." By extending the asset management systems across its entire U.K.-based data center operation, Fujitsu Services has reduced its operational costs and then passed the savings on to customers, Scott says. Since customers are generally charged for the floor space they use within the data centers, the removal of unproductive equipment has allowed Fujitsu Services to reduce specific hosting charges for those customers. So, for example, if Fujitsu can reduce the footprint by 10%, Scott says that 10% savings can be passed along to the customer. The company is also developing the ability to invoice for the actual power required by individual customer installations, versus the current practice of invoicing for power based on the floor space used by the servers. "We also found lots of badly installed equipment," Scott says. The poor installation processes had led to what he characterized as vulnerabilities that meant his company had difficulties meeting customer uptime requirements. "The result was really that we had actually built vulnerabilities into the operation. It was a real wake-up call. We now have a complete vulnerability check and can control installations from the beginning to end." Alticor Inc., the parent company for such businesses as Amway Corp., Quixtar Inc. and Access Business Group LLC, handles IT demands for affiliates all over the world, says Randy Gast, supervisor of server technology at Alticor. The company uses a combination of management tools to keep track of its software and hardware assets, including BMC Software Inc.'s Remedy service management and Hewlett-Packard Co.'s Systems Insight Manager. Using the software, Alticor early last year found that more than 200 servers, or about a third of its 650 x86 processor-based servers, were running at utilization rates of 10% or less, Gast says. Even scarier, these underused servers had accumulated without IT knowledge, over the previous three years as new equipment was bought to handle individual applications or affiliate requirements. Working with virtualization specialist VMware Inc., Alticor has embarked on an effort to consolidate the unproductive system and increase overall utilization rates to between 60% and 70%, Gast says. To date, Alticor has consolidated 150 of the unproductive servers onto seven servers using virtualization software. Alticor has donated the majority of the servers it has taken out of operation during the consolidation effort to charitable organizations. Servers that were too old to be used by groups such as local schools and churches have been sold for scrap. "Using Remedy, we can track the lease agreements for our hardware, when they start and expire, and use it to have automatic triggers that let us know we need to consider replacing certain equipment," Gast says. That said, there is generally a great deal of inefficiency regarding how customers get rid of old gear. "A lot of businesses don't have a disposition strategy, primarily because it's easier to buy" new servers than dispose of the old ones, says Daniel Ransdell, general manager of IBM's Global Asset Recovery Solutions business unit. "Even if you've unplugged the equipment and stuck it in some closet or corner, the business is losing the opportunity to recoup value. Servers aren't like fine wine. They don't get better with age." Energy cost is the major issue that is changing attitudes inside corporations. In August, the Environmental Protection Agency published a report that documented data center power usage. According to the report, data centers in the U.S. consumed about 60 billion kilowatt-hours in 2006, or about 1.5% of total electricity consumption in the country. The EPA points out that data center energy use has doubled in the past five years, and is expected to double again in the next five years to an annual cost of about $7.4 billion. The EPA says existing technologies and strategies could reduce typical sever energy use by 25%, and even greater savings are possible using more advanced technologies. Rockwell Bonecutter, head of data center technology and operations at Accenture Ltd., believes that a large percentage of the ghost server problem was alleviated when most businesses engaged in extensive Y2k efforts within their infrastructure. In the intervening years, however, there has been a significant growth of systems that operate at 5% utilization or less, often because of poor communication and asset management within the company. "When it comes to servers that nobody knows about that are sitting for years and nobody has touched, there are probably examples in every IT environment, but it's obviously impossible to measure what you don't know exists," Bonecutter says. "What we have found is that it is not unusual to find that 40% of all servers on a floor could be consolidated and virtualized out of the environment." Consolidation through virtualization has also led to the new phenomena of virtual ghost servers. The ease and quickness with which virtual servers can be created can often leave servers cluttered with numerous poorly documented virtual machines created for short-term or abandoned projects. With tools allowing businesses to get a more holistic view of their assets and policies in place to guide a formal decommissioning process, businesses can now reduce the risk and associated costs of ghost servers, without the need to call on the aid of another great Hollywood institution, Ghostbusters.