By Joshua Reynolds, SHI International
ITAK V10 I4
Ghost in the Machine
The vast majority of IT environments are haunted. Large scale infrastructures, by virtue of their operational requirements, value high capacity and high availability over asset management. This inevitably means that there are “ghost” assets lurking in most environments – devices whose purpose withered and passed on some time ago but were not removed or repurposed. Still connected to power and probably a network, they serve no material business purpose; they simply sit, absorbing space, power and resources. It’s a well-known fact that decommissioning ghost servers saves money on utility bills and the cost of datacenter space. However, these wraiths also embody a potentially much more serious risk: they are a software and regulatory compliance exposure.
The term ghost asset can encompass the hardware, software, maintenance value as well as any supporting systems that might be needlessly consumed by assets that no longer have a meaningful contribution to make. Power management, facilities maintenance, middleware, storage, backup, DR – secondary resources consumed by a ghost add to its overall cost. When ghost assets negatively impact compliance, the cost they represent increases exponentially.
What are ghost assets? They were once production or test systems but have dropped out of focus. They are rogue machines, falling outside the spectrum of active management and often effectively invisible to daily IT operations. Ghost assets present a risk because they aren’t taken into account when determining compliance.
These eidolons generate a wide variety of financial risks to an organization: from the purchase of redundant assets to manufacturer audit penalties to fines imposed due to regulatory compliance violations. A few of the most common examples are below:
Manufacturer compliance risk: Licenses that originally covered ghost assets may have been “reallocated” to other systems under the mistaken impression that they’re no longer in use. In an audit there is no exemption for ghost assets. Large enterprises usually have volume licensing contracts containing specific terms that override the normal EULA; there may be restrictions on usage/redeployment of existing licenses that wouldn’t be apparent prima facie. Risk is compounded in this case: multiple products from multiple manufacturers are not only over-deployed, they’re deployed in violation of use rights.
Overspending risk: Nearly as undesirable as being under licensed, ghost assets can just as easily trigger overspending on software; net new purchases are made when existing assets could be repurposed.
State and/or Federal regulatory risk: In some verticals these phantoms can have a substantial effect on regulatory compliance with governing structures like the SEC and HIPAA; the fines imposed for non-compliance are considerable.
Security and vulnerability risks: From a security perspective, these devices create risk exposure in many ways. Are they internet accessible? Are they behind the same firewalls and network security layers as the rest of the environment? Is sensitive data accessible through them? Could a ghost asset also be a back door to the underworld?
Accounting/Company Valuation: Ghost assets and the value of the software on them can significantly affect the accuracy of calculations related to Sarbanes–Oxley compliance, insurance valuation and tax reporting.
Abandon All Hope Assets Who Enter Here
If there is so much value in these assets how are they so easily lost? From sepulchral server farms to phantom PCs and laptops entombed in storage closets and desk drawers, there are countless ways assets become ghosts. One of our customers calculated that ghost assets were costing them $1.7 million per month. How can over twenty million dollars a year just vanish? Here are some of the most common scenarios we see every day.
Migrations: The primary goal of a migration is getting essential systems up and running quickly and reliably. A ghost town of servers can be left behind – often with no defined plan to decommission or repurpose them.
Mergers/Divestitures: Devices that are part of a merger or acquisition typically fall into two categories; the infrastructure that encompasses the business value being acquired, and “everything else.” “Everything else” systems that are never incorporated into the primary environment are consigned to purgatory. Divestiture, the other side of Charon’s coin, can be just as tricky. Devices not sold off during a divesture no longer have a clear purpose and end up alongside the “everything else” assets, trapped on the wrong side of the river Styx.
Project or Product Specific: Servers and software are routinely purchased expressly to serve a specific project or product. The project completes or the product falls out of favor and there is no clear follow up plan for repurposing the assets acquired. New projects and products spur the purchase of more assets and the cycle continues. As the business moves forward ghost assets appear in its wake.
Legacy Compatibility: Older – often antiquated – systems often remain in an environment because a previously business critical product or system was incompatible with more current technology. Telnet based financial transaction systems, manufacturing interfaces, custom built mainframes and similar specialty systems easily become zombies – present but devoid of purpose. Often, the personnel that managed those systems are no longer with the company.
A customer I worked with a few years ago had two rooms full of mainframes and legacy servers we discovered weren’t connected to anything but power. No one in their IT department could explain what function those assets served or who was responsible for them. They’d always just “been there.” No one had touched them for fear of breaking something they didn’t understand and might not be able to fix.
Lack of Dependency Mapping: Fear is one of the most prevalent drivers in the creation ghost assets; how many thousands of servers are out there because no one knows what would happen if they were removed? Dependency mapping is often overlooked at many organizations because, frankly, it’s very difficult to do. Application dependency mapping is particularly labyrinthine, creating Visio diagrams that look like they sprang from the imagination of M.C. Escher.
The resources you commit to create and maintain dependency maps are resources well spent. They play a vital role in “what if” planning which is in turn critical to understanding what aspects of your environment serve what business needs. Using dependency maps will uncover far more ghost assets (and savings and risk mitigation) than would be found without them.
Exorcizing the Specter
Tools are not the answer – they’re only an element of it. There is no such thing as a one shot, out of the box, works perfectly and does everything product, though many companies claim to provide such a solution. These products are open-structured and highly configurable but require tremendous time, effort and expertise to set up and maintain. Many customers find that they can’t fully leverage the capability of these products without hiring a dozen or more subject matter experts or paying exorbitant rates for long term on-site consultants.
The Polaris team at SHI works with hundreds of customers per year and has analyzed millions of end points; we’ve pretty nearly seen it all. We believe ITAM should be thought of not as a tool but as process – one that encompasses tools, personnel, expertise and procedure. And, there are ways to return ghost assets to their graves and eliminate the risks they pose.
1. Data is Legion
Most ITAM projects start with the usual suspects: configuration management data and purchasing records. You won’t find ghost assets there; they exist in the twilight area outside of daily use systems. They’re identified by inference, as astronomers identify black holes by the effect they have on nearby stars and on light that passes near them.
Broaden the spectrum of data collection. Virtualization systems, scripting, agents, infrastructure systems, invoices, manufacturer records, licensing contracts, personnel records, and so on are all viable sources of data. The more robust the data the more comprehensive the results are. For example: if a delivery manifest from last year lists 200 servers but SCCM only reports 165, some ghost hunting may be needed.
Involve the proper stakeholders, but do so with the understanding that insight can come from places you might not expect. IT administration, product custodians, project managers, procurement, accounting, HR, facilities management, shipping and receiving are all potential sources of information. For example: There might bill of lading from the loading dock for a truckload of recycled assets that haven’t been purged from your asset tracking CMDB.
2. Iterate! Improve! Iterate! Improve! It’s important to keep going – repeat this process over and over again. Subsequent iterations will be more accurate, more automated and more useful. The unreconciled, unreported, unmanaged, and unnoticed will be found. Ghosts will be put to rest. Over time, you’ll see ways to improve all aspects of lifecycle management. Also, bear in mind small adjustments can make a big difference. Moving a couple of clusters that were misidentified as production to DR can profoundly affect your compliance position.
Walk toward the Light
Building an IT Asset Management program that is comprehensive enough to prevent assets from becoming ghosts takes time and expertise but is absolutely possible. Combining reliable, stable and automated data collection methods with appropriate, enforced and practical processes lets you be confident that there are no poltergeists in your cloud, no banshees in your datacenters and no phantasms in your storage closets.