David Cappuccio

One of the hidden beauties of any virtualized state is the clear disaggregation of hardware and software, the logical separation of traditionally tightly coupled environments, allowing us much greater flexibility in deciding what applications to run where, and for what reasons.  This flexibility in and of itself is a good thing, and we all benefit from it, but I suspect that if we’re not careful in how we use virtualization some apparently intelligent decisions today may in fact turn out to be significant problems in the future.

Take for example some of our legacy applications.  Not the ones we’ve been living with on mainframes or large Unix systems for the past 30 years, but legacy x86 applications.  You know, those early Windows applications written in C or C++ (or even early Java) which now run quietly every day on those older Windows NT or Windows 2000 sever platforms.  Like many older applications from the Big Iron days, these are often poorly documented and not designed with the reusable constructs of SOA, but as stand alone, end to end systems.

In many companies these are clear targets for virtualization.  They often have stable performance characteristics (few peaks and valleys), allowing a high number of images per physical server.  Since they are older applications the platform itself is not likely to change and the amount of enhancements to the application have been minimized over the years.

As newer Operating Systems emerge and servers grow into 8 and more cores there is a risk that these legacy applications will suffer compatibility or performance problems and will require significant and costly retooling.  But with development budgets shrinking, or staying stable at best, higher priority projects will most often get the funding over legacy rewrites, so the prudent IT manager will keep these applications running on the older OS’s in a standardized virtual container as long as possible.  This costs very little, does not impact the performance or reliability of the application, and gives IT some breathing room before a large, and possibly complex rewrite begins.

Sounds like yet another side benefit of virtualization, and for the near term it certainly is, as emulating older environments on newer technologies has been practiced for years, and the financial benefits are obvious. 

However, as the underlying operating systems get older, updates, patches, and support begins to get marginalized by vendors, and eventually formal maintenance support from the vendor will reach an end.  At this point enterprises will have another difficult decision to make – to continue emulation of an older application on an unsupported platform, or to begin the replacement (or rewrite) process for the application.  Either choice is fraught with risks, costs and business impact, and in many organizations this will not be a single decision, because as we continue this march towards virtualization the number of legacy applications marginalized into self contained environments will continue to grow.  

So what are the impacts?  In a best case scenario this is just the ranting of a cynical naysayer and we will continue to upgrade and improve applications as needed, and migrate them to newer platforms when appropriate.  It’s a non-issue.  But the alternate world view is that the beauty of virtualization is that once an environment is created it can run “as is” indefinitely, with little or no attention from the outside.  If this happens IT will almost always focus on near term issues, because these issues are what drive us – after all we are a reactive crowd – and when budgets are always tight the funding to fix what isn’t broken rarely materializes. 

In the second world view we may find ourselves scrambling to update scores or even hundreds of these applications to run on newer platforms when we are least prepared to do so.  This is similar to Y2K in some respects in that these problems are not caused by lack of knowledge or awareness, but just by years of pushing the issue into that low priority bucket that’s so convenient to use during the planning cycle. 

Just a thought……

This post is response to a question from Allison relating to the series of questions that need to be asked before embarking on a data center build project.

Where to begin?

In my first post of this series Allison sent along a quick note with an interesting observation;  “I found this article very insightful, but it left me with one resounding question. Where do I begin? If I’m a company who thinks that a data center build might be in my future, who do I go to first?”

I realized when I read this comment that there is an underlying problem with almost all data center build projects, and that is the simple fact that these are one-off projects, only executed once a decade.  The person who ran the last one may have been wildly successful, but the likelihood that they documented all the steps, or are still around to share them with you are pretty slim.  Retaining the information gathered and lessons learned, and then documenting them for future reference, is almost never done.  So when someone like Allison (or any of us) gets volunteered to run one of these projects we begin the process by relearning everything that’s been done in past, both the good and the bad.   So, lesson 1;  document everything.

And to your point Allison on where to begin, let me offer this.  I would begin by creating a very small tiger team of perhaps only 5 or 6 people.  Early in the project these might even be part time positions, but the focus is on data gathering, idea generation and team building.  Members should include folks with skills in IT architecture, servers, storage, networking, facilities and building security.  The facilities folks would include disciplines in air conditioning, (focused on IT cooling), UPS and generators and power distribution. I would be looking for people who didn’t come into project with pre-conceived notions of what a data center should look like, because many bad designs have resulted from repeating the past (today’s well designed data center is radically different from older versions). 

Lastly, early on in the project I would be looking for a representative from either the PMO or an assigned Project Manager to begin tracking and linking deliverables, and also to begin documenting the expected desired state as input to the commissioning process.  Commissioning is a critical part of the build process, and the most efficient scripts built for the commissioning process begin early in the project.

As always, all comments and questions are welcome.

This post is the third in a series of questions that need to be asked before embarking on a data center build project.

How much energy will I need?

Great question.  Somebody always asks the obvious, especially when there is no way to definitively answer the question.  In the old days (you know, 8 or 10 years ago), a data center was built with a single power footprint – meaning a consistent watts per square foot design.  This was done historically with data centers because IT equipment generally followed a fairly obvious and linear energy consumption growth path and projecting future growth was at least feasible.   Not so much in today’s world.  Both server and storage energy growth has skyrocketed over the past 8 years due to the introduction of newer technologies (blades, 2u and 1u servers, higher density cores etc).  Couple that with a rapid rise in the use of virtualization and we have exceeded the physical capacity of the facilities mechanical and electrical equipment to deliver power to support IT loads in many data centers around the world.

So what can you do?  You can plan for the unknown (and I wish you luck with that boardroom funding request), or you can design for the most logical expected energy loads, but with the flexibility to shift loads and usage patterns as needed.  From a basic design point today a 3 zone structure might support high density racks at 10-12kW, medium density at 6-8kW and low density up to 5kW.  The size (percentage) of these zones obviously depends on you (your mileage will truly vary) but these zones can shrink or grow as workloads demand.

Ok, let’s say we’ve done that and 8 years from now things are getting a bit hot in the data center and power demands continue to rise, one option would be to refocus my cooling efforts away from forced air, especially as power requirements for equipment begin to exceed the 6 to 8kW per rack range.  Augmenting traditional air cooling with in-rack cooling, either water or refrigerant, will fee up a significant amount of power for other uses.  The general rule of thumb is that for each watt of power for IT load you should assume at least an additional watt of power for cooling the same equipment.  For low utilized equipment this could be less (below 4kW per rack), but as you move into the higher density zones above 8kW the ratio begins to shift towards cooling needing more power than IT load (sometimes as much as 2 to 1).   However, if you decide to cool inside the rack with refrigerant based systems the cooling to IT load ratio never changes – as air leaving the rack is approximately the same temperature as that going in and the room air conditioning workload does not change.

There are other options as well which we will explore in other posts, things like bringing in outside air instead of just using the mechanically cooled variety, or refocusing what loads actually need the UPS and which ones don’t.  The bottom line though is that with flexible designs, scalable PDU’s, and a willingness to use alternative cooling techniques the energy question is not quite as critical as one might think.

I’m thinking that, especially in IT, history has a habit of repeating itself. A friend and I were talking about the drivers and catalysts in IT the other day and the topic of Public Clouds came up.  He was convinced that the catalyst for the success of public cloud was going to be enterprise class (or even global class) service levels and that until they were realized by Cloud providers most companies would only dabble in public cloud services, and then only for non-strategic commodity type services (backup, archiving, email, etc).

The more I thought about it the more I began to realize that he was analyzing things as if logic and common sense had anything to do with the success of a market.  Yes, willing customers help drive a market, but the motivation isn’t always so obvious (anybody out there still have a Pet Rock?).   So I began thinking about the history of IT and realized that there is an interesting corollary to Public Cloud which might come into play here if we’re not careful. 

Back in the very early days of PC’s I remember a business unit coming to our IT staff and asking for approval to buy one of these new devices.  The request was quickly rejected for two, what we believed, very good reasons.  It was an expensive, non-standard device, and there was absolutely no additional business value to be gained, as all applications and IT services were already provided by the guru’s of the day – central IT (of Information Systems as we were called in the day).  What, we asked, were they going to do, play solitaire?  So this local geek (one of the first no doubt) from the business unit went away and found a way to buy a PC anyway, without our approval, and sure enough they did play solitaire – but they also found VisiCalc (or some variation) and soon built a small spreadsheet budget on it. This was a pretty radical way of using technology, and all without any controls (or delays) by central IT.   Soon another “non-essential” PC was bought and the two were quickly connected and voila, the department had a LAN, all out of the control of central IT, and the beginnings of distributed computing was launched. 

What’s interesting here is that IT knew what was going on, but chose to ignore it, because after all, these weren’t business machines, and if the user really wanted a new application they only had to request it (right?).  The catalyst for change became the perceived ease of use and freedom from IT, which quickly morphed into a rapid proliferation of machines, all out of the control of IT. 

Fast forward to today and you could easily argue that the consolidation projects over the past 10 years have been  a direct result of IT letting go of the reigns back then, so to speak.

So what does this have to do with Cloud you ask?  Two things I think.  First, the primary motivator I’m hearing from many customers is speed or ease of use.  If IT is so busy doing other projects (like consolidation, or trying to fix the power and cooling issues, or worrying about audit and compliance data) and is perceived as being less than responsive to the business units needs, these same business units will look elsewhere for answers – like to an external Cloud provider.  And the solutions need not be perfect – they just need to be good enough to solve the current problem, since we all understand that good enough is often just that – good enough.  And the other underlying reason is that in many companies today, that geek from the mid-80’s who branched away from IT with the first PC is more than likely to be a very senior person in the business, or the unit head themselves, and still has an inherent distrust of central IT, whether it’s justified or not.

If you don’t pay attention Public Cloud services may be adopted in your company, whether you’re ready or not.

Just a thought…..

I know, I know, the title alone can be misleading, but what I’m thinking here is a result of yet another trip down memory lane.   I was talking with a client at the IT Operations and Management conference this week and he was a huge VMware fan (as are many).  During the conversation he was completely dismissing the potential impact of Microsoft (or anyone else) on future virtualization markets, so I convinced him to do a little what if analysis with me, just to see what would evolve.  I started by putting him in the way-back machine and telling him the short version of “what ever happened to Novell”. 

Back in the early 1990’s Novel owned the local area network market – they were as dominant as VMware is today with well over 90% of the market and had an incredibly loyal following (I can attest to this having attended multiple Brainshare events with 10,000+ attendees – and I have the t-shirts to prove it). 

But what Novell and their clients loved was Novell’s technology, and the thought that anybody, especially Microsoft running that paltry LAN Manager product, could supplant them was heresy.  Every year new products (or plans) were announced, and every year the fan club grew.  But in the background there was a small chink in the armor, led by a commodity product that was available everywhere.  Windows 95 was released to great fanfare from the media as a desktop OS but  bundled  under the covers was a TCP/IP stack and some reasonable peer to peer capabilities.  Nobody really cared, especially the Novell fans (you can start thinking about hyper-V here, bundled into Windows 7,  just as a heads up).

As more and more of those new machines came in with Windows 95, more and more companies began using this free IP stack and good enough networking as a departmental alternative to “enterprise Netware”, or in some cases as a departmental add-on.  Novell announced Netware 4.0 with NDS (Netware Directory Services) and while an elegant product in it’s own right, nobody really cared.  It became the Betamax of it’s generation – a great technology that was more complex (or complete) than most customers needed and the march towards good enough networking continued.  It wasn’t until years later that Active Directory from Microsoft even came close to doing what NDS could do, but by that time it no longer mattered, Netware was on the long slide to Nevermore.

Ok, fast forward to today and ask what the heck this has to do with VMware.   Well, let’s see;  VMware owns the market, well above 90%, and continues to come out with more and more innovative products.  VMware has a loyal following of customers who see no reason to change direction – after all, the product works, the vision is sound, and the future is clear.  But lurking in the background is this little thing called hyper-V;  not as robust, or as tested as VMware, with almost no install base, and certainly not ready for prime time in most peoples minds.  However, it will be an integral part of Windows 7, Windows Server 2008 and Windows Server 7 in 2010.  Why should you (or VMware) care?  Because like “free networking”, or “free SharePoint”, hyper-V will get used, slowly at first, but as more and more systems get installed the base will increase and within just a few short years companies will discover “(surprise, surprise!) that they have business applications running on both VMware and Hyper-V. 

Do you care?  We all run heterogeneous environments anyway, right?  But if over time I have two VM’s and need to manage them – which management tool do I use?  And here’s the rub – VMware is making great management tools – for managing VMware.  But Microsoft’s management suite is designed to manage multiple VM’s from multiple vendors, including VMware.  Now, if you find yourself in a situation where VM’s of all types are proliferating with each new system brought in (and they will), the key to reducing your complexity becomes the management tools – and over time standardization of the core products tends to track towards those tools.  If the choice is multiple tools to manage multiple vendors vs. a single tool, which decision will most likely prevail?  And if you eventually standardize on a single management tool, what’s the likelihood that the “preferred” platform was one designed specifically to run under that vendors tool suite? 

Just a thought….It’s VMware’s market to keep.  Let’s wish them luck.

This post is the first in a series of questions that need to be asked before embarking on a data center build project.

How big is big enough?
The first question asked is often the most difficult to answer, or the simplest.  “It depends” might be valid for an analyst, but not when you’re potentially spending 10’s of millions of dollars on a new data center.  And the difficult part of this question is not figuring out how much you need – it’s figuring out what you need in 15 years.

In the old days (you know, last year) the method of figuring out how large the new data center was going to be was a combination of guesswork and extrapolation (based on that guesswork).  I remember doing future value calculations on floor space by substituting financials with square feet in order to come up with a defensible number for our plans.   This sounded good at the time, and impressed the bean counters, but essentially it was a more formalized version of traditional back of the napkin analysis – resulting in a number we could believe in – not necessarily one with any level of accuracy.  Sometimes called GIGO (garbage in, gospel out), planners tend to believe spreadsheets, regardless of the results.  (This has always baffled me, but that’s the topic of a future post….)

In today’s world though I would recommend some reverse engineering to get to a reasonable size.  Take a walk out on your floor (if it’s handy) and get a good feel for how many racks you have, and at what capacity those racks are populated.  A standard 42U rack today takes up about 30 square feet of floor space, if you include aisle ways, door swing space and room for maintenance.  If your computer room is mostly racks it’s then an easy task to multiply 30 square feet times the number of racks you have, and then apply a growth factor to it.  You’ll be surprise how much space is actually needed.   

The second factor to consider is the capacity of those racks you have today.  In most IT shops racks are populated in the 50-60% range, primarily to reduce concentrations of heat on the floor.  Racks and servers are spread out to share the heat, so to speak, but when considering building a new room you need to assume it will be build to handle proper heat loads, and therefore racks can be much more densely populated – upwards of 80-90%. 

Modern data centers are being built smaller, not larger, and are designed to support high levels of vertical growth, resulting in the same relative compute density as older designs, but with less capital outlays and greater flexibility for future growth.  We will address long term growth when question 5 is answered; How long should it last.

Data Center managers are continually looking for methods to provide the resources to support the business, but at the most optimal costs.  The mantra of “do more with less” is something IT has had to live with for years, and difficult economic times just sharpen the focus, especially towards capital budgets.   The days of building new data centers large enough to last 20 years are gone, at least for prudent designers.  This has been replaced by the idea of designing data centers than can be extended well beyond 20 years, both in physical footprint and power requirements.     

The latest design change to emerge is around the idea of data center tiers.  Traditionally IT managers built data centers with a high enough tier rating to support their most mission critical systems at the highest levels of availability affordable.  This has created situations where all applications became highly available, just as an offshoot of running in the same data center.  The fact that the data center was very costly in order to support high availability was seen as the cost of doing business. While this may have been a nice side effect of design principals in prior years, when looked at prudently the obvious question emerges;  is there a better way to design data centers?

The simple answer is; yes there is.   Historically these data center designs were fairly simple – Tier 1 through 4, and a specific square footage and power envelope – that many would consider the traditional method of data center design.  Over the past few years companies have begun building multi zoned data centers though, putting different power density levels, or zones within the same data center.  This multi-zoned approach provided a method of reducing capital expense dramatically since the electrical/mechanical equipment required was often reduced, while still maintaining scalability in the data center.  The basic idea was that different application types would require different levels of growth and power (density) with their servers (as we know, all applications are not created equal), and therefore designing floor-space to support the same density differences made sense. 

An obvious extension of this idea would be to apply a different uptime or availability goal to each of these zones, assuming that different application types would likely have different business impacts if they became unavailable.  Essentially you would be assigning specific service levels to discreet areas of the data center.  This is a non-traditional approach to design, but the impact on overall data center construction pricing can be huge.  Rather than build a tier 4 fully redundant data center that supports all mission critical systems, and everything else, why not build a tier 4 zone that supports mission critical (which may only be 15% of my overall workload), and assign tier 1, 2, or 3 status to other areas in the same building? 
Bottom line:  This concept is gaining acceptance in the industry but as you talk with engineering firms about potential design options expect some reticence at first, as non-traditional designs take longer to develop, and a firm may be reluctant to try something outside the standard design templates they have used in the past.  However, integrating concepts like the power zone approach and multi-tiering can reduce your capital expense by 40% or more, and with constrained IT budgets every little bit helps.

Historically IT people have had a fairly straightforward task – keep the systems running and recover quickly when they fail.  I say when, not if, because systems all fail at one time or another, no matter how well we plan.  Now this is a far to simplistic view of the real world I agree, but if you look at traditional IT service level agreements they were all designed around the same concept.  Data Center managers focus on 99.995 percent availability; network managers are chartered with 24/7 access; while storage or server administrators focus first and foremost on continuous access.

Now there is nothing wrong with these objectives – they satisfy the business needs and give us measureable metrics, but over the years these types of objectives have created a laser like focus on the singular goal, at the expense of the big picture.  If our focus is only on the those top line objectives, and not on how we got there, we may in fact be missing some very significant opportunities.

The recent interest in Green IT, or better yet, “cost effective” IT, has opened the eyes of many CIO’s and CFO’s to an underlying problem we have caused over the past 40 years. Ever since the EPA report on Data Center Energy Consumption was produced in August 2007 folks have been asking embarrassing questions about IT efficiency.  In many companies the CFO asked the CIO (or Data Center manager) how much energy their data centers were consuming every year, and in most cases the answer was a resounding “I don’t know – it’s the Facilities budget”.  When Facilities was asked, and the question was answered (to shock and dismay), the follow-on question invariably was “and how are you going to improve that next year?”.  To which the resounding reply was “it’s not a Facilities issue, it’s an IT issue”.  The problem is, going forward, it’s both a Facilities and an IT issue – and needs to be solved jointly.

So what’s happened in the past year?  I’ve seen a lot of arm waving and hand wringing, and in some cases even seen attempts to integrate some Facilities process and IT together, with minimal success.  Those that attempt to merge the groups get mired down in politics, but those who have created a IT Resource Manager role focused as a liaison between the groups seem to be having better success.

But this is a stop gap solution and over time will not solve the underlying problem, which by the way is NOT a power and cooling issue, but an asset utilization issue, and the key to it all is the definition of an asset.  IT folks look at assets as all those things they can kick – and the software that runs (or runs on) them, from servers to desktops to networks and storage.  Facilities folks tend to look at assets more traditionally – anything contained in (and including) the building itself, and in the case of data centers, excluding the IT equipment.   Forward thinking companies have realized that to truly run data centers efficiently, and to extend their life indefinitely, it is not logical to treat facilities and IT assets separately, because without one the other can’t survive.  And if one is inefficient (e.g. inadequate cooling equipment on the raised floor) the other will suffer the consequences.

The first step along the path to solving this problem is to redefine IT in our own minds.  Ask any IT person for the definition and you’ll hear Information Technology, because that’s what it’s all about, isn’t it?  But if you take a birds eye view of your data center, and include all the assets needed to make it succeed (and to help you reach those SLA’s), you’ll realize that the IT we should all be focused on is Infrastructure Technology.  Not until we start to focus on ALL the components that make up the ecosystem will we begin to make informed long range decisions.  Not until we understand the cascade effects of one decision on all the other components will IT stop working in a vacuum.  And until this happens it will not be possible to create truly efficient, scalable, high performance and low energy environments, because without the tight integration and cooperation between IT and Facilities it will never happen.

And long term think of the possibilities;  we’ve all used asset life cycle techniques for years, on our subset of the environment.  But what if I but the entire Infrastructure Technology for my data center on a continuous improvement life cycle?  This would include all IT and Facilities equipment that supported IT.  The end result would be a continuous modernization project that not only insured current technologies were available to support the business, but that the underlying infrastructure was always ready to support it.  When this happens the life cycle for a data center can be extended from the current average of 20 years to 60 years or more.   And think how that would change the analysis when trying to decide to build the next generation data center or not.

Perhaps I’m being a bit cynical here, but after reading the well written blog post The Elephant in the Social Software Room from Craig Roth of Burton I decided to re-read it, substituting the arguments around social software with the arguments about allowing the use of Instant Messaging in corporations from 8 years ago, or even eMail from 30 years ago.  Take the first paragraph and apply any of these technologies to the argument….

“…, organizations often fret about potential negative impacts of breaking down organizational and, to some extent, social barriers.  Some stakeholders wonder whether execs really want borderless discussions among their staffs, whether engineers really want sales people to be able to contact them directly, whether employees will spread poor practices without gatekeepers, etc….”

It seems that the issues and potential organizational impacts, and therefore the decisions IT execs need to focus on, haven’t really changed that much over the years, all that’s changed is the specific application.  The real impact is that all future hires will have social networking built into their personal interactions, and assume that it’s a standard way of doing business.  This is basic consumerization doing what it always does – forcing IT to adapt to technologies they aren’t ready for.  The challenge for IT is to put a strategy in place to adopt social software and integrate it into their business process, and to do it proactively, rather than in reaction to business demands.

IMHO