I would be remiss if I didn’t mention the Infrastructure Operations and Management conference coming up the 2nd week of June in Orlando.  At a time when every IT dollar counts, our 5-track agenda provides the strategic guidance and pragmatic recommendations needed to demonstrate the business value of I&O as you meet the challenges of modernization, transformation and convergence. We’ll extensively explore today’s I&O hot spots, including virtualization environments, assessing a public vs. private cloud strategy, aggressively optimizing legacy systems, achieving operational excellence despite limited resources and much more. Get ready to identify new business models, data center design strategies and technology approaches that can deliver ROI fast, efficiently and effectively. Your takeaway: the ability to leverage real-world experience and knowledge gleaned from more than 40 sessions, including analyst-user roundtables and end-user case studies.

The agenda will feature comprehensive tracks to drill down on the I&O topics most important today, with track sessions containing 6 to 8 highly focused presentations from Gartner analysts. .

• Track A: Optimizing IT Operations for the Future
• Track B: Deriving Second Generation Benefits from Virtualization
• Track C: Leveraging Cloud Computing Opportunistically & Strategically
• Track D: Key Innovations and Best Practices to Modernize your Data Center
• Track E: Managing Mobility and Communications

If you’re going to attend, stop me in the hallway and say hello.

What Are Containers or Container Based Data Centers?

A data center container is a shipping container set up to accommodate IT equipment. The typical data center container is based on ISO standards for ease of shipping; also, it may be modified in a number of ways to better support the secure and practical use of IT equipment. The basic equipment that most of these containers are designed to support includes servers, storage and networking gear. In addition, containers may be designed to support some combination of uninterruptible power supply (UPS), generators and/or chillers, with some of that equipment supported in the same containers as the servers and storage equipment, or in separate and distinct containers. Generally, data center containers have some connectivity elements, so that power input, cooling capabilities (e.g., water pipes) and network traffic can be fed into the container from the outside.

Containers are designed to be weather resistant, and, in some cases, are weather hardened for use in extreme environments, although most use cases see them being implemented within existing buildings or shells of buildings. Containers are also designed to be TIA-942 Tier 3 capable (depending on supporting infrastructure), and focus on high levels of energy efficiency, typically with a PUE of 1.3 or below.

What Are Modular or Component-Based Data Centers?

Modular data centers evolved from the basic premise of containers — that, if designed appropriately, extreme levels of performance could be attained in data centers using a consistent design technique, and capital costs could be reduced by standardizing components, construction and the supply chain. The modular design approach has actually split in two directions over the past few years. While some vendors have focused on the overall design of a complete data center solutions (e.g., HP FlexDC and i/o’s Data Systems i/o Anywhere), others have moved the modular design concept down to the rack or row level (e.g., APC InfraStruxure or Emerson SmartAisle).

In modular data centers, a core infrastructure design is developed that will eventually support multiple IT areas. The IT areas can be added as needed to support future growth, until the base power requirements are exceeded, at which time either the base module can be upgraded or another modular unit is put into place. Most of the vendor offerings in this space start with a base module of 250kVA or more and IT space of at least 500 to 2,500 square feet, which can then be scaled, but, as the market evolves, we expect to see a greater variety of choices (essentially modules on demand) to support individual needs.

In modular component designs, vendors have begun to focus on self-contained row or rack solutions where a predefined unit (e.g., eight racks) is delivered on-site and is already configured for power and cooling support. These units can be as simple as basic racks with monitoring and directed air cooling, or as complex as completely self-contained environments with either air, water or refrigerant-based cooling for extreme densities. UPS support can be delivered by the customer or integrated as part of the solution.

Bottom Line

When planning for data center growth, it is important that all alternatives be reviewed. Newer modular design techniques and container-based solutions should be a critical piece of your analysis. When used appropriately, they can solve specific problems, while reducing capital costs and the time it takes to implement new capacity.

The crisis is developing from three directions at once. First, it is becoming very difficult for data center managers to retain their best and brightest staff. The U.S. Department of Labor estimates that today’s labor force will have an average of 14 different jobs by the time they are 38. The days of job security and company loyalty as motivating factors are long gone, and data center managers need to develop new ways to retain staff. Worse, the staff at greatest risk are those with the most in demand skills, and consequently the most difficult (and costly) to replace.

Secondly the first Baby Boomer’s are beginning to reach retirement age in 2011. The first wave of Boomers will reach 65 this year and in North America we can expect an average of 10,000 people reaching retirement age, every day, for the next 19 years. But the issue is not the people in many cases – it’s the knowledge that’s retiring with them.

And third, a staffing crisis is being driven by virtualization – or the movement towards converged infrastructures as a whole. IT managers are realizing that as more and more technologies converge, the responsibility (end to end) for a given process changes. Take for example the movement towards converged storage infrastructures. Traditionally a storage manager had end-to-end responsibility for capacity, throughput, performance and availability. But when virtual infrastructures are using SAN’s, and Fiber Channel is running over Ethernet, who has responsibility for the overall storage environment; the network team, storage team, virtualization team, or server team? The answer is “all of the above”. Which brings up a huge issue in vertically organized shops today. And the solution in many organizations is to stress a broader vs. a deeper skill set – or as some like to call it: the T shaped employee.

If you look at any employee within the IT group, try to imagine them as represented by a capital letter “T,” where the center post of the T represents their depth of knowledge in their primary skill set. Some people can drill down that center post to a frightening level of detail, but, in a sense, the depth of the T represents their strength as a technologist, and, in most organizations, it’s the key to their credibility in the organization, and to their success within IT. It’s their core.

The cross bar of the T is not about technology per se, but about the relationship between converging technologies (and the business) and how much that employee understands about these relationships. As you increase the number of relationships (or technology linkages), the broader the crossbar on the T becomes. Now sit back and think about the most valuable people in the IT organization — those people who always get the projects handed to them, because we know they’ll get things done, regardless. If you look at the strengths they bring to any project, it’s rarely depth of knowledge (the vertical T), but breadth of understanding (all the linkages). Drill-down specialists are available in all disciplines, but the linkage masters are hard to find.

This then becomes a motivational tool for organizations. Let’s begin to recognize what we really value in IT — not depth (except in junior people), but breadth across multiple disciplines coupled with depth in a primary discipline. It’s difficult to train people with this knowledge, except through real world experiences with a business context; your business, and your projects. In some cases, it may take forced change in disciplines, but force the change in a related discipline with the idea of expanding knowledge and creating those linkages. The most effective IT people are always looking for new things to learn, and, in many cases, the most interesting areas are in the unknown, not the known areas. Enabling this learning — even incenting it — is a critical success factor as we move toward fully virtualized environments. And when employees realize that their value is not only how much they know in a discipline, but how much they understand the linkages between disciplines, IT as a whole will become a much stronger organization, and more able to adapt to changing environments.

Developing people with these kinds of skills can help alleviate both the baby boomer/skills transfer issue, as well as the issue of keeping your best people challenged and growing their skill sets, while continually increasing their value to the company.

However, these forces mentioned above can actually work in your favor, providing the means and motivation to apply innovative designs, reduce both capital and operating cost, increase long term scale, and keep up with the business as well. Sound impossible? Let’s take a look at each of these trends at a macro level.

Smarter Designs - Traditional methods of designing data centers were created during the mainframe era, and subsequently many of us did all the wrong things, for all the right reasons. Consolidation mainframe and Unix environments in many cases consisted of large groups of the same equipment, with similar operating characteristics (power draw, heat generation, etc), and with similar performance targets. Because of their high costs many mainframes were targeted for average performance ranges in the mid 90% during production time slots, and therefore there was minimal variation in the operating temperature or power consumption over long periods of time. With these “known” variables it was a simple exercise in mathematics to design a large floorspace with enough power drops and underfloor cooling to support projected workloads, even for an “at capacity” data center.

Not anymore. Today’s data centers have many different demands on mechanical/electrical systems, depending on workload mix, function and the age of equipment. New designs have taken this into account by adding different density zones for different workload types. It’s not uncommon to see a high density zone supporting fully configured blade enclosures, or high density racks positioned in a cold or hot containment room. This zone might employ directed cold air, or even in rack cooling (air or refrigerant) to support very high density workloads with minimal disruption, or impact, on the rest of the floor. Secondary zones would support steady state applications that consume a consistent amount of power and produce manageable heat loads, while low density zones would be designed to support low power equipment – perhaps telecom and storage.

Green Pressure – The Green movement has had a curious impact on data centers and data center design. Most data center managers paid little attention to the “greening of IT” unless they were pressured into it by senior management or the public (particularly in EMEA). But as awareness has increased there has been a constant uptick in the attention being paid to energy consumption within data centers. Traditionally this was a concern of the Facilities team, since they usually paid the electric bill each month, but even they didn’t look at efficiency relative to the consumption of energy as something in the purview of IT. All that has changed and new data centers take a very hard look at energy efficiency in both design and execution. The development and marketing of PUE by The Green Grid continues to gain ground in the market and many new data centers are being developed with specific PUE targets in mind, both for the energy efficiency advantages as well as the public relations impact (e.g. the Yahoo Coop Data Center with a sub 1.1 PUE target).

Conquering Density – With smarter designs and Green pressures data center managers and designers have begun to focus on the compute density within their environments. Most data centers today are woefully underutilized from a space perspective. The physical floor space may be nearing capacity, but in many cases the actual compute space within racks, and within servers themselves, are very poorly used. Average rack densities today approach 60% worldwide except in very well managed (and designed) environments. This is not a failing of IT, but a reality IT had to live with when higher densities created excess heat which the existing infrastructure could not deal with. The solution was to buy more racks and spread the load across more floor space. Newer designs focus on this issue and are developed to allow optimal rack density, often approaching 85-90% on average, thus increasing the compute per square foot dramatically. Likewise server performance has become another target where optimal performance averages are approaching 55%-70% for virtualized devices, depending on workload mix. This increases the compute per kilowatt significantly, as that server running at 15% utilization is using almost as much energy as when it runs at 55%.

Cloud Computing – No, we are not going to move all workloads to the cloud, at least not in the foreseeable future. But the concept of cloud computing – obtaining resources outside of my data center, when I need them, for however long I need them, is a compelling argument. The question becomes; what resources, and for what applications or workloads? An intriguing choice data center managers are beginning to deal with is the possibility of shifting non-essential workloads to a cloud provider, freeing up much needed floor space, power and cooling that can then be focused on more critical production workloads, and extending the useful life of the data center.

Shifting workloads is not new, many companies use colocation facilities as an overflow mechanism, but the difference is that with colocation the compute resource is still owned and managed by the application owner. With offloading services to the cloud, ownership and management of the IT assets is shifted to the provider, essentially outsourcing the service to someone else.

As this practice increases in popularity the landscape for what remains of the corporate data center will change significantly. Only core business functions – those that differentiate a business from it’s competition, or are truly mission critical will remain in the primary data center. All other non-critical services will eventually migrate to external providers, having the long term affect of shrinking physical data center requirements.

By 2018 expect data center requirements to shrink by as much as 60%, with a focus on only core business services. And even those these services continue to demand more IT resources, the shrinking landscape of servers and storage (and telecom equipment) will more than offset that growth.

But as we move towards more and more virtualization the lines are beginning to blur about who is responsible for what – and who has the ultimate authority to implement change.  If you can create dozens of images on a single server who has the final say in what applications are allowed to run ?  The virtualization experts, who likely have a clear idea of how resources are utilized in a VM?  The applications team, who understand how applications interact and when or where peaks in performance will happen?  Or the server team who are continually looking at optimal performance levels across all servers, power usage and heat generation?  Or is it the storage team, who are more and more often concerned with I/O bottlenecks in highly virtualized environments and the most efficient placement of VM’s relative to I/O throughput.  It turns out that all of these groups have a vested interest in what’s going on, but that current organizational constructs may in fact inhibit their ability to be effective in their roles.  Worse yet, these vertical structures often exacerbate issues around who has control and can cause chaos in an organization where employees are compensated on their overall impact.  Giving up technology turf is not in our DNA – and expecting people to do it for the good of the organization not likely to happen consistently.

So how do we motivate people to change – for the good of the organization – and for their own good?   There are two common techniques which will most often fail miserably (without VERY strong management), and a newer technique which has been emerging from the ground up which I’ll discuss in a moment.  The two tried and not-so-true techniques used most often are organizational shift and financial incentives.  Organizational shift is a technique whereby slight changes in management structure take place which in turn create a shift in technical (employee) responsibilities, with the express goal of forcing greater control on the assigned group.  What often happens is that the newly formed group is comprised of individuals from the former vertical coverage areas, and those individuals first assignment in the new group is to maintain their current coverage area “during the transition”.  Not surprisingly the transition period rarely ends and IT groups find themselves with new organizational constructs still supporting the same vertical groups – with the same basic staff. 

Financial incentives can work, but rarely do it effectively.  Financial change tends to be a short term motivator and unless there is some inherent impact to the individual’s self-worth, within a short time the perceived incentive no longer has value.   This idea of individual self-worth has been known for many years, but it’s surprising how little we pay attention to it, even when the advantages can be incredible – and at very little additional cost.   Given that, here’s the third alterative that is beginning to gain momentum ;   call it the T Shaped Technician for lack of a better term.

If you look at any employee within the IT group try to imagine them as represented by a capital letter “T” where the center post of the T represents their depth of knowledge in their primary skill set.  Some people can drill down that center post to a frightening level of detail, but in a sense the depth of the T represents their strength as a technologist, and in most organizations it’s the key to their credibility in the organization, and to their success within IT.  It’s their core.  The cross bar of the T is not about technology per se, but about the relationship between converging technologies (and the business) and how much that employee understands about these relationships.  As you increase the number of relationships (or technology linkages), the broader the crossbar on the T becomes. 

Now sit back and think about the most valuable people in the IT organization – those people who always get the projects handed to them, because we know they’ll get things done, regardless.  If you look at the strengths they bring to any project it’s rarely depth of knowledge (the vertical T), but breath of understanding (all the linkages).  Drill down specialists are available in all disciplines, but the linkage masters are hard to find.

This then becomes a motivational tool for organizations.  Let’s begin to recognize what we really value in IT – not depth (except in junior people), but breadth across multiple disciplines coupled with depth in a primary discipline.  It’s difficult to train people with this knowledge, except through real world experiences with a business context; your business, and your projects.   In some cases it may take forced change in disciplines, but force the change in a related discipline with the idea of expanding knowledge and creating those linkages.  The most effective IT people are always looking for new things to learn and in many cases the most interesting areas are in the unknown, not the known areas.  Enabling this learning – even incenting it, is a critical success factor as we move toward fully virtualized environments.  And when employees realize that their value is not only how much they know in a discipline, but how much they understand the linkages between disciplines, IT  as a whole will become a much stronger organization, and more able to adapt to these changing environments.

Take for example some of our legacy applications.  Not the ones we’ve been living with on mainframes or large Unix systems for the past 30 years, but legacy x86 applications.  You know, those early Windows applications written in C or C++ (or even early Java) which now run quietly every day on those older Windows NT or Windows 2000 sever platforms.  Like many older applications from the Big Iron days, these are often poorly documented and not designed with the reusable constructs of SOA, but as stand alone, end to end systems.

In many companies these are clear targets for virtualization.  They often have stable performance characteristics (few peaks and valleys), allowing a high number of images per physical server.  Since they are older applications the platform itself is not likely to change and the amount of enhancements to the application have been minimized over the years.

As newer Operating Systems emerge and servers grow into 8 and more cores there is a risk that these legacy applications will suffer compatibility or performance problems and will require significant and costly retooling.  But with development budgets shrinking, or staying stable at best, higher priority projects will most often get the funding over legacy rewrites, so the prudent IT manager will keep these applications running on the older OS’s in a standardized virtual container as long as possible.  This costs very little, does not impact the performance or reliability of the application, and gives IT some breathing room before a large, and possibly complex rewrite begins.

Sounds like yet another side benefit of virtualization, and for the near term it certainly is, as emulating older environments on newer technologies has been practiced for years, and the financial benefits are obvious. 

However, as the underlying operating systems get older, updates, patches, and support begins to get marginalized by vendors, and eventually formal maintenance support from the vendor will reach an end.  At this point enterprises will have another difficult decision to make – to continue emulation of an older application on an unsupported platform, or to begin the replacement (or rewrite) process for the application.  Either choice is fraught with risks, costs and business impact, and in many organizations this will not be a single decision, because as we continue this march towards virtualization the number of legacy applications marginalized into self contained environments will continue to grow.  

So what are the impacts?  In a best case scenario this is just the ranting of a cynical naysayer and we will continue to upgrade and improve applications as needed, and migrate them to newer platforms when appropriate.  It’s a non-issue.  But the alternate world view is that the beauty of virtualization is that once an environment is created it can run “as is” indefinitely, with little or no attention from the outside.  If this happens IT will almost always focus on near term issues, because these issues are what drive us – after all we are a reactive crowd – and when budgets are always tight the funding to fix what isn’t broken rarely materializes. 

In the second world view we may find ourselves scrambling to update scores or even hundreds of these applications to run on newer platforms when we are least prepared to do so.  This is similar to Y2K in some respects in that these problems are not caused by lack of knowledge or awareness, but just by years of pushing the issue into that low priority bucket that’s so convenient to use during the planning cycle. 

Just a thought……

Where to begin?

In my first post of this series Allison sent along a quick note with an interesting observation;  “I found this article very insightful, but it left me with one resounding question. Where do I begin? If I’m a company who thinks that a data center build might be in my future, who do I go to first?”

I realized when I read this comment that there is an underlying problem with almost all data center build projects, and that is the simple fact that these are one-off projects, only executed once a decade.  The person who ran the last one may have been wildly successful, but the likelihood that they documented all the steps, or are still around to share them with you are pretty slim.  Retaining the information gathered and lessons learned, and then documenting them for future reference, is almost never done.  So when someone like Allison (or any of us) gets volunteered to run one of these projects we begin the process by relearning everything that’s been done in past, both the good and the bad.   So, lesson 1;  document everything.

And to your point Allison on where to begin, let me offer this.  I would begin by creating a very small tiger team of perhaps only 5 or 6 people.  Early in the project these might even be part time positions, but the focus is on data gathering, idea generation and team building.  Members should include folks with skills in IT architecture, servers, storage, networking, facilities and building security.  The facilities folks would include disciplines in air conditioning, (focused on IT cooling), UPS and generators and power distribution. I would be looking for people who didn’t come into project with pre-conceived notions of what a data center should look like, because many bad designs have resulted from repeating the past (today’s well designed data center is radically different from older versions). 

Lastly, early on in the project I would be looking for a representative from either the PMO or an assigned Project Manager to begin tracking and linking deliverables, and also to begin documenting the expected desired state as input to the commissioning process.  Commissioning is a critical part of the build process, and the most efficient scripts built for the commissioning process begin early in the project.

As always, all comments and questions are welcome.

How much energy will I need?

Great question.  Somebody always asks the obvious, especially when there is no way to definitively answer the question.  In the old days (you know, 8 or 10 years ago), a data center was built with a single power footprint – meaning a consistent watts per square foot design.  This was done historically with data centers because IT equipment generally followed a fairly obvious and linear energy consumption growth path and projecting future growth was at least feasible.   Not so much in today’s world.  Both server and storage energy growth has skyrocketed over the past 8 years due to the introduction of newer technologies (blades, 2u and 1u servers, higher density cores etc).  Couple that with a rapid rise in the use of virtualization and we have exceeded the physical capacity of the facilities mechanical and electrical equipment to deliver power to support IT loads in many data centers around the world.

So what can you do?  You can plan for the unknown (and I wish you luck with that boardroom funding request), or you can design for the most logical expected energy loads, but with the flexibility to shift loads and usage patterns as needed.  From a basic design point today a 3 zone structure might support high density racks at 10-12kW, medium density at 6-8kW and low density up to 5kW.  The size (percentage) of these zones obviously depends on you (your mileage will truly vary) but these zones can shrink or grow as workloads demand.

Ok, let’s say we’ve done that and 8 years from now things are getting a bit hot in the data center and power demands continue to rise, one option would be to refocus my cooling efforts away from forced air, especially as power requirements for equipment begin to exceed the 6 to 8kW per rack range.  Augmenting traditional air cooling with in-rack cooling, either water or refrigerant, will fee up a significant amount of power for other uses.  The general rule of thumb is that for each watt of power for IT load you should assume at least an additional watt of power for cooling the same equipment.  For low utilized equipment this could be less (below 4kW per rack), but as you move into the higher density zones above 8kW the ratio begins to shift towards cooling needing more power than IT load (sometimes as much as 2 to 1).   However, if you decide to cool inside the rack with refrigerant based systems the cooling to IT load ratio never changes – as air leaving the rack is approximately the same temperature as that going in and the room air conditioning workload does not change.

There are other options as well which we will explore in other posts, things like bringing in outside air instead of just using the mechanically cooled variety, or refocusing what loads actually need the UPS and which ones don’t.  The bottom line though is that with flexible designs, scalable PDU’s, and a willingness to use alternative cooling techniques the energy question is not quite as critical as one might think.