Traffic Server was a Web proxy server — essentially, software for running big caches. It delivered significantly greater scalability, stability, and maintainability than did the most commonly-used alternative, the open-source Squid. It was a great piece of software; at one point in time, I was one of Inktomi’s largest customers (possibly the actual largest customer), with several hundred Traffic Servers deployed in production globally, so I speak from experience, here. (This was as ISP caches, as opposed to the way that Yahoo uses it, which is a front-end, “reverse proxy” cache.)

Now, as ghosts of the dot-com era resurface, Yahoo is open-sourcing Traffic Server. This is a boon not only to Web sites that need high scalability, but also to organizations who need inexpensive, high-performance proxies for their networks, as well as low-end CDNs whose technology is still Squid-based. There are now enterprise competitors in this space (such as Blue Coat Systems), but open-source remains a lure for many seeking low-cost alternatives. Moreover, service providers and content providers have different needs from the enterprise.

This open-sourcing is only to Yahoo’s benefit. It’s not a core piece of technology, there are plenty of technology alternatives available already, and by opening up the source code to the community, they’re reasonably likely to attract active development at a pace beyond what they could invest in internally.

Given that it’s now clear that there are capacity constraints on EC2 (i.e., periods of time where provisioning errors out due to lack of capacity), this is something that’s of direct concern to users. And for all the cloud-watchers, it’s a fascinating study of IaaS adoption.

Randy Bias of CloudScaling has recently posted some interesting speculation on EC2 capacity.

Guy Rosen has done a nifty analysis of EC2 resource IDs, translated to an estimate of the number of instances provisioned on the platform in a day. Remember, when you look at provisioned instances (i.e., virtual servers), that many EC2 instances are short-lived. Auto-scaling can provision and de-provision servers frequently, and there’s significant use of EC2 for batch-computing applications.

Amazon’s unreserved-instance capacity is not unlimited, as people have discovered. There are additional availability zones, and for serious users of the platform, choosing the right zone has become minimal, since you don’t want to pay for cross-zone data transfers or absorb the latency impact, if you don’t have to.

We’re entering a time of year that’s traditionally a traffic ramp for Amazon, the fall leading into Christmas. It should be interesting to see how Amazon balances its own need for capacity (AWS is used for portions of the company’s retail site), reserved EC2 capacity, and unreserved EC2 capacity. I suspect that the nature of EC2’s usage makes it much more bursty than, say, a CDN.

While this is a good move for Rackspace, it’s not ground-breaking stuff, although the portal is notable for a design that seems more consumer-friendly (by contrast with Amazon’s highly text-dense, spartan partner listings). Rather, what’s interesting is Rackspace’s ongoing (successful) efforts to encourage an ecosystem to develop around its cloud APIs, and the broader question of cloud API standardization, “de facto” standards, and similar issues.

There are no small number of cloud advocates out there that believe that rapid standardization in the industry would be advantageous, and that Amazon’s S3 and EC2 APIs, as the APIs with the greatest current adoption and broadest tools support, should be adopted as a de facto standard. Indeed, some cloud-enablement packages, like Eucalyptus, have adopted Amazon’s APIs — and will probably run into API dilemmas as they evolve, as private cloud implementations will be different than public ones, leading to inherent API differences, and a commitment to API compatibility means that you don’t fully control your own feature roadmap. There’s something to be said for compatibility, certainly. Compatibility drives commoditization, which would theoretically lower prices and deliver benefits to end-users.

However, I believe that it’s too early in the market to seek commoditization. Universal commitment to a particular API at this point clamps standardized functionality within a least-common-denominator range, and it restricts the implementation possibilities, to the detriment of innovation. As long as there is rapid innovation and the market continues to offer a slew of new features — something which I anticipate will continue at least through the end of 2011 and likely beyond — standardization is going to be of highly limited benefit.

Rackspace’s API is different than Amazon’s because Rackspace has taken some different fundamental approaches, especially with regard to the network. For another example of significant API differences, compare EMC’s Atmos API to Amazon’s S3 API. Storage is a pretty simple thing, but there are nevertheless meaningful differences in the APIs, reflecting EMC’s different philosophy and approach. (As a sideline, you might find William Vambenepe’s comparison of public cloud APIs in the context of REST, to be an interesting read.)

Everyone can agree on a certain set of core cloud concepts, and I expect that we’ll see libraries that provide unified API access to different underlying clouds; for instance, libcloud (for Python) is the beginning of one such effort. And, of course, third parties like RightScale specialize in providing unified interfaces to multiple clouds.

One thing to keep in mind: Most of the cloud APIs to date are really easy to work with. This means that if you have a tool that supports one API, it’s not terribly hard or time-consuming to make it support another API, assuming that you’re confining yourself to basic functionality.

There’s certainly something to be said in favor of other cloud providers offering an API compatibility layer for basic EC2 and S3 functionality, to satisfy customer demand for such. This also seems to be the kind of thing that’s readily executed as a third-party library, though.

Earlier today, I summarized what VPC is and isn’t, but I realize, after reading the other reactions, that I should have been clearer on one thing: Amazon VPC is not a private cloud offering. It is a connectivity option for a public cloud. If you have concerns about sharing infrastructure, they’re not going to be solved here. If you have concerns about Amazon’s back-end security, this is one more item you’re going to have to trust them on — all their technology for preventing VM-to-VM and VM-to-public-Internet communication is proprietary.

Almost every other public cloud compute provider already offers connectivity options beyond public Internet. Many other providers offer multiple types of Internet VPN (IPsec, SSL, PPTP, etc.), along with options to connect virtual servers in their clouds to colocated or dedicated equipment within the same data center, and options to connect those cloud servers to private, dedicated connectivity, such as an MPLS VPN connection or other private WAN access method (leased line, etc.).

All Amazon has done here is join the club — offering a service option that nearly all their competitors already offer. It’s not exactly shocking that customers want this; in fact, customers have been getting this from competitors for a long time now, bugging Amazon to offer an option, and generally not making a secret of their desires. (Gartner clients: Connectivity options are discussed in my How to Select a Cloud Computing Infrastructure Provider note, and its accompanying toolkit worksheet.)

Indeed, there’s likely a burgeoning market for Internet VPN termination gear of various sorts, specifically to serve the needs of cloud providers — it’s already commonplace to offer a VPN for administration, allowing cloud servers to be open to the Internet to serve Web hits, but only allow administrative logins via the backend VPN-accessed network.

What Amazon has done that’s special (other than being truly superb at public relations) is to be the only cloud compute provider that I know of to fully automate the process of dealing with an IPsec VPN tunnel, and to forego individual customer VLANs for their own layer 2 isolation method. You can expect that other providers will probably automate VPN set-up so in the future, but it’s possibly less of a priority on their road maps. Amazon is deeply committed to full automation, which is necessary at their scale. The smaller cloud providers can get away with some degree of manual provisioning for this sort of thing, still — and it should be pretty clear to equipment vendors (and their virtual appliance competitors) that automating this is a public cloud requirement, ensuring that the feature will show up across the industry within a reasonable timeframe.

Think of it this way: Amazon VPC does not isolate any resources for an individual customer’s use. It provides Internet VPN connectivity to a shared resource pool, rather than public Internet connectivity. It’s still the Internet — the same physical cables in Amazon’s data center and across the world, and the same logical Internet infrastructure, just with a Layer 3 IPsec encrypted tunnel on top of it. VPC is “virtual private” in the same sense that “virtual private” is used in VPN, not in the sense of “private cloud”.

So what, exactly, is VPC?

VPC offers network isolation to instances (virtual servers) running in Amazon’s EC2 compute cloud. VPC instances do not have any connectivity to the public Internet. Instead, they only have Internet VPN connectivity (specifically, an IPsec VPN tunnel), allowing the instances to seem as if they’re part of the customer’s private network.

For the non-techies among my readers: Think about the way you connect your PC to a corporate VPN when you’re on the road. You’re on the general Internet at the hotel, but you run a VPN client on your laptop that creates a secure, encrypted tunnel over the Internet, between your laptop and your corporate network, so it seems like your laptop is on your corporate network, with an IP address that’s within your company’s internal address range.

That’s basically what’s happening here with VPC — the transport network is still the Internet, but now there’s a secure tunnel that “extends” the corporate network to an external set of devices. The virtual instances get corporate IP addresses (Amazon now even supports DHCP options), and although of course the traffic is still coming through your Internet gateway and you are experiencing Internet performance/latency/availability, devices on your corporate WAN “think” the instances are local.

To set this up, you use new features of the Amazon API that lets you create a VPC container (a logical construct for the concept of your private cloud), subnets, and gateways. When you actually activate the VPN, you begin paying 5 cents an hour to keep the tunnel up. You pay normal Amazon bandwidth charges on top of that (remember, your traffic is still going over the Internet, so the only extra expense to Amazon is the tunnel itself).

When you launch an EC2 instance, you can now specify that it belongs to a particular VPC subnet. A VPC-enabled instance is not physically isolated from the rest of EC2; it’s still part of the general shared pool of capacity. Rather, the virtual privacy is achieved via Amazon’s proprietary networking software, which they use to isolate virtual instances from one another. (It is not intra-VM firewalling per se; Amazon says this is layer 2 network isolation.)

At the moment, an instance can’t be both be part of a VPC and accessible to the general Internet, which means that this doesn’t solve a common use case — the desire to use a private network for back-end administration or data, but still have the server accessible to the Internet so that it can be customer-facing. Expect Amazon to offer this option in the future, though.

As it currently stands, with an EC2 instance with VPC limited to communicating with other instances within the VPC, as well as the corporate network, this solves the use case of customers who are using EC2 for purely internally-facing applications and are seeking a more isolated environment. While some customers are going to want to have genuinely private network connectivity (i.e., the ability to drop an MPLS VPN connection into the data center), a scenario that Amazon is unlikely to support, the VPC offering is likely to serve many needs.

Note, by the way, that the current limitation on communication also means that EC2 instances can’t reach other Amazon Web services, including S3. (However, EBS does work, as far as I know.) While monitoring is supported, load-balancing is not. Thus, auto-scaling functionality, one of the more attractive recent additions to the platform, is limited.

VPN connectivity for cloud servers is not a new thing in general, and part of what Amazon is addressing with this release is a higher-security option, for those customers who are uncomfortable with the fact that Amazon, unlike most of its competitors, does not offer a private VLAN to each customer. For EC2 specifically, there have been software-only approaches, like CohesiveFT’s VPN-Cubed. Other cloud compute service providers have offered VPN options, including GoGrid and SoftLayer. What distinguishes the Amazon offering is that the provisioning is fully automated, and the technology is proprietary.

This is an important step forward for Amazon, and it will probably cause some re-evaluations by prospective customers who previously rejected an Amazon solution because of the lack of connectivity options beyond public Internet only.

Cloud services are evolving with extraordinary rapidity. I always caution customers not to base deployment plans for one year out on the current state of the technology, because every vendor is evolving so rapidly that the feature that’s currently missing and that you really want has, assuming it’s not something wacky and unusual, a pretty high chance of being available when you’re actually ready to start using the service in a year’s time.

Gartner’s very first Hype Cycle for Cloud Computing features a whole array of cloud-related technologies and services. One of the most interesting things about this hype cycle, I think, is the sheer number of concepts that we believe will hit the plateau of productivity in just two to five years. For a nascent technology, that’s pretty significant — we’re talking about a significant fundamental shift in the way that IT is delivered, in a very short time frame. However, a lot of the concepts in this hype cycle haven’t yet hit the peak of inflated expectations — you can expect plenty more hype to be coming your way. There’s a good chance that for the IaaS elements that I focus on, the crash down into the trough of disillusionment will be fairly brief and shallow, but I don’t think it can be avoided. Indeed, I can already tell you tales of clients who got caught up in the overhype and got themselves into trouble. But the “try it and see” aspect of cloud IaaS means that expectations and reality can get a much faster re-alignment than it can if you’re, say, spending a year deploying a new technology in your data center. With the cloud, you’re never far from actually being able to try something and see if it fits your needs.

My hype cycle profile for CDNs appears on our Media Industry Content hype cycle, as well as our brand-new TV-focused (digital distribution and monetization of video) Media Broadcasting hype cycle. Due to the deep volume discounts media companies receive from CDNs, the value proposition is and will remain highly compelling, although I do hear plenty of rumblings about both the desire to use excess origin capacity as well as the possibilities that the cloud offers for both delivery and media archival.

I was involved in, but am not a profile author on, the Hype Cycle for Data Center Power and Cooling Technologies. If you are a data center engineering geek, you’ll probably find it to be quite interesting. Ironically, in the midst of all this new technology, a lot of data center architecture and engineering companies still want to build data centers the way they always have — known designs, known costs, little risk to them… only you lose when that happens. (Colocation companies, who have to own and operate these data centers for the long haul, may be more innovative, but not always, especially since many of them don’t design and build themselves, relying on outside expertise for that.)

We have some questions that we know we want to ask (and that we know our clients, both end-users and vendors, are curious about), and some hypotheses that we want to test, but I’ll ask in this open forum, in an effort to try to ensure the survey is maximally useful: What are the cloud-adoption survey questions whose answers would cause you to change your cloud-related decision-making? (You can reply in a comment, send me email, or Twitter @cloudpundit.)

I expect survey data will help vendors alter their tactical priorities and may alter their strategic plans, and it may assist IT buyers in figuring out where they are relative to the “mainstream” plans (useful when talking to cautious business leadership worried about this newfangled cloud thing).

Somewhat peripherally: Following up on earlier confusion, a potshot was taken at the popularity of surveys at large analyst firms. I’ll note that I’m very much a fan of surveys, and if I had infinite budget to work with, I’d probably field a lot more of them. Surveys are (hopefully) not just blind firing of questions into the populace. Intelligent survey design is an art form (as is proper fielding of a survey). Asking the right questions — forming testable hypotheses whose implications are actionable by clients, and getting good information density out of the questions you ask (looking for patterns in the correlations, not just the individual answers) — is incredibly important if you’re going to get something maximally useful out of the money you spent. Data analysis can drive insights that you wouldn’t have otherwise been able to obtain and/or prove.

This MQ is not a pure cloud computing MQ. It is a hosting MQ. Titling it as such, and making it such, is not some feeble attempt to defend the traditional way of doing things. It is designed to help Gartner’s clients select a Web hoster, and it’s focused upon the things that enterprises care about. Today, our clients consider cloud players as well as traditional players during the selection process. Cloud has been highly disruptive to the hosting industry, introducing a pile of new entrants, revitalizing minor players and lifting them to a new level, and forcing successful traditional players to revise their approach to the business.

The most common question asked by outsiders who just look at the chart and nothing more is, “Why doesn’t Amazon score higher on vision and execution?”

The answer, simply, is that the hosting MQ scores five use cases — self-managed hosting, mainstream (low/mid-end) managed hosting, highly complex managed hosting, global solutions portfolio (ability to provide multiple types of service packages at multiple price points, globally, for very large multi-nationals seeking global hosting options), and enterprise applications hosting. The final rating is a weighted composite of these scores. Amazon scores extremely highly on self-managed hosting, but has a much more limited ability to support the other four scenarios.

Amazon lacks many capabilities that are important in the overall Web hosting market, like managed services, the ability to mix in dedicated equipment (important to anyone who wants to run things that don’t virtualize well, like large-scale Oracle databases, as well as colocate “black box” hardware appliances, like those used for transaction functions for some e-commerce sites), the ability to isolate the environment from the Internet and just use private network connectivity, etc. Their lack of these capabilities hurts their scores. (Note that some capabilities that were missing may have been disclosed to us as part of Amazon’s roadmap, which augmented their Vision score positively, but similarly, stances taken that would definitively shut out some features would be penalized.)

Clearly, we don’t think that Amazon sucks as a cloud provider; it’s just that they don’t play as broadly in the hosting space as the best of the traditional players, although they are certainly a competitor against the traditional players, and a disruptive entrant in general.

The same could be said for many of Amazon’s cloud competitors, although those with some background in traditional hosting may have fewer product-portfolio gaps. Original innovation is a component of Vision but it’s only part of the overall Vision score, so being a fast follower only hurts you so much.

We recognize the need for a “pure cloud compute” vendor rating, and have one in the works.

Amazon’s revocation of Orwell novels on the Kindle has stirred up some cloud debate. There seems to have been a thread of “will this controversy kill cloud computing”, which you can find in plenty of blogs and press articles. I think that question, in this context, is silly, and am not going to dignify it with a lengthy post of my own. I do think, however, that it highlights important questions around content ownership, application ownership, and data ownership, and the role that contracts (whether in the form of EULAs or traditional contracts) will play in the cloud. By giving up control over physical assets, whether data or devices, we place ourselves into the hands of thir parties, and we’re now subject to their policies and foibles. The transition from a world of ownership to a world of rental, even “permanent” lifetime rental, is not a trivial one.

Engine Yard has expanded its EC2 offering. Previously, Engine Yard was offering Amazon EC2 deployment of its stack via an offering called Solo, for low-end customers who only needed a single instance. Now, they’ve introduced a version called Flex, which is oriented around customers who need a cluster and associated capabilities, along with a higher level of support. This is notable because Engine Yard has been serving these higher-end customers out of their own data center and infrastructure. This move, however, seems to be consistent with Engine Yard’s gradual shift from hosting towards being more software-centric.

The Rackspace Cloud Servers API is now in open beta. Cloud Servers is essentially the product that resulted from Rackspace’s acquisition of Slicehost. Previously, you dealt with your Cloud Server through a Web portal; this new release adds a RESTful API, along with some new features, like shared IPs (useful for keepalived and the like). Also of note is the resize operation, letting you scale your server size up or down, but this is really handwaving magic in front of replacing a smaller virtual server with a larger virtual server, rather than expanding an already-running virtual instance. The API is fairly extensive and the documentation seems decent, although I haven’t had time to personally try it out yet. The API responses, interestingly, include both human-readable data as well as WADL (Web Application Description Language, which is machine-parseable).

SOASTA has introduced a cloud-based performance certification program. Certification is something of a marketing gimmick, but I do think that SOASTA is, overally, an interesting company. Very simply, SOASTA leverages cloud system infrastructure to offer high-volume load-testing services. In the past, you’d typically execute such tests using a tool like HP’s LoadRunner, and many Web hosters offer, as part of their professional services offerings, performance testing using LoadRunner or a similar tool. SOASTA is a full-fledged software as a service offering (i.e., it is their own test harness, monitors, analytics, etc., not a cloud repackaging of another vendor), and the price point makes it reasonable not just for the sort of well-established organizations that could previously afford commercial performance-testing tools, but also for start-ups.

Various pundits are apparently surprised by these results. I’m not. I haven’t been able to obtain a copy of the Forrester report, but from the comments I’ve read, it appears that software as a service and hosting (part of infrastructure as a service) are included as part of the surveyed services. SaaS and IaaS are both well-established markets, with significant penetration across all segments of business, and interest in both IaaS and SaaS models has accelerated. We’ve wrapped the “cloud” label around some or all of these existing markets (how much gets encompassed depends on your definitions), so it shouldn’t come as a surprise to already see high adoption rates.

Gartner’s own survey on this topic has just been published. It’s titled, “User Survey Analysis: Economic Pressures Drive Cost-Oriented Outsourcing, Worldwide, 2008-2009“. Among its many components is a breakdown of current and planned use of alternative delivery models (which include things like SaaS and IT infrastructure utilities) over the next 24 months. We show even higher current and planned adoption numbers than Forrester, with IaaS leading the pack in terms of current and near-term adoption, and very healthy numbers for SaaS as well.