Blog post

Trialing a lot of cloud IaaS providers

By Lydia Leong | November 08, 2011 | 4 Comments


I’ve just finished writing the forthcoming Public Cloud IaaS Magic Quadrant (except for some anticipated tweaks when particular providers come back with answers to some questions), which has twenty providers. Although Gartner normally doesn’t do hands-on evaluations, this MQ was an exception, because the easiest way to find out if a given service can do X, was generally to get an account, and attempt to do X. Asking the vendor sometimes requires a bunch of back-and-forth, especially if they don’t do X but and are weaseling their reply, forcing you to ask a set of increasingly narrow, specific questions until you get a clear answer. Also, I did not want to constantly bombard the vendors with questions, since, come MQ time, it tends to result in a firedrill whether or not you intended the question as urgent or even particularly important. (I apologize for the fact that I ended up bombarding many vendors with questions, anyway.)

I’ve used cloud services before, of course, and I am a paying customer of two cloud IaaS providers and a hosting provider, for my personal hobbies. But there’s nothing quite like a blitzkrieg through this many providers all at once. (And I’m not quite done, because some providers without online sign-up are still getting back to me on getting a trial account.)

In the course of doing this, I have had some great experiences, some mediocre experiences, and some “you really sell this and people buy it?” experiences. I have online chatted with support folks for basic questions not covered in the documentation (like “if I stop this VM, does it stop billing me, or not?” which varies from provider to provider). I have filed numerous support tickets (for genuine issues, not for evaluation purposes). I have filed multiple bug reports. I have read documentation (sometimes scanty to non-existent). I have clicked around interfaces, and I have actually used the APIs (working in Python, and in one case, without using a library like libcloud); I have probably weirded out some vendors by doing these things at 2 am, although follow-the-sun support has been intriguing. Those of you who follow me on Twitter (@cloudpundit) have gotten little glimpses of some of these things.

Ironically, I have tried to not let these trials unduly influence my MQ evaluations, except to the extent that these things are indisputably factual — features, availability of documentation, etc. But I have taken away strong impressions about ease of use, even for just the basic task of provisioning and de-provisioning a virtual machine. There is phenomenal variation in ease of use, and many providers could really use the services of a usability expert.

Any number of these providers have made weird, seemingly boneheaded decisions in their UI or service design, for which there’s no penalty to anything in MQ scoring, but did occasionally make me stare and go, “Seriously?”

I’m reluctant to publicly call out vendors for this stuff, so I’ll pick just one example from a vendor that has open online sign-up, where it’s not a private issue that hasn’t been raised on a community forum, and they’re not the sort of vendor (I hope) to make angry calls to Gartner’s Ombudsman demanding that I take this post down. (Dear OpSource folks: Think of this as tough love, and I hope Dimension Data analyst relations doesn’t have conniptions.)

So, consider: OpSource has pre-built VMs, that come with a set amount of compute and RAM, bundled with an OS. Great. Except that you can’t alter a bundle at the time of provisioning. So, say, if I want their Ubuntu image, it comes only in a 2 CPU core config. If I want only 1 core, I have to provision that image, wait for the provision to finish, go in and edit the VM config to reduce it to 1 core, and then wait for it to restart. After I go through that song and dance once, I can clone the config… but it boggles the mind why I can’t get the config I want from the start. I’m sure there’s a good technical reason, but the provider’s job is to mask such things from the user.

The experience has also caused me to wholly revise my opinion of vCloud Director as a self-service tool for the average goomba who wants a VM. I’d always seen vCD as a demo being given by experts, where it looked like despite the pile of complex functionality, it was easy enough to use. The key thing is that the service catalogs were always pre-populated in those demos. If you’re starting from the bare vCD install that a vCloud Powered provider is going to give you, you face a daunting task. Complexity is necessary for that level of fine-grained functionality, but it’s software that is in desperate need of pre-configuration from the service provider, and quite possibly an overlay interface for Joe Average Developer.

Now we’ll see if my bank freezes my credit card for possible fraud, when I’m hit with a dozen couple-of-cents-to-a-few-dollar charges come the billing cycle — I used my personal credit card for this, not my corporate card, since Gartner doesn’t actually reimburse for this kind of work. Ironically, once I spent a bunch of time on these sites, Google and the other online ad networks have started displaying ads that consist of nothing but cloud providers, including “click here for a free trial” or “$50 credit” or whatever, but of course you can’t apply those to existing accounts, which makes every little, “hey, you’ve spent another ten cents provisioning and de-provisioning this VM” charge which I’m noting in the back of my head now, into something which will probably annoy me in aggregate come the billing cycle.

Some things, you just can’t know until you try it yourself.

Comments are closed


  • Treb Ryan says:


    Great point. And thanks for the assumption that we’re not the angry type who calls the Ombudsman. We’re just happy to see an analyst who actually uses the system.

    The problem is the flexibility of the system. We offer 16 different OS images with the flexibility to go from 1 to 8 CPU’s and 1 – 64 gigs of ram. To have preconfigured images for all we would need 8162 separate images listed (and that is growing all the time.) so we just put up the most popular images based on deployments (which might be an interesting research nugget for anyone who was paying attention.) and then let people modify post deployment. Our users love the ability to modify up or down at any time post deployment.

    Knowing that we don’t want to put up 8k+ images we could allow people to preconfigure the size pre deployment but that just adds a step to every user even though 90% of our users use one of the listed image sizes.

    We could allow a hybrid system where you can choose a popular image or pre-create an image prior to deployment. That would save the reboot time post deployment for the first image (though it would make no difference for later images.) That’s an interesting idea. We’ll have the product team take a look at it.

    Let us know if you have any other questions.


  • Lydia Leong says:

    Sure, but why not do it the way lots of other providers do? You get a configurator, you pick CPU and RAM and the OS image independently. It would be a lot cleaner than scrolling through the current images, too, which are ordered in a more or less random way. If you have to clone-and-resize in order to do that, it can happen on the back-end provisioning as just a “resizing” step.

  • Treb Ryan says:

    That came from the fact that unlike other providers we don’t offer fixed image sizes. We allow you to go up and down at any point (most providers don’t let you resize the image so they have to do it pre-deploy) So we put that post deployment so it would be eliminated for most users and they could get going faster.

    Agreed there is an opportunity for hybrid. Pre-configs for people who want a common size, a pre-deploy configurator for people who want a customer size but don’t want to handle configs post deploy.

  • Lydia Leong says:

    Some don’t, but a lot do support dynamic resizing. For instance, Terremark and NaviSite both offer arbitrary OS + CPU + RAM configs at instantiation, as well as resizing of CPU and RAM later. NaviSite can even do it as part of their auto-scaler.