Gartner Blog Network


I Don’t Always Evaluate PaaS with a List of Criteria, But When I do, it’s This List.

by Richard Watson  |  July 16, 2015  |  1 Comment

I talk to many Gartner clients tasked with evaluating PaaS for developing and operating cloud applications.  Many of them are creating sets of requirements to use in RFPs and RFIs with PaaS providers. Others are just kicking the tires of the services and seeing what happens when they create and deploy applications. I think both approaches are valuable, particularly with cloud services that are easy to get started with.

My colleague, Kyle Hilgendorf uses the analogy that we must become food safety inspectors, to use structure and a repeatable process to get away from ad-hoc cloud provider evaluation. We can’t inspect every restaurant ourselves before we eat there, so we rely on 3rd party inspectors. Similarly, when we inspect, assess, and evaluate public cloud services, the details really matter. Deploying a simple demo app on a PaaS will usually work. But, those first impressions can deceive.  What about the first real app you attempt to deploy on that platform? Or the tenth? Or will the provider’s failover automation work when you really need it to stay in production serving requests?  Together, Gartner and our clients must be the public cloud inspectors, inspecting cloud services on the behalf of our business and end users.  But, what should go into that inspection checklist?

This week, Gartner published a research project that I’ve been developing for quite some time. “Evaluation Criteria for Enterprise Application Platform as a Service” (Gartner for Technology Professionals subscription required) is the result of multiple years worth of customer discussions, personal PaaS testing and working with providers (and as a provider) in the PaaS market.

This recenly published document provides cloud decision makers with a comprehensive set of criteria for assessing application PaaS (aPaaS) cloud providers and services such as Microsoft Azure, Salesforce (Force.com and Heroku), Google Cloud Platform, Red Hat OpenShift Online, Progress, Mendix or IBM Bluemix.

The list of criteria identifies current capabilities that must exist in an aPaaS for it to be considered enterprise-ready for Web-facing customer service and internal applications, and it describes future-looking requirements that enterprises will need as their cloud application development, delivery and operations grow in sophistication and scope.

The evaluation criteria are organized in the following categories:

  • Baseline criteria: This is the baseline functionality that a service requires to be considered an aPaaS.
  • Planning the architecture: This set of criteria covers aspects of a PaaS that architects and developers need to consider as they choose whether the application fits on the platform and as they plan an application.
  • Developing the application: This involves capabilities that you need when you are building and testing applications for the platform, including tools you can use in a local development environment, APIs you can code against and services that are available with which to compose applications.
  • Deploying and operating the application: This specifies DevOps and infrastructure operations functions that need to be automated to help you deploy code and operate applications in production.
  • Governing and managing service levels: Once the app is in production, you need to manage its lifecycle and service levels and govern usage of the platform itself.

The criteria categories deliberately use both verbs (plan, develop, deploy, operate, manage and govern) and nouns (architecture, development, DevOps and service level) to illustrate that these criteria are certainly PaaS features, but they are features that are necessary to implement the activities of a software delivery process.  The figure below gives you an idea of the subcategories within those categories that are covered by the 187 separate criteria (yep!) across the 4 categories.

paas criteria categories

You may recognize that these categories roughly fit a software development life cycle (SDLC) for an application. The document is not a guidance framework to implement a process for the life cycle of an application, but the categories are organized this way to recognize that an aPaaS is much more than an application platform: It is a set of services that should catalyse a team standardizing development and operations of applications on the platform.

So – boiling this down … it’s a big list of requirements. Is that it?

Looked on from above, as a complete set, this list, with categories and subcategories represents my functional view of PaaS. In the document, I prioritize those criteria, refining my view of what is essential for PaaS and what is not. So, a subset of the requirements in the list are ‘required’, These are essential, must-have features that are needed to build, deploy and run applications and to manage the platform itself. Some of the requirements in the list are ‘Preferred’. These are supplementary features that are not necessary to satisfy the fundamental requirements of the typical enterprise but that are frequently desired to address specific needs such as disaster recovery (DR), or portability across an OS/infrastructure as a service (IaaS). aPaaS providers that fulfill more preferred criteria are likely to be the most feature-rich and forward-looking. finally, some of the criteria are ‘Optional’. These are features that are necessary for specific use cases but not needed for all applications. For example, building mobile applications requires a specific set of extra functionality that is not needed for other web applications.

Or, in other words ….

i dont always paas

 

The research document (40 pages in length – and no, I didn’t have time to write a shorter one) includes a downloadable spreadsheet customers can cut and paste into RFIs/RFPs or to change the priorities themselves for their particular scenarios.

I am proud of this research and many customer discussions went into creating the criteria list. Market leading PaaS vendors also reviewed the criteria, and I’m very happy to say we disagreed on the importance and inclusion of many criteria. This illustrates that no two PaaS offerings are the same, and that it is always worth clients’ time to dive down and inspect the fine details of what they are offering.

I am confident that this documented set of criteria and the spreadsheet will help many organizations cut down the time it takes to build evaluation criteria and questions for PaaS providers.

If you are a Gartner client, but not a Gartner for Technical Professionals (GTP) client, please contact your account rep to ask about GTP.

I will also be discussing the major differences between PaaS providers in a session entitled ‘Public PaaS are not all the Same’ at Gartner’s Catalyst conference in San Diego, Aug 10-13. Hope to see you there!

Additional Resources

View Free, Relevant Gartner Research

Gartner's research helps you cut through the complexity and deliver the knowledge you need to make the right decisions quickly, and with confidence.

Read Free Gartner Research

Category: catalyst  cloud  paas  

Tags: cloud  paas  

Richard Watson
Research VP
5 years at Gartner
21 years IT industry

Richard Watson is an analyst in Gartner's Technical Professionals Research Service. He advises clients on cloud computing, application architecture, and application platforms. ...Read Full Bio


Thoughts on I Don’t Always Evaluate PaaS with a List of Criteria, But When I do, it’s This List.


  1. […] I Don’t Always Evaluate PaaS with a List of Criteria, But When I do, it’s This List. “…The list of criteria identifies current capabilities that must exist in an aPaaS for it to be considered enterprise-ready for Web-facing customer service and internal applications, and it describes future-looking requirements that enterprises will need as their cloud application development, delivery and operations grow in sophistication and scope. … The criteria categories deliberately use both verbs .. and nouns .. to illustrate that these criteria are certainly PaaS features, but they are features that are necessary to implement the activities of a software delivery process. …” Via Richard Watson, Gartner […]



Comments are closed

Comments or opinions expressed on this blog are those of the individual contributors only, and do not necessarily represent the views of Gartner, Inc. or its management. Readers may copy and redistribute blog postings on other blogs, or otherwise for private, non-commercial or journalistic purposes, with attribution to Gartner. This content may not be used for any other purposes in any other formats or media. The content on this blog is provided on an "as-is" basis. Gartner shall not be liable for any damages whatsoever arising out of the content or use of this blog.