Open source at Gartner: Point taken, I was too broad in my characterization. I read the reports on information management-related topics which for me was maybe a dozen or so areas of coverage, and heard or read a lot of “pay no attention to open source, it’s a fad” advice. I think the right answer was “pay attention, but don’t jump in just yet unless you have good reason”.

Second point on open source: OSS in the information management / BI space 18-24 months ago was still in the techno-geek ghetto and was just hitting the elbow in adoption, so it made sense not to cover. But pooh-poohing the topic is what I saw many analysts doing, e.g. I read a number of “kids in the basement coding” proclamations. 18-24 months ago was too early so your point is valid, but I have this sense that Gartner consistently downplays disruptors until the enthusiasts and early adopters are in, then suddenly Gartner is right there at the meaty part of the vendor curve saying “we called this all along.” With .8 probability no less

Regarding whether I’ve talked with vendors or customers: Yes, a lot of them. Donald’s comment about “the Informatica quadrant” was a pretty broad perception after the shift from the ETL MQ. This one requires an aside:

ETL is perceived as BI only. ETL vendors can only get good growth by expanding outside that niche market and eyeball the huge opportunities in the larger application market. DQ, DP, etc. get them partway there. Your shift in MQ helps them say “use our tools for other things and better manage your data.” The assumption is that completely centralized DI works for operational (as opposed to analytic) DI. Yes for migrations, consolidations, etc. But for broad-based low latency application DI, I am not convinced for a lot of reasons. Licensing, scalability, low latency, high concurrency, architectural match, skill match, etc. are real problems.

This makes a DI MQ hard to accept because the use cases between BI and OLTP have different characteristics and one-size-fits-all product model may not be right. I was harsh in saying this was a travesty, but I still believe the ability to inform is fairly low because of the generality.

Regarding customers / clients of the DI vendors and of Gartner, yes I talk to them all the time. Clients ask me to advise them on enterprise scale DI architecture and product selection. I have to sit in the trenches with the evaluators and users, and I get nailed if I do a face plant when it’s in production. I’ve seen some of the evaluation spreadsheets given to Gartner clients and I stand by the utility criticism. We can talk about those another time as this isn’t the right forum.

You said “The MQ is not the tool to make that decision for you.” My criticism is that clients often use the MQ as a crutch for thinking. While you personally can’t be held to account for misuse of the MQ, there is an undercurrent in Gartner’s messaging that this is exactly what the MQ is for.

You’re being disingenuous when you say there’s no profit in the MQ. Then why do it? This is the vehicle that creates a market perception others noted: “if you aren’t there you aren’t anywhere”. Of course no vendors opt out of participation even though the MQ process is an expensive undertaking and may not match their marketing position. If they don’t they run the risk of bad press through mischaracterization which is a pretty big pressure. It would be fairer to not include them if they didn’t answer, or to stop relying on them for answers and do completely independent product research.

Many vendors see the MQ as the gateway to respectability in large corporate IT shops and I would argue that Gartner plays this up, despite the pragmatism of individual analysts like yourself. My view of the MQ is that it’s vendor research, not product research, but that’s not how clients perceive it.

I have to apologize for steering the discussion into generic MQ-land. That topic and the DI MQ criticism are related but it puts you in a no-win argument defending Gartner policies vs. one quadrant.

On your last point, my statement on breadth was poorly worded.

The DI MQ is in a tough transition period where vendors are shifting from batch movement tools to suites and someday maybe to platforms, The challenge I see is that the MQ rates vendors and not products. Why not state which of a company like Oracle’s or IBM’s products are in scope with the DI MQ? That removes an element of criticism.

I do DI evaluations by looking look at the entire DI suite offerings and the subdomains like DQ, ETL, federation. If the need is a point solution, the point vendors in that domain can be included in the eval. If the need is for components outside their scope then suites are more appropriate and point tools can be excluded, or included for clients with a “best of breed” model.

My experience with some components of suites in the leader quadrant is that they are bad or poorly integrated in the suite. Point vendors can’t easily meet the entry requirements yet have better products for that component.

All the single-technology vendors have their place, so why not provide an MQ that evaluates suites and suite integration, and then separately the core technologies? That would best serve client needs when emphasis is on a subset of the components. My experience is that this is almost always the case.

First of all, thanks for the kudos. Much appreciated. I never thought of myself as tight-lipped and always welcomed open discussion. As long as it is truly open. There were times when I got flak for something I wrote from people that were themselves hiding in anonymity. Not so here. Thank you.

So here it goes, trying to address the major points of each response.

@Yves: If writing 3000 documents mentioning “open source” is not an indicator for “recognizing open-source” then I don’t know what is. Even if only 10% of those did a detailed evaluation of any open-source technology or topic, that would still amount to a lot of research available to our clients (and that’s what counts). We are doing primary research of open-source adoption trends as well as qualitative research in form of things like MQs and other research notes. There is even an open-source software research community that meets every week. I’m afraid, Yves, your outside-in perspective does not reflect the facts, just perception.

Regarding any “skyrocketing sales figures” as in the data that you had provided: it’s still all relative. Just for argument’s sake: If Vendor A, a large blue-chip software company, has 2000 customers and added 20 net new customers in 2008, that would amount to a 1% increase. Good, but not extraordinary. If Vendor B, a small VC-backed software firm, has 10 customers and has also signed 20 net new accounts in 2008, they would have tripled their install base, a 200% increase. Fantastic, but still a blip compared to Vendor A. So, when it comes to who “defines the market” we are obviously going top-down and at some point there is a cut. That Vendor B, despite huge absolute growth, may be excluded from an MQ because it is still largely irrelevant in the greater scheme of things. Plus, there is a finite number of vendors that the analysts producing the MQ can handle in the process.

Right-on regarding the melting tracks in the snow. I would have interpreted it a little different, though, particularly when talking about open-source. Over time, many tracks disappear because the open-source projects seize to exist or become acquired by someone else, and there is no point following each and every new open-source project until they really start laying tracks in the snow. For the record: we are recognizing Talend’s success.

@Carter: Although your comments don’t address the MQ in question, but rather the MQ process in general, I will still need to respond to your comments.

It’s great if you provide feedback about the MQ process and other things at the quarterly AR call. The DI MQ, the one that triggered this whole thread, is updated every year during the summer, and as such, your “18 to 24 months” actually is completely inaccurate.

There is a certain amount of truth in Yves’ comment that MQs reflect “…past adoption of certain technologies by large accounts…”

Again, nobody denies this, but you make it sound as if large accounts were the only clients we talk to. I typically don’t interact with companies á la “three guys and a website”, but a large portion of information comes from the midmarket, not only GE, HSBC, or Vodafone-type organizations. In every other inquiry with a client, someone asks “considering a company our size and industry, who else has done what we are planning?” And that’s when they want to know about “past adoption” not “potential developments”.

A significant source of information for the average Gartner analyst comes informally via phone-based end-user client inquiry.

Informally? It’s not that clients scribble something on a napkin and slide it across the lunch table for us to consume as research. Not really sure what you mean. As far as I’m concerned, those client inquiries are as formal as the come, via phone or face-to-face.

This reliance on end-user clients as a source obviously skews the pool of data points.

You must clearly be joking. What other sources would you expect to see than end-user clients? There are only two types of sources: the vendors and the users. And we obviously talk to both. Would you have any other constituents that I could add to the pool of data points?

Gartner’s client base is not a statistically valid sample for all types of research and only clients that choose to set up an inquiry are counted.

Another bold claim, and rather unsubstantiated. Even if the client base wasn’t a statistically valid sample, so what? I would still argue that I have a very good understanding of what is happening in all kinds of organizations, relative to my coverage. I have also never heard of anybody dismissing my opinion because I didn’t have the statistical proof that my advice was representative of the whole world. Secondly, it’s absolutely untrue (and as a former Gartner analyst, you should know) that we get input only through client inquiry. I don’t live in a vacuum and I interact with people through a wide variety of channels, inquiry just being one.

Thus analysts are getting an incomplete picture and they probably do not realize it.

Oh, please. Thanks for pointing out that we did not interview 6 billion people for every piece of research just so we can claim a “complete picture.” Trust me, we do realize (and so do our clients) that we don’t have the 360 degree view of everything. Any suggestions as to how to correct that?

Finally, regarding those probabilities, I have no problem with you being snarky. Hey, that adds the spice to conversations. I can only talk for myself here and that’s why there is my name on this blog. You won’t find any mention of probabilities in my research notes. If you take issue with probabilities published by others, please raise it there.

@Mark: Thanks for adding your perspective. Like in the posts above I’ll try to address those points that stick out.

I’d side with Yves that until recently open source was largely dismissed as nothing in almost all coverage areas. I think that’s changed significantly in the past 18-24 months.

It’s interesting that you claim to know our view on open-source for “almost all coverage areas”. Have you read all those documents and talked to all analysts that touch open source? I’m on the inside and yet I don’t have that insight just off the bat. Wow. I would agree that we are spending more cycles in the last two years on open-source software. Regarding open-source data integration two or three years ago, there just wasn’t anything to research or report that had a major influence on our clients (and again, that’s what counts).

I think the data integration quadrant is a travesty because of the decision to make it “data integration”. While I understand reason (shift to multiple techniques/technologies, consolidation, drive to a DI platform), it is a unusable for most IT organizations deploying at a tactical level

Sorry, Mark, a travesty? Have you talked to the vendors in question lately? Most of them do not consider themselves “ETL vendors” anymore, and for some time already. Quite the contrary, they provide many different components, ETL being just one, that all sit under a data integration umbrella. As you said you understand the reason, why would that be “unusable” and how would you know that? You see, the MQ is one data point for our clients when they need to be making a decision, in this case about data integration tools. While I’m sure there are clients that just pick whoever is in the Northeastern corner of the MQ, the smart clients also talk to us about their specific requirements, in which we provide more specific advice, whether that’s for a tactical solution or a strategic enterprise-wide data integration standard. The MQ is not the tool to make that decision for you.

Since the evaluation is not the products but the vendors, it’s not possible to use the MQ to choose DI technology appropriate to the situation. When I’m with clients who are Gartner subscribers I have to guide them in their use of these results

Very good. As I just wrote above, the MQ is not the ultimate decision maker (or breaker). Every client is different and so is every DI situation. The point about those megavendors that own a wide variety of tools all relative to the MQ is, of course, a challenge to address. How should one reasonably compare an IBM portfolio of data integration products with, for example, an open-source product that is basically an XML transformation engine with JDBC and flat-file connectivity? You compare the components like-for-like (and there may be many similarities) and then you check what breadth of technology is provided by the vendor (and that’s where you’ll find huge differences).

The challenge for Gartner is having an MQ that has enough vendors in it to be profitable, that makes sense for vendors to participate in

What you describe as a challenge is really a non-issue. There is no profitability of an MQ. And the number of vendors on any specific MQ is irrelevant. Sure, nobody will publish an MQ with a single dot (what market would that be?), equally, nobody will go through the exercise of having 100 vendors evaluated either. It typically balances out somewhere between 5 and 25. If the numbers get too high, the market definition is probably to broad, if too few vendors make the inclusion criteria, the definition is too rigid. It’s also a matter of resource management and timing. Simple as that. If we accepted every vendor in an MQ, we’d be talking to them and their references until the cows came home. And then Carter would come along and start complaining again about the MQ not being published on time.

Secondly, it’s not a vendor’s choice to “participate” in an MQ. In other words, they cannot opt-in or opt-out. If they make the inclusion criteria, based on what we know, they’re in. If the vendor chooses not to respond to the RFI and provide the information required for the ranking (I don’t recall that to happen, though) they will still be evaluated using the information at hand. Equally, if a vendor wants us to drop them from the MQ, mostly because of a sub-optimal position (and that has happened before), we acknowledge that but it’s not going to happen.

Open source is certainly a problem for analyst reports due to inclusion metrics. … I’d hazard a guess that the R project isn’t included in the BI or related MQs yet it’s hugely influential, enough so to make an NYT article

As I wrote before, inclusion criteria are actually adapted so that open-source vendors are not getting shut out because of a lack of license revenue. And you’re guessing right, R is not included in the BI MQ. Whether it’s “hugely influential” is open for debate. Maybe in academic circles, but it sure isn’t mainstream, from where I can see. There are other technologies that are growing like wildfire, think Twitter, and even though the NYT or the WSJ write an article about it, it doesn’t mean, every CIO would stop in their tracks and jump with both feet into whatever new gimmick comes around the corner.

I think you may be right to exclude Talend or Pentaho from the DI MQ on the basis of market penetration and size last year (sorry Yves) but that may be incorrect given the use in operational rather than pure BI ETL scenarios, e.g. Gartner may be looking in the wrong places.

I think you are contradicting yourself here. First you’re saying that the DI MQ is too broad in terms of technologies, now you’re saying we would be looking only in “pure BI ETL scenarios”. The DI MQ is exactly not focusing on ETL, that’s why we expanded the market definition two or three years ago, much in line, in fact, with how the majority of vendors are addressing this market.

@Bob: Thank you for your comments. Your comments about Talend or Expressor not being included in the DI MQ are somewhat incorrect. They are both mentioned in the document (which contains 33 pages, btw, which is much more than just the graphic), along with a significant number of other vendors, such as Attunity, Datawatch, Embarcadero, GoldenGate, SchemaLogic, or XAware, to name a few. All of the above have, however, not made the inclusion criteria and are as such not evaluated with their own dot on the graphic. Ab Initio has been dropped from the evaluation two years ago, but they are also still mentioned.

I believe that analysts have a responsibility to paint a complete picture of the vendor landscape to their client community and to the industry.

Similar to my response to Carter’s comment above, I guess I’m still struggling with what a “complete picture” means and how to get there. Completion in what sense, every vendor, every product, every user? To be honest, I don’t think that’s possible.
Also, our responsibility to our clients is not just one-way, telling the end-user about the vendor landscape. Equally, our vendor clients want to get our perspective on the current end-user trends.

… let them know about emerging players whether it is thru a phone conversation or a write-up on their site. I’d be very surprised if they weren’t doing this.

That’s exactly what we do. While we can’t proactively call every client and tell them about this brand-new emerging technology that just popped up, through written research and during interactions like inquiries, 1-on-1 sessions or even on stage at a conference, we absolutely talk about new developments, trends, or the new cool gadgets.

Why really does the MQ have to be so static?? I think I’ve heard Gartner mention the world Real Time a few times. Maybe Gartner needs a MQ 2.0? If so, I suppose Talend and Expressor-Software would show up.

I think you’re mixing a lot of things here: MQ cycles, real-time (?), and inclusion criteria. First, why do say the MQ is “static”? It’s not, as it is updated on pre-defined schedules, which are necessary because of the time it takes to produce one MQ. It’s not that we could churn out a new version of an MQ over a weekend, after throwing a few eggs at a wall and see where the vendors stick. (I do understand that many people not familiar with the process think that’s exactly how we come up with the positions for the dots.)
Second, “real-time” what? Changing criteria in real-time, monitoring every sale a vendor does, or watching a vendor’s cash position, stock trades, or SEC filings? No. An MQ is a snapshot of a certain market, which typically doesn’t have any erratic movements anyway.
Third, if you really want to put versions against MQ, then we’d probably be at MQ 32.5., with all the changes that have happened over time. Still, every MQ will continue to have inclusion criteria, and both Talend and Expressor need to fulfill those, “real-time” or otherwise.

@Donald: Thanks for your valuable addition to this thread. Great to see another vendor position, no wait, “personal position” by an expert who happens to work at another vendor, and a fairly large one at that. That ok, no danger to your coffee supply?

Ironically, this difficulty (calling out market numbers for inclusion metrics) is very similar to the problems that an open source vendor would face. The monetization is not easily allocated, and I can hardly call up Andy and say “Look at these great numbers from IDC.”

I understand that the way products are sometimes bundled are posing a challenge to break out market-relevant numbers. In an extreme case, I was once evaluating some product adoption where the vendor kept saying “we have 2100 customers”. Portal: 2100 customers. Application server: 2100. Any product component had the same number of customers. That was true magic! Of course, only a fraction of each product component was used, or maybe even installed. Well, I’m getting off track here…

In order to get a sense for how a particular product is used in the market, we rely on that information provided by the vendor. Microsoft is clearly more challenged there than a “one-trick pony” company (no offense to anyone). Of course, Donald, you could always call me up and show me IDC numbers. No issue on this end.

On the other hand, I think Mark makes a good point about the data integration quadrant being too general – it really is difficult to see how it supports actionable purchasing decisions, compared to the old ETL quadrant.

I can’t win, can I? The vendors keep adding stuff to their “data integration suites” (note that they stopped talking about their “ETL product”!) but want to be evaluated only as a provider of just ETL, or just replication, or just changed-data capture? While some tactical decisions, e.g. for solving an imminent replication problem, continue to be made by end-users, there is a large number of clients looking at data integration beyond ETL. Going forward, and certainly not to a surprise by anyone reading up to here, the markets of data integration and data quality continue to converge. Think about all those offerings from Business Objects, DataFlux, IBM, Informatica, Group1, etc. Instead of breaking out individual components from the DI MQ, I could much rather picture a DIDQ MQ at some point in the future. Now, consider this “thinking out loud”, and my colleagues Ted and Mark certainly will throw in their perspective on this.

Think about how customers buy ERP solutions today. Would you expect an MQ just for financials, one for sales and distribution, one for marketing, one for HR, … you get the picture? While a potential user may just deploy the finance and HR modules, they would still make the decision for the ERP provider and look for some insight from the analyst. The MQ helps steer the discussion, but the devil is as always in the details and that’s why clients book inquiries.

We used to joke that the integration quadrant was the Informatica quadrant. Not to suggest favouritism in evaluation, but rather that it appeared that quadrant definitions were weighted in such a way that a specialist, independent, vendor would typically be smack in the middle of the leader’s corner.

Oy. The Informatica quadrant? I’m sure some folks in Redwood City are pleased to hear that. But glad you don’t think that we are giving out favors. No way, we’re a tough bunch.

When I look at the current MQ, there are quite a few “specialist, independent” vendors on the chart, but just one is really “smack in the middle of the leader’s corner” and all the rest are scattered across the MQ. So, not sure I can agree with that assumption. You see, we define criteria and their individual weightings long before we have started talking to any vendor. Criteria and weights also change year-over-year. That’s why a dot sometimes can make significant jumps from one year to the next. Not always a pleasure to explain, particularly when a vendor’s dot went South.

… good old DTS, was a classic cheap operational data movement tool, but I’m not at all sure I could defend it against Gartner’s new DI criteria (as opposed to the old ETL criteria), whereas the new Integration Services clearly does play in that space.

Aha! There you have it. Microsoft has been one of the companies that addressed data integration from a broader perspective than ETL. Case in point: the product’s name is “Integration Services”, not “ETL Services”. And Microsoft wasn’t the only one doing it, obviously, which means, that the market for data integration has progressed, and some companies have boarded the train, and some other have not. Some are running to catch the train (and I would include open-source vendors here), some others deliberately stay on the ETL platform. What a nice visual.

I, for one, would love to get Gartner’s view on how OSS data integration vendors are performing in the high-end scenarios where they are being successful.

You know, you could just set up an inquiry.

I think I need to wrap up here. Don’t want to get RSI from just one blog post. Thanks, again, for all your input. Happy to continue the discussion, here or offline.