More recently I have seen a new pattern emerging – that is a bit of a half-way house.  I would like to call this “middle stream MDM”.  Several different industries have described a need to “master data” with active, living, business led governance, even though the data so governed is used in a downstream data warehouse.  One example would be healthcare; the business driver being the Sunshine Act.  In this scenario healthcare companies are looking to build a “single version of the physician” in order that they can publish the required reports for spend.  Clearly the output is a report – so this is downstream of business applications.  But the scenarios described by users suggest that the this needs a whole lot more than data transformations; what is being asked for are actual data steward who will take day to day ownership of the data and relationships.  The intent is that this new, qualified consistent master data will eventually be the basis for new business process; and may eventually be integrated back to source systems.

This led to two ideas – one newish, the other more obvious.  The obvious one (which we have talked about before) is the “creeping use case” that implies that users may start with a focus on one MDM use case, and over time, grow, expand, or evolve that use case – toward a more operational use case.  The other idea leads to a distinct business requirements and hence new technology or newly assembled technology. 

The need for “middle stream” MDM would be exceeded by traditional operational MDM offerings; and would be woefully under met by analytical MDM or traditional hierarchy management tools.  What is needed is a skinnied down MDM solution that excels at supporting “a day in the life of a data steward”.  At the same time, this skinny down MDM solution could help those midmarket organizations that don’t want to pay for expensive, full blown MDM programs.  This could be a new “in” for the MDM market as s whole.

This is not research; this is not a statement of the market.  This is an idea I wanted to express to see what others thought.  What do you think?  Does this resonate?  Could be an interesting line of research if it does…

PS – come share your MDM story with your peers at Gartner’s upcoming MDM Summit May 4-6 in LA.  Submit your case study for the prestigious Gartner MDM Excellence Award!  Check out the process here.

However, some financial applications, that is, business applications used by financial users that operate on transactions published from operational systems actually behave like business applications.  The example would be a corporate reporting function that initially harmonizes disparate master data for global/corporate reporting that then has to author its own versions (or views) of the same hierarchy and master data.  As such, this application is no longer purely “down stream” or “analytical MDM” in that the data that is now authored has to be governed much as other application specific data is authored.

This domain has been serviced in part by traditional BI vendors; and more recently, by a smaller set of vendors and technologies, as represented by Kalido, IBM Cognos Business Viewpoint, and Oracle Hyperion DRM.  Oracle’s offering came via its acquisition of Hyperion, that had itself acquired Razza (where this functionality originally came from).  Interesting Microsoft’s entry into the MDM market also came from this direction since that vendor acquired Stratature, which was focused more on this “analytical MDM” domain.  

This get’s even more confusing if that same newly authored hierarchy is then to be shared and re-used across the rest of the organization and operational side of the business.  This would lead to the need to govern the new data as if it were re-usable master data, not application specific data.  Of the vendors mentioned above, Kalido has first hand experience of this shift “from managing master data primarily in the analytical world into the operational world”.   Just to round out the point, IBM and Oracle would point other MDM technologies to the operational world.

So this is a tricky area: What seems to be one use case of MDM actually shifts subtly into another; and the technology needed to support that shift changes.  Governance becomes more active, meaning it has direct, day to day involvement from line of business; in analytical MDM governance is more passive (or, as my colleague Mark Beyer would say, nonexistent).

I thought this was a good example of a wrinkle on the way to MDM land.

See you at our upcoming MDM Summit!

In both use cases, the need to provide business users with analytics and analysis on master data, or on evaluations of changes to that master data, is common.  The market (i.e. vendors) addressing the needs of users across both use cases have so far brought to market solutions that tightly embed business analytics, and analysis capability, with (master) data management capabilities.  This tight coupling is reflective of how business applications have evolved over the last 20 years; and it is for this reason that this ongoing development is dangerous to long term success of MDM.

The issue is that MDM is not a business application per se, but it requires a business application that is used to operationalize and support the MDM process.  These MDM processes support numerous business processes, actions and initiatives.  MDM has come about because IT realized that these services need to be independent of business processes and applications – and it is this separation of the master data from the application that crated/needs it that makes MDM work.  We are coming from an era where data and application were designed, built and deployed together, to a new era based on layers of services.  

When Analytics IN MDM is NOT Needed

The tight embedding of business user oriented “what if” capability that models changes in master data, in a business context (as opposed to a purely MDM context) is dangerous to MDM.  This “requirement” has caused vendors to develop business application functionality ‘on top’ of MDM solutions.  This is what we need to get away from!

If we keep designing and developing business application functionality on the actual MDM solutions we will perpetuate the architecture of the last 20 years. And it is this architecture of the last 20 years that has caused the problem that MDM was conceived of to resolve.  The “what if” capability is needed by business users, but it should be designed and built as business applications that consume master data via services, such that the MDM processes can operate as they were meant to – independent of all consuming systems.

When Analytics ON Master Data is Needed

This is not to say that analytics on master data, as part of the MDM discipline, is not needed.  This is a very different statement.  There is a great need for applying analytics and metrics within the context of MDM.  This is what supports the active governance of the process, to ensure that MDM is doing what it is supposed to be doing.  However, most vendors are pretty poor at this.  Most address metrics as if it was the output from a batch routine for loading or processing data.  Most vendors do this because this is how most data quality has been executed in the past.  This is a start, but this is not enough (see next blow).

How has this Risk to MDM Evolved?

Ever since MDM got started there have been times when users have pushed vendors to develop their technology to the point where MDM becomes a lot like a business application – one that does not actually do only what MDM is supposed to do.  In the MDM of Customer Data domain, several early MDM vendors were asked to develop marketing analytics that describe conditions and status of customer interactions and relationships.  In the MDM of Product data domain, several early MDM vendors were asked by their customers to develop workflow engines that supported the creation of product data in one place, rather than across the myriad systems that were used before hand. 

More recently, as the collective view of MDM evolved, and “analytical MDM” as a distinct use case emerged, it was clear that this pattern was repeated in other areas.  In the example given at the start of this blog, financial systems merged “what if” analysis of pending changes to hierarchies (master data) that might signal a change in how corporate reports roll up the transactional data.

The generalized observation is this: as soon as you put into the hands of the business user good, clean, validated master data, they start to ask really good questions that business applications vendors and designers have been trying to answer for years – and were generally unable since the source of the master data needed was outside the purview of those same business applications.

Now there is risk in “doing what customers ask”.  Some authors have said, “your customers are the last place to look for vision” since they are highly motivated NOT to undermine the investment they just made in your technology.  As such, customers should NOT be listened too for ideas for step-change innovation; only for incremental improvements.  So are these examples, of customers driving MDM to be more “business application-like” examples of step-change innovation of where MDM should evolve too?  Or are they short term, erratic, incremental developments?   I think the latter – and a very dangerous latter too.  What do you think?  Care to disagree?  Why?

Operational MDM centers on assuring “single view” of master data in the core systems used by business users; it is where master data is created first; and it is often a common source of concern for many organizations since this is where poor (MDM) process integrity fails.

Analytical MDM centers on assuring “single view” of master data in the downstream data warehouse used most often to supply the data for a business intelligence (BI) solution.  We coined the term a few years ago to highlight the overlap (and differences) between the two environments even though “single view” seemed to be a common goal or requirement.

There are differences in what is called, “master data” in the two environments (hierarchy), and also a big difference in how “governance” is effected.  There are similarities in the use of some (not all) technology, most especially related to data quality and data transformation.

At our recent MDM Summit this topic was of great interest.  Hopefully we have provided “the” note that will close out most, if not all, of the open questions.  There were two other notes just published summarizing some common questions users shared regarding MDM.  Ted Friedman just published Q&A for Data Quality and Data Integration From Gartner’s 2009 Master Data Management Summit.  Don’t be mistaken – this is not all about technology; much of data quality is dependent on “context”; the reason why the data is used and the understanding the business user brings to the “question” has a huge impact on what the data means. 

And John Radcliffe published Q&A on Organizing for MDM From Gartner’s 2009 Master Data Management Summit in which he explores some of the issues related to the organizing for MDM.  Organizing for MDM is never too far from process, and governance, so these questions can get pretty complex, and very quickly. 

Want to meet and chat about all things MDM?  I will be at the Gartner Application Architecture, Development, & Integration Summit  in Las Vegas, NV, December 7^th-9^th. 

In analytical MDM there is no message or alert sent to a business user, that is resolved during the normal cycle of work events.  It would be more of a project, rather than a process.  This is a major difference between the two domains; and a difference that must be taken heed of.  Too many users miss the point and assume that what works with BI will work with MDM.  I hear, too often, the comment, “I can ‘do’ MDM with my BI/data warehouse”.  This is partially true in terms of the mechanics; but in terms of the process.

Mark is very fair, and logical, and he says that there is no governance in BI.  He is very right, since for him, governance implies activity (not just mechanical rules).  So we do agree; we just differ how we name the condition.

Guided Governance versus Mechanical Governance

Given that analytical MDM and operational MDM share common mechanics, it is easy for BI technology to be used (i.e . demonstrated) to support governance.  The problem is that this is not the entire solution.  I saw a vendor demonstrate a new “governance application” that showed, reasonably nicely, numerous mechanical activities a user would follow day-to-day.  These mechanics supports an operational MDM environment – they could be seen to support day-to-day, business user activities that they would have executed anyway.

What was missing from the briefing was, as I said on the briefing, “guided governance”.  The vendor had no demonstrable technology that provided the business user with analytics and metrics related to data quality, process exaction, or process design.  As such, there were no guard rails or searchlights highlighting where business users (i.e. stewards) need to work their magic.  It is like piloting an aircraft with a manual; everything goes great until environmental conditions change for which the manual cannot predict and instrumentation is needed to guide the user to make a decision.  As with the example, bad news is what happens as a result of this lack of instrumentation.

I called this instrumented environment, “guided governance” versus the more common “mechanical governance” that this, and most other vendors, is initially focused on.  It won’t be long before users help the vendors out and tell them what they need (once they start flying those planes).

• identify erroneous master data that need resolution (in order to yield “single view”)
• explicit intent to clean up master data at source
• resulting output (clean master data) is used in any and all business applications used by line of business

The second item could be said another way: explicit intent to clean up the business processes that crate the bad data in the first place.

As such, operational MDM involves business users in real work (over time, less then what had been executed before MDM) to resolve issues with bad master data.  Process chances result in possible changes to application strategy; changes have to be made to legacy and packaged applications that hitherto fore had not awareness of the MDM discipline.

A slightly weaker pattern related to MDM “use case” was visible in Business Intelligence (BI) land.  When IT builds a data warehouse to support BI it has to identify data sources, and that data is gathered together (federated), and cleaned up (merged, transformed) and stored in the data warehouse – on which all manner of analysis is performed.  Mechanically, some of the cleaning routines and rules generated from them, look and smell like the same as those that would be used in operational MDM.  But, there were also some differences, and this is what helped us create the name for this next pattern (analytical MDM).  The common characteristics that describe this use case were (and still are):

• identify erroneous master data (among all the other data targeted for the data warehouse) that need resolution (in order to yield, in the data warehouse, “single view”)
• no desire to clean up master data at source
• resulting output (clean master data, and all the other data to be stored in the data warehouse), are not used in any business application, only in the BI applications using the data warehouse.

What gets real interesting with analytical MDM is the third bullet.  In the last few years more and more applications have been built on this “BI data warehouse” that create new data; and this data has to be stored, managed, and governed, like any other business data.  As such, this “BI application” behaves like a business application.  This is important.  We need to keep things simple, and re-define things.  A BI application is one that is used for reporting and analysis only and does not create any new business data; a business application may also have analytics, but it does tend to create business data.

For BI applications, analytical MDM is adopted.  For BA, operational MDM is adopted.

There are at least two other use cases for MDM – but they are much weaker to discern.  I will follow up on those in another post.

See you next week at Gartner’s MDM Summit!

Over the last 5 or 6 years this focus has broadened in scope and has since spawned other drill down topics, such s Procurement MDM which has, itself, split into MDM of Supplier data and MDM of Purchased Part.  Holistically though, MDM of Product Data generally means mastery of “thing”.  Thing is a kind of ‘province’ in that most “things” have common characteristics or challenges; and characteristics and challenges across ‘provinces’ are not similar (think customer/supplier master data).

Well this forward looking user was just starting to roll out an MDM application (single solution, single vendor) for both buy-side product data as well as sell-side product data.  I say this is forward looking because to date most users, if they have gone this far, have assumed that a different solution and possibly different vendor, are needed for each.  Sell-side MDM is much more mature – the mastery of the data was separated from the business applications that created – and consumed the data – 6 or more years ago.  For buy-side, the procurement and spend data management vendors have not yet “let go” the data and so users on the buy side assume that procurement vendors and spend data management vendors are also providers of MDM of purchased part solutions.  Like sell side, the separation of the data and the application will take place – it is just taking a long time.

So this client has taken a leap of faith – they hope several things:

• The MDM vendor can in fact cop with the complexity of requirements for both buy and sell side
• The business applications and analytics solutions, used by business users, to guide selling and buying decisions, will easily integrated with emergent MDM infrastructure
• IT can create a new discipline that, when needed to develop and deploy new business applications and analytics, they do NOT develop or deploy them in the MDM application itself, but as part of the business application and analytics layer.

When users ask for more analytics and business process support the natural and obvious choice will be to “add it to the MDM application”.  After all (the logic will run), that is where the really good clean data resides.  But in adding business application and analytic logic to the MDM infrastructure layer, IT would be reverting back to what it has been doing for the last 20 years – that created the mess in the first place that led to the conclusion that some change in “Integration” needs to take place!

There is a trade-off that emerges: MDM leads to the separation of master data from business application/intelligence.  IT is shifting from 20 years of vertical silos (application plus data) to a new framework of horizontal services (application, analysis, information management).   In the process, and it is a long process to get from a) to b), the IT stack does get more complicated.  So short term costs could increase, flexibility may not accrue immediately, and complexity may increase also.  The pressure to get MDM firmly established for all important master data objects needs to be kept up else the business will not see the benefits and the risk is that MDM will fall from favor for a traditional (silod) business-as-usual approach.

So it will be an interesting time for this user: Great vision, great strategy, hopefully not too far ahead of where the vendors can meet the user needs.  I will be watching very closely – as will many of you, I know.

I have written about “rules” in an MDM context before.  How an organization masters rules is not a new topic; analysts have written about (business) rules engines before.  But MDM is bringing a focus to the topic again, in the context of:

• What type of rules are there?
• Who authors such rules?
• How are such rules used?
• If rules or parts of rules (or rule making processes) are re-used, how can this be made effective?

It is the last question that is making users of MDM conclude that rules and MDM need to be related more closely.  And because BI is immersed in rules, the connection between BI and MDM is more important.  A client inquiry brought this connection unto clear focus today with the following description:

“We have been developing BI for many years, and we recognize that rules that adjust data in some fashion exist at many places in our systems, from initial identification, transformation, load, then aggregation, and even ‘post BI’ since users then take data and drop it into a spreadsheet and do more manipulations.”

These rules, in the context of BI, are what constitute aspects of what I call, “passive governance”.  My college, Mark Beyer, would chuckle and correct me with, “Andrew, this is not governance at all” and Mark is correct; I am just a little kinder and I accept that these rules have a job, that is to clean up and process data in a way that should be meaningful to the user/business.  As such, when “active governance” is established (aka MDM) then rules are again created to clean up and process data in a way that is meaningful to the user/business.  The different is that in BI land, these rules are created and managed by IT independent of the operational system; with MDM these rules are defined by business and directly interacting with the operational system.

So the challenge for me is much clearer – after this one dialog with a client – spelled out the obvious.  Over time more and more users will rationalize that they need to apply MDM disciplines to rules that process master data; and also, beyond that, integrity and re-use of those rules, even those that process non master data, can also yield business benefit through more effective management.  The connection between mastery of business rules, and how MDM tools help, will only increase in strength in the next few years.

It seems the Rime of the Ancient Apps Manager/Mariner should be something like:

Rules, rules, every where,

And all the re-use did shrink,

Rules, rules, every where,

And not an object to link.

Preparations this week focused on getting our last presentations into our editing machine such that they can be formatted, made consistent, and cleaned up for public consumption.  Once the presentations are completed we then begin the slow process if distilling from the key messages, and creating a body of work that is to be published over the next couple of quarters.  Our Summits force us to put on paper all the good, new ideas and content that we have explored with users recently that has not hitherto been published.  These Summits, though a lot of work, are a great way to keep forcing our brains to dump content onto paper, thus freeing up more space for the next set of client interactions.

I had some interesting inquiries this week.  Here is a smattering of the more notable:

On Monday I had a real interesting call with a large industrial end user client that wanted to explore options for – wait for it – “cloud based MDM”!  Well, I knew I was in for an exciting conversation already since that is an odd topic.  I have blogged on this before(SaaS moves the process boundary, does not necessarily change the process, and can make integration of master data more complex); there are not real cloud based MDM offerings today, and really only a very small handful of end users have asked about this – more for interest than anything else.  Only one serious vendor has talked to me about this opportunity – and a credible vendor at that.  But, as I have said before, this is a complex topic.  On the one hand it makes sense that aspects, even all of MDM, should/could go outside the firewall, but the costs to move to that state increase for each and every object.  Those costs relate to integration, synchronization, management and so on.  Application infrastructure gets more complex as a result – until and if enough master data exists outside the firewall at which point the costs, and complexity, fall.  But there must be a tipping point at which this takes place.  Question is – which industry, which business processes, and which master data, are close to that tipping point?  

On Tuesday I had a real good call with an energy user.  This was a large organization with a strong IT shop; I spoke with several senior architecture folks were trying to invigorate their overall information management (IM) strategy.  They knew that MDM played a role in overall IM but they were not totally clear what the relationship was; and they also had an issue with “getting started” – how could they get interest from the business to help them with their IM strategy?  We explored MDM and its connections to IM, as well as to Business Intelligence and Business Applications, and very quickly the client realize that MDM is a great place to start their IM invigoration efforts!  I suspect that I will be talking with them a few times in the next few months to help guide them through their MDM launch.

On Wednesday I presented on MDM vendors to our MDM Special Interest Group (SIG) that is part of our EXP program’s Best Practice Council.  I highlighted the lenses through which we look at the MDM landscape (data domain, use cases, industries) and then talked about how the vendors align both in terms of capability, versus direction.  Many vendors are “good” at one thing/area, and have plans to be “good” at many others.

On Thursday I had a call with an end user firm that I know very well.  In a previous life I worked for an organization that had a business relationship with this client, so I knew something about their business.  They are in consumer goods/fine chemical industry segment, and they are struggling with one aspect of MDM – that being the link between MDM and product design/development, clinical trials, and operations/manufacturing.  The client has a lot of experience with operational MDM for traditional objects, such as customer and product, but this aspect of MDM was more focused on multi-enterprise considerations of sharing complex and confidential information that drive product design, development and acceptance.  

On Friday I joined one of our more interesting virtual research communities (B2B) that span several other formal communities, to explore how business to business is evolving.  My interests, including Supply Chain Management (SCM), concerned both the drivers for why business processes are moving outside the firewall, and with Master Data Management (MDM), what are the barriers and inhibitors to that movement. By exploring one specific phenomenon, related to e-Invoicing, we concluded that we need to identify scenarios by which users can evaluate a longer term B2B strategy.  You might not see the connection – between e-Invoicing and long term B2B strategy – but it is there.  More on this later…

So it was a great week – brimming with client interaction, and research production.  Time for a rest, and a cold one.  Have a good weekend.

“…MDM projects harmonize after the fact, that they pull from the source system, they merge and match, they run  rules and check results on the back end.” 

This was in reference to a specific user implementation of MDM, apparently.  I see a lot of this type of activity, and I would not call this MDM.  I would call it data quality activity, or a data quality project: You do something “after the fact”, to clean the data up, as oppose to change the process that created the data in the first place. ‘Real’ MDM is more focussed on the process change AND data quality.  Data quality is a capability that has to be applied.

Analytical MDM is all about cleaning up the master data after event; post transaction; downstream, in a data warehouse in support of building reports etc.  Operational MDM is all about cleaning the data source (which COULD include, the point of authorship), and also about cleaning up the processes by which the poor quality data would have been created in the first place.  That is the main difference between the two efforts.  

Also, MDM is not a project.  Projects have an end or complete data; MDM does not.  MDM is a way of managing certain types of data – so it does not “end”, it just changes, and continues to change as the business changes.

Overall the article goes on to clarify that the description was an MDM “case study” and the article did represent MDM fairly, but I was not quite “on board” with the earlier part, as you can see above.  Seems like the author wanted to create a little hype in order to drive hits to his site.  Well, we are all subject to that little sin now and then…

Submit a customer case study to win the Gartner MDM Excellence Award 2009!  Go here: http://www.gartner.com/it/page.jsp?id=851612&tab=special&params=pg,zzz.html