As I wrap this week up in preparation for a much-needed vacation with my wife to the Dominican Republic, I’d like to reflect on one of the most frequent questions I get having to do with metadata management – how to effectively handle the scope of the problem.
Historically, the majority of organizations who tried to capture all the information about their metadata in one huge “uber-repository” failed miserably. Some feedback I got back from customers over the last 15 years suggests that perhaps 2 out of 3 organizations who attempted to implement a single corporate-wide repository killed the project within the first two years. And most of those who would say that “succeeded” cut back the scope of the information they planned to capture in the repository to a subset focused on a single mission-critical system or a more specific limited focus of metadata – such as data analysts capturing information about the data architecture.
The predominant reason for failure was not technological in nature. From what I’ve seen over the years, failure tends to have more to do with two key challenges of metadata management: 1)raising governance issues which previously were hidden based on data redundancies and inconsistencies caused by dysfunctional politics and culture and 2)trying to manage too broad a set of different types of metadata – or involve too many people in changes to metadata management processes - too quickly and at too detailed a level.
Best practices include scoping metadata management into smaller areas where governance can be more easily applied at a more abstracted level of detail in repositories which I call “metadata marts” (see “Applying Data Mart and Warehousing Concepts to Metadata Management*”). While organizations should not underestimate the challenge in federating the metadata across multiple metadata marts, most organization will find this provides a more pragmatic approach to achieving the metadata management goals supported by an “uber-repository”. We discuss how to deal with the broader issue of changes to metadata management processes and governance for service-oriented architecture (SOA) and business process management (BPM) projects in “Aligning SOA and Process Governance: Steps Toward Your Business Service Repository*”.
Recommendation: Prioritize and scope your metadata management initiatives into more manageable bit-sized chunks using metadata mart approaches and tooling, and expect the governance issues with which you have to deal to be more challenging than the technological ones.
*Available to Gartner clients or for a fee
4 responses so far ↓
1 Jeff Pryslak // Apr 14, 2009 at 12:23 pm
Metadata Collection…
for every time i hear that phrase, i can think of two times where customers are thinking that it is an additional project beyond creating a system. While this might be true in some cases, the one you point out here, data analyst capturing information on the data architecture, is something that is built into most data architecture tools now.
If an IT department goes through the steps of properly architecting the use of their data, the end result is a rich set of metadata that spans the Physical Data, Conceptual Data, Business Process Data, Application Data, etc… The metadata captured in this architecture process should be linked to a Data Dictionary or Business Glossary. The really useful metadata here comes from reivewing how the data is used in every one of these architecture diagrams and any additional comments gathered along the way to justify why a set of data is used, and for what result.
In the end, data about the information in a business is where metadata always starts, and typically stays, due to the effort beyond that sphere.
i need to read the SOA/BP paper, that is an area that a few of my customers have delved unsuccessfully in the past few years. Not meeting re-use goals, undefined processes, missing schema source information, in ability to shift existing processes with business changes… this list goes on and on, and metadata about the information is always the first place they turn.
Look forward to an article on how ‘Marts Meet’ or the ‘Mart Broker’ for getting federated answers.
2 Michael Blechar // Apr 26, 2009 at 8:40 am
Yes, increased attention to metadata management raises governance issues – frequently invovling sensitive politically and cultural biases within the organization – which can both slow down the metadata management initiative or jeapordize it. Therefore selecting the “right” metadata to manage frequently must take a pragmatic approach to what can be done successfully given the specific organizational constraints for goveranance.
I address Jeff’s excellent comments in my recernt post of Data Dictionaries….
3 Luciano Quadraccia // Oct 21, 2009 at 6:43 pm
I vigorously disagree.
re: “one huge “uber-repository” failed miserably”
that is not true for us at all. We have been extremely successful. Since the late 80’s we have placed considerable metadata and all change management for our mainframe systems in a single repository.
However, in the last decade, as our systems started expanding outside the mainframe they did not continue to use the “uber repository” for their metadata. As a result these systems have a variety of problems that do not exist at all with the mainframe systems.
The reason why these systems do not use the repository is that the repository team has been starved of funds and the opportunity to extend the repository to hold the new structures, utilities and interfaces required by the new technologies.
And the reason why this is happend is because of the poor understanding of what can be achieved with metadata, particularly when it is centralised.
Articles like this perpetuate this misunderstanding, and encourage our management to starve repository enhancement projects to the point where we no longer have people even thinking about this issue.
Thank You
Luciano
4 Master Metadata Management // Oct 31, 2009 at 3:39 pm
[...] received a response to my blog entitled “Down With the Uber-Repository, Long Live Metadata!” criticizing my recommendation to not try to consolidate all your metadata into one [...]
Leave a Comment