Michael Blechar

I recently received a response to my blog entitled “Down With the Uber-Repository, Long Live Metadata!” criticizing my recommendation to not try to consolidate all your metadata into one “uber-repository”. Interestingly, while the respondent mentioned some success his company has had with that approach, he then went on to lament that the amount of metadata being managed in the repository was incomplete and that management did not appear to see the value in expanding its use to all the other metadata in the enterprise. And as a pragmatist, my response would be – good for them! Different business opportunities and threats ought to factor heavily into the decision of the how much rigor and breadth of scope you should apply to your data (and metadata) management.

I tend to find that there are two main issues which are incorrectly getting bundled together when Gartner clients want to address metadata management. And while they seem to want to make this mostly a technological decision, the fact is that it should be secondary to other issues. The first and more fundamental question needs to be “which metadata needs to be managed, and with what degree of rigor”. Metadata is pervasive in the organization – so managing all metadata is not a viable option. Like all other forms of data, there will be a subset of the metadata which is more critical to the organization. The most obvious example is the metadata about the subset of your information assets involving your data, processes, architectures, etc, which you consider to be “master data”.

In other words, as part of your enterprise information management strategy, you need to identify the master metadata and provide a more rigorous management strategy for it than other forms of metadata. And like all other forms of master data, there needs to be a business case in terms of governance, risk, compliancy, cost and opportunity for spending the extra resources to manage that master metadata (see "Mastering Master Data Management*").

Typically, an organization places top priority on improving the usability of its most valuable information assets (for example, customer, product and supplier data). Generally speaking, the more valuable the information asset, the more metadata there will be about it, and the more valuable the metadata will be. In other words, metadata is what unlocks the value of data and, therefore, requires management attention. The most valuable information assets ought to receive more management attention than those which are less valuable.

This is a related but different issue to the one concerning how to manage the metadata. As indicated in Gartner research note  “Applying Data Mart and Warehousing Concepts to Metadata Management*” there are many approaches to managing metadata ranging from leaving it uncoordinated in multiple places and tools, to consolidating subsets of it in metadata marts/warehouses for reporting purposes, to everything in between including bridges between metadata stores and real-time federation of metadata across stores. The key is applying the right approach to the right problem. While a single uber-repository for all metadata is not feasible, there will generally be a subset of metadata which needs to be managed in a robust manner – such as through physical consolidation or federation of master metadata for change impact (and other forms of) reporting.

Therefore, the take-away point here is that you ought to look at your master data management (and critical business process management and service-oriented architecture) efforts to identify which data is most important and make sure the ”master metadata” related to that information gets adequate attention – perhaps in terms of it being placed in a metadata mart/warehouse. Other metadata may best be managed using other approaches like bridging, federation, or simply looking in multiple places for metadata related to information assets. In the end, good metadata management involves evaluating risk, cost, opportunity and governance issues to make the right business decision on which metadata to spend the most resources managing, while also picking the most appropriate approach to meet those management needs – one size metadata management does not fit all!

*Available to Gartner clients or for a fee

The main path for applications development and deployment has historically been the business analyst (frequently residing in IT) getting requirements for some build or buy project and running it through the development or implementation process with limited collaboration until the application solution is ready to be tested and deployed. But the times they are a’changing. Increasingly, the non-IT operating units are taking on the responsibility for defining the business architecture and business process analysts reporting directly to the business units are more proactively working with developers and package implementers to assure the right solution gets implemented. But this is only the tip of the iceberg….

New business process management suites – technologies which allow the business analyst and users to design and implement applications using assembly of software and data services and workflow automation of processes – is creating a whole new path for applications development and deployment. This, of course, significantly impacts both the application and data architectures and requires increased collaboration and coordination between IT and the business units – involving people, process and technology.

As I mentioned in my last post on “Building Greater IT Collaboration With The Business Units”, a good place to start in fortifying these relationships is on enterprise business process improvement (BPI) initiatives, since they require greater collaboration among business users, enterprise and business process architects, and analysts and their counterparts in IT to be successful. These roles are supported by technologies – including business process management suites, BPMSs, that are part of an enterprise BPI reference architecture (see A Reference Architecture for Collaborative Enterprise Business Process Improvement Technologies*). These technologies must be bridged to be effective (for example business modeling tools and IT modeling and requirements management tools).

For more information on BPMSs I recommend reading the research notes below written by my colleagues in the business process management team*.

-Magic Quadrant for Business Process Management Suites
-Selection Criteria Details for BPMSs, 2009
-Using the BPM Four Corners Framework to Evaluate a Vendor’s BPMS Strategy
-Signs That a BPMS Vendor is Following One or More Technology Evolutionary Paths
-Using the Four Corners Framework for BPM and BPM Usage Scenarios to Select BPM Consulting Vendors
-Gartner Evaluates Microsoft’s BPM Strategy and Partner BPMS Products
-Gartner Evaluates Oracle’s BPM Strategy and BPMS Product
-Gartner Evaluates SAP’s BPM Strategy and BPMS Product

*Available to Gartner clients or for a fee

One of the highest priority objectives of most IT organizations is to collaborate better with the business units they serve. A good place to start in fortifying these relationships is on enterprise business process improvement (BPI) initiatives, since they require greater collaboration among business users, enterprise and business process architects, and analysts and their counterparts in IT to be successful. These roles are supported by technologies that are part of an enterprise BPI reference architecture (see A Reference Architecture for Collaborative Enterprise Business Process Improvement Technologies*).

Business architects and process analysts are increasingly using business process analysis/modeling tools to collaborate with enterprise architects and to visually document requirements and pass them on to IT application development and application package implementation personnel (see Focusing Your Business Process Analysis Tool Selection*). At the same time, these models are being used to do process composition using a business process management suite (BPMS) which enables the business units to develop application solutions on their own (see Magic Quadrant for Business Process Management Suites*).

The complexity of new service-oriented architectures (SOAs) and business process management (BPM) initiatives is driving organizations to demand technologies and approaches that enable them to use more-abstract "model-driven" ways to deal with the problem; however, not all model-driven solutions are the same (see The Changing Concept of Model-Driven Approaches*). Although current technologies fall short, new collaborative platforms which are based on modeling principles are on the horizon (see Trends in Collaborative Modeling*).

Regardless of what may be in the longer-term future, new trends in IT and business process modeling methods, as well as the tools that support SOA and BPI initiatives, are changing the face of model-driven development right now (see Trends in Model-Driven Development, 4Q09-3Q10*). This includes the use of agile application development methods in concert with reusable design patterns, frameworks, technical models and pre-built software as a solution (SaaS) content (see Determining Whether OOAD or CBD Best Supports Reusable Software and Services*).

*Available to Gartner clients or for a fee

When service-oriented architecture (SOA) platforms started to be implemented several years ago, I tracked the success and failure of the early adopters doing service-oriented development of applications (SODA). Most seemed to understand that building reusable software and data services was different from building siloed applications and required different development approaches. Most elected to use methodologies based on object-oriented analysis and design (OOAD) principles. While many originally thought they were being successful, subsequent attempts at reuse showed that many of the services being built were at the wrong level of granularity for assembly purposes when related to business processes.

Since many of the computer-aided software engineering (CASE) tools from the 1980s and 1990s were focused on assembly and reuse of business and data logic, and successfully used flavors of methodologies with a “component-based development” mentality, it became obvious that this might be a better approach to the design of services – or at a minimum some of its concepts could be applied to OOAD methods (and, also business process management methodologies) to get a more practical level of service granularity for reuse than objects.

Application and solution architects and analysts should be interested In the just-released research note by David Norton and me called  “Determining Whether OOAD or CBD Best Supports Reusable Software and Services*” where we present three findings:

When surveying stakeholders during planning activities, architects and analysts often hear a common refrain: "Managing information as an asset is core to our business strategy." While this objective has been popular for decades, it is getting renewed attention, notably from the executive suite – especially in terms of supporting business process management (BPM) and improvement opportunities enabled by service-oriented architecture (SOA) .

In my recent post “Master Data Management Enables BPM and SOA” I discussed the importance of proactively (re)designing data and creating new data services based on the needs of current and future BPM and SOA initiatives. But which information requires attention, and how should it be designed and managed?

Enterprise information is not the same as all information in your enterprise. Without this scope of understanding at the outset, the goal of managing information as an asset may quickly devolve into a "boil the ocean" activity with no tangible business result. Each day, organizations generate and consume vast amounts and different types of information, within and across their borders. Only some of this information is considered fundamental to the business strategy and worthy of enterprise-level attention. In "Gartner Defines Enterprise Information Architecture*" we stated that the focus should be on those information assets deemed to have enterprise significance or necessary to effect business change.

Definition: Enterprise information is the diverse representation of significant content that is consistently shared across multiple business processes and capabilities.

Master data is an example of enterprise information. Master data is the consistent and uniform set of identifiers and extended attributes that describe the core entities of the enterprise — and is used across multiple business processes (see "Mastering Master Data Management*"). Master data is enterprise information, because it is significant to the success of strategies requiring the accuracy and dependability of information across core processes.

I’d also like to point you to two research notes by Gartner which go into greater detail on the scope and best practices for enterprise information management:

Managing Information as an Asset: Enterprise Architects, Beware!*.  This describes differences between information assets, liabilities and distractions.  It also includes criteria for determining the impact of information on the enterprise.  Determining value based on impact is a logical place to start as well.

Best Practices for Managing Enterprise Information* uses the Run, Grow, Transform Business Value Framework to describe the principles and methods for managing enterprise information to maximize its value across the organization.

Net: Remember to coordinate changes to business processes and applications with the enterprise information architecture in order to maximize the value of information assets. It is critical to enabling BPM and SOA projects.

*Available to Gartner clients or for a fee

While I am not a guru on Agile development methodologies (I defer to my colleague David Norton on these matters), nonetheless it is a topic of discussion on almost every call I do with clients interested in improving their speed of delivery of applications or wanting to improve their understanding of user requirements to deliver the right solution. Agile methods and approaches like Lean, Scrum and eXtreme Programming (XP) are valuable when used selectively based on project needs and objectives.

Agile methods are not a panacea

Whenever I hear managers talking about Agile methods I am reminded of a Dilbert cartoon where Dilbert asks his manager for 3 more developers due to the expanding workload. And the manager suggests, instead, that Dilbert use Agile programming methods to improve productivity. When Dilbert responds that agile programming doesn’t mean doing more work with fewer people, the manager effectively replies “Then find me something which does!”. So, the decision to go to Agile methods generally has more to do with how it changes the business outcome versus Agile methods being “better” than traditional development ones.

So what are Agile development methods?

Agile development methodology is a highly accelerated, iterative development process with monthly, weekly and even daily deliverables of priority requirements as captured in user stories, often documented in test cases prior to coding. The principles of collaboration, refactoring and promoting ownership are key differentiators. Agile methods prefer to define themselves in terms of "values, principles and best practices," rather than as "processes and procedures." Agile methods attempt to establish a high level of collaboration among developers and business users. They also attempt to "flatten" the project and organizational structure, often through self-organizing teams. They embrace the notion that requirements change, unexpected requirements appear, priorities shift and development practices must enable quick, accurate adaptation to these changes.

What impact is Agile having on organizations?

In some Type A (aggressive technology adopter) development organizations, agile approaches on certain projects already are providing the benefits of fast, accurate delivery of priority application requirements. Although agility is rarely the dominant approach, even the lessons that Type A organizations have learned from agile development have a positive impact on other methodological approaches in the development organization. Tight business collaboration (on-site customers) is a key success factor with agility, but it’s also the most broken principle ("You can have my domain expert for three weeks, but no more"). The successful adoption of agility requires more business involvement in the development process than most organizations have committed or experienced with older methods.

Net: Every organization should find that at least a subset of their applications are good candidates on which to use Agile methods to improve time-to-market, cost, functionality and agility– and arguably should be the dominant method of choice for most new service-oriented applications development (SODA) projects. Gartner clients will find the following research of value in understanding Agile development methods and when to use them.

“The Current State of Agile Method Adoption*”, “Don’t Let Short-Term Agile Create Long-Term Pain*”, “Agile Estimation and Planning Is Moving From a Dictatorship to a Democracy*”, “Making the Most of ‘Agile’ in a Distributed Development Environment*” and “Case Study: Inovis Uses Agile Methods to Accelerate Product Development*”

*Available to Gartner clients or for a fee

Business agility requires shifting to a service-oriented mentality, enabled by service-oriented architectures. However, unless organizations also shift their people, processes and technologies to service-oriented development of applications (SODA), SOA will fail to achieve its anticipated benefits.

The emergence of Web services and service-oriented development of applications (SODA) provides a major boost for enterprise agility. Without SODA, efforts to bring service-oriented architecture (SOA) into mainstream use will fail, and business process improvement projects will find process agility constrained by the organization’s traditional application architectures. SOA and SODA support the rapid composition of new service-oriented business applications (SOBAs), including software as a service (SaaS) from sources outside the enterprise. SOBAs will increase the need for loosely integrated processes, a hallmark of the SODA concept. Because SODA is a hybrid of other application archetypes, it requires significant people, process and technology changes to be successfully adopted.

In the newly published research note “Revisiting The Definition and Realities of SODA*” we examine a variety of issues ranging from the definition of SODA to assessing the readiness of an organization to take on SODA and implications of how SODA changes applications development. Many of these changes are explored in the companion research notes “The Role Of SW Components and Building Blocks in AD*”, “Use Component-Based Development Methods to Maximize SOA Success*” and “Software As A Service Component Development Challenges*”.

To effectively use SOAs, organizations must shift from traditional forms of building siloed applications based on engineering principles to SODA building shared services across business processes that are deployed incrementally into the application ecosystem in a more integrated fashion. This is not simply a technology problem, but also a paradigm shift requiring substantial changes involving people and process.

*Available to Gartner clients or for a fee

I find it both interesting and frustrating that most business process improvement projects focus so much on process improvement to the exclusion of data improvement. Improving business agility by breaking down years of siloed transactions, programs and processes into more flexible and reusable software services is clearly a good thing to do. But what about the siloed data? These initiatives seem to forget that in most organizations the data also needs to be redesigned for consumption – and have its governance rules changed to be managed – at a level of granularity which is consistent with those of the shared, agile business, software and data services. A failure to address service-oriented data redesign at the same time as process redesign is a recipe for disaster.

In the newly published research note “Why Application and Business Architects and Analysts Should Care About MDM*” we look at how to address this need for collaboration between business and IT when designing services and governing them. We also look at how the business and application architectures relate to the enterprise information architecture, especially with those doing master data management (MDM; see “The Seven Building Blocks of MDM: A Framework for Success*” and “Mastering Master Data Management*”). MDM data services can be a key enabler of business process management (BPM) and service-oriented architecture (SOA).

MDM projects focus on improving and managing the most important data within the enterprise –such as customer and product data. Because MDM takes on the issues of designing shared, reusable data services which use the best source of record for data, it ensures that when new application solutions are being composed/created, the best high quality data will be made available. Moreover, MDM proactively addresses the governance of the data services and publishes which services exist and how to use them. In other words, MDM proactively establishes the data services needed by new BPM and SOA projects in advance of them, allowing them to be implemented more quickly.

On the other hand, MDM data services cannot be built in a vacuum. There is a need to understand how best design the data structures and services based on the needs of the new software services and workflows coming out of the BPM and SOA projects. So, ensuring that the architecture and analysis efforts of BPM, SOA and MDM projects are coordinated in a collaborative manner should be of extreme importance to business and IT management.

*Available to Gartner clients or for a fee

Because I get to deal with both Gartner customers involved with business process management (BPM) initiatives and service-oriented development of applications (SODA) – and the collaboration issues between business and IT analysts – I find that there remains considerable confusion concerning the term “service”. On one hand the business users speak at both coarse-grained levels of abstraction such as having an “order-to-cash service” or in terms of services being more fine-grained business processes, transactions and workflows.

And, on the other hand, I find IT architects, analysts and developers who refer to just about anything which is reusable as a “software service”, especially if it can be invoked from a web browser. This, despite the fact that service-oriented architecture (SOA) application services have a much more specific set of requirements than just “reuse”.

So, to help our clients understand these differences in terminology and what the requirements are to be a “SOA service or application” we have just published a research note on the subject of “Defining Business and SOA Services*”.

In it we explain that, when implemented in a SOA,  a business service is a SOA-based software capability that executes the steps, tasks and activities of one or more business processes in an SOA or EDA runtime environment. A SOA service refers to a unit that follows five principles; it is modular, can be distributed, has a clearly defined interface "contract," can be swapped (substituted for a different implementation) and can be shared.

And, of course, we go into much greater detailed discussions regarding both business and SOA services and provide guidelines and recommendations concerning how to avoid confusion when the term “services” is used in business and IT contexts. One important recommendation is to “know your audience” and how they generally use the term “service”.

*Available to Gartner clients or for a fee

What’s on my mind today is the Open Service Oriented Architecture (OSOA) specification on Service Component Architecture (SCA), "Assembly Model Specification Extensions for Event Processing and Pub/Sub" and the possible confusion concerning how this relates to design and assembly of software and data services from a service-oriented applications development (SODA) perspective. They are related, but have a slightly different focus.

The SCA Assembly Model includes (1) Event Processing, which is computing that performs operations on events, including creating, reading, transforming, and deleting events or event objects/representations and (2) Publication and Subscription (often shortened to Pub/Sub), which is a particular style of organizing the components which produce and consume events in which the producing components are decoupled from the consuming components. Not surprisingly, the Gartner analysts who cover primarily cover SCA are our gurus in service component assembly from an applications integration perspective. Note: for a better historical understanding of the origins and purpose of SCA see Jess Thompson’s research note entitled “Service Component Architecture Is a Winner in the Quest to Establish a Common Notation for SOA*”.

But what of those of us concerned with the design and implementation of software and data services as part of SODA and business process management (BPM)? Are these not the same objects, services and components being referred to as part of SCA? Well, both yes and no. The design of business-oriented software and data services as part of SODA needs to be primarily driven by the business rules and process/workflows identified by the business architects and analysts. The tasks and steps of those process/workflows need to be implemented “at the right level of granularity” by application architects/analysts to address the agility, understanding, reuse and governance of those services by business analysts and developers as part of an evolving application architecture. Those well-designed software and data services in turn need to be implemented as part of a SCA in physical ways which make sense for application integration purposes including transparency, security and performance factors. This requires the application architects/analysts to work closely with both the business architects/analysts and the integration architects/analysts.

So, IMO, SCA is an important standard for those focused on application integration and assembly of physical objects, components and services.  I see a slightly different focus of design used by application architects and analysts for software and data services which are more closely related to the assembly needs of the business analysts – while also collaborating with the integration analysts to get the best physical implementations of those services using SCA types of specifications.

For more information on architecture and developer collaboration issues see my blog on “Why Architects and Developers Need To Collaborate” or my research note with Dan Sholler on “Defining the Discipline of Application Architecture*”.

*Available to Gartner clients or for a fee