Mark Beyer, Father of the Logical Data Warehouse, Guest Post

Another guest post, this time from my colleague and friend Mark Beyer.

My name is Mark Beyer, and I am the “father of the logical data warehouse”. So, what does that mean? First,  if like any father, you are not willing to address your ancestry with full candor you will lose your place in the universe and wither away without making a meaningful contribution. As an implementer in the field, I was a student and practitioner of both Inmon and Kimball. I learned as much or more from my clients and my colleagues during multiple implementations as I did from studying any methodology. My Gartner colleagues challenged my concepts and helped hammer them into a comprehensive and complete concept. Simply put, I was willing to consider DNA contributions from anyone and anywhere, but through a form of unnatural selection, persisted in choosing to include the good genes and actively removing the undesirable elements.

I first used the term “logical data warehouse” while delivering a market analysis and guidance session for one of our Gartner clients at Sybase headquarters two and one-half years ago. I was outlining that Search added to the data warehouse was not “finished” and was an incomplete evolution. We were discussing technical architectures of the future. I kept pushing because the warehouse was always “supposed to” include all information in the enterprise, but it did not. As a field practitioner, it always seemed there was much more to be done and left even those projects perceived as highly successful as “not being done here”. We relayed how leading Gartner clients were making attempts to put every sort of content into their warehouse beyond structured data and outlined the missing components of the architecture. As we completed the discussion, I was asked “So, what do you call this trend?” At which point I was temporarily at a loss. I did not want to call it a federated warehouse. I did not want to call it a virtual warehouse. Both of those terms fell short and both had negative baggage in their history. I did not adhere to Warehouse 2.0—because it focused too heavily on adding Search (that was not the intent, but that was what the market had decided). So, in a flash of brilliance inspired by frustration, I muttered  “the best name is probably a Logical Data Warehouse,…because, it focuses on the logic of information and not the mechanics that are used.” At this time, my Gartner colleague, Donald Feinberg responded “That’s a stupid name. I’m not going to use it.” Three months later, Donald was using it and like a good uncle in your own family tree, he was touting the accomplishments of the family golden child. Now, in fairness to Donald, he expressed his misgivings to me about the term as well and there was some “convincing” in between.

After on-going vetting of both the architectural approach and the methodology with clients and vendors, and studiously NOT using the term in public forums, we had completed our research and validated its conclusions. The logical data warehouse was part of Gartner’s Big Data, Extreme Information and Information Capabilities Framework research in 2011. It was published in the first week of August, and at the October Symposium in Orlando, Peter Sondergaard mentioned it as evidence of how the use and access to information is becoming a market changing force. The logical warehouse leveraged all of Gartner’s information management themes and brought them together in, well, a “logical information delivery platform”—hence, the logical data warehouse. The logical data warehouse inherits the genetics of best practices for 30+ years as well as the formative concepts of old warehouses and diverse data marts. As leading organizations begin to deploy this new generation warehouse, it will demonstrate its ability to meet the long-desired mission of the enterprise data warehouse of giving integrated access to all forms of information assets. The logical data warehouse will finally provide the information services platform for the applications of the highly competitive companies and organizations in the early 21^st Century.

We are already seeing early inputs to the reference architecture and it is becoming clear that the “title fight” Gartner predicted in the 2009 Magic Quadrant for Data Warehouse Database Management Systems is underway. The DBMS vendors, the data integration and service bus vendors and even some of the application vendors have laced up their gloves and entered the ring with their mouth guards in place and their fists up. The logical data warehouse is the next significant evolution of information integration because it includes ALL of its progenitors and demands that each piece of previously proven engineering in the architecture should be used in its best and most appropriate place. This architecture will include and even expand the enterprise data warehouse, but will add semantic data abstraction and distributed processing. It will be fed by content and data mining to document data assets in metadata. And, it will monitor its own performance and provide that information first to manual administration, but then grow toward dynamic provisioning and evaluation of performance against service level expectations. This is important. This is big. This is NOT buzz.  This is real.

Call me. I have a picture and different forms of the reference architecture are already filling up.