Hyping DataOps

This year’s Hype Cycle for Data Management included a new innovation profile for DataOps, a fledging term that’s just entering the story of some data integration and data pipeline vendors. Since I was one of the authors on the DataOps profile, I wanted to clarify a few things for end users likely seeing this term for the first time. We’ve defined DataOps as:

…a collaborative data management practice focused on improving the communication, integration and automation of data flows between data managers and consumers across an organization. The goal of DataOps is to create predictable delivery and change management of data, data models and related artifacts. DataOps uses technology to automate data delivery with the appropriate levels of security, quality and metadata to improve the use and value of data in a dynamic environment.

First off, don’t get bilked. Just because a new topic shows up on a Hype Cycle doesn’t mean Gartner’s validating it as a legitimate topic you should be spending money or time on. The appearance of DataOps on a Hype Cycle means just that – we’re recognizing there’s some hype around a new term or topic. (If you’re a Gartner client, you can read the full profile here.)

You cannot buy DataOps
While the hype is a long way from peaking, we expect vendors to begin to use this term to denote an entire category of tools encapsulating self-service data preparation, streaming data ingestion and, likely, traditional data integration. Add in some level of automation or machine learning, and you can expect a massive amount of hype, competitive messaging and overpromises, all within the short term. The net result will be the dilution of DataOps as a concept until best practices, and next practices, emerge. However, DataOps is a practice, not a technology or tool; you cannot buy it in an application. It’s a cultural change supported by tooling, and many of your existing tools may be adequate to bring in the required levels of automation demanded by DataOps.

DataOps is about organizational change
Like DevOps, the core of DataOps is a new way of working and collaborating. Unlike DevOps, DataOps collaboration typically occurs between technical and non-technical staff. Because this experience and skills mismatch creates a language barrier between these two parties, we believe a core enabler of DataOps will be something we’re calling data literacy. Data literacy is the ability to read, write and communicate data in context, including an understanding of data sources and constructs, analytical methods and techniques applied, and the ability to describe the use case, the application and resulting value. (Valerie Logan is heading up our data literacy work. For more on data literacy, you can find an excellent special report here.)

Over the short term, we expect to put out some early research on DataOps. Currently, there are no standards or known frameworks for DataOps. Today’s loose interpretation makes it difficult to know where to begin, what success looks like, or if organizations are even “doing DataOps” at all. We want to add some clarity. As always, a core driver of our research is real-world examples and use cases. If your organization is pursuing DataOps, or planning to pursue it, I’d love to speak with you. Leave your contact information in the comments.