Justin + Guy, founders of DataOps.live (The Data Products Company), have been thought leaders in the Data market since they started developing the DataOps philosophy and the DataOps.live SaaS platform over four years ago.
Over that time they have delivered the #TrueDataOps philosophy, authored the book DataOps for Dummies, were among the first to describe building and managing “Data Products” like software products in 2019, joined many industry panels on the DataOps market, presented Master Classes on DataOps for cloud data platforms like Snowflake, and presented alongside global enterprise companies on how they achieved $10’s of millions of dollars in savings deploying the principles of, and underlying technologies to support, #TrueDataOps.
Justin and Guy have put their heads together to share several predictions for the data industry in 2023 around DataOps, Data Observability, Data Catalogs and Data Products.
2023 will ultimately be seen as a breakthrough year for the data market driven most notably by the economic challenges we are facing. Coming off the back of 2 years of COVID, the war in Ukraine, the resulting energy crisis in Europe, lower global economic growth, rapidly increasing inflation, the collapse of crypto-company FTX, and many other factors, the global markets are on the brink of recession. However, the need for organizations to leverage their data to survive, to improve and accelerate their decision making, and to gain economic advantage against their competitors during this period is only increasing.
This means that corporate spending around data, the “manufacturing processes” that refine that data, and technologies that automate, integrate, and generate insights from data, will only increase.
That said, organizations will now evaluate 1) how they can be more cost-efficient and automated around the existing data refinement processes they run, 2) how they can be more productive and deliver faster with their existing skilled technical resources, and 3) how to optimize the overall costs, quality, and reliability of the end data products they produce.
With this context in kind, here are our predictions for 2023.
2023 Predictions:
- “Data Products”, having become a mainstream concept in 2022, will become a collaboratively defined metadata standard that is well-adopted by all the major ISVs, SIs, businesses, and other organizations. Data Products will become a “contract” that includes a clear definition of that product contract, a description of what they will receive, and the expectations consumers should have when they select to “consume” that product (performance, quality, timeliness, etc.)
- Extending this, both Certified Data Products and Community Data Products will evolve as acknowledged categories of Data Products that will be available for consumption within an organization. Certified Data Products will come with greater levels of corporate commitment and higher levels of security, availability, quality, and governance. Community Data Products will be made available as more ad-hoc assets with an associated acceptance of the risk they may represent (much like open source software). A promotion process will exist to promote qualifying community data products to certified data products
- End-to-end, unified metadata observability will rapidly become an absolute expectation for organizations to enable them to operate and manage their estate of Data Products as efficiently and cost-effectively as possible
- Declarative, low-code and configuration will become the norm for building data pipelines, data products, cloud data platforms, and data environments (including long-lived environments like Production, QA, DEV and per feature ephemeral environments)
- Technology independent Cloud Development Environments (CDE), rather that proprietary Cloud Development Environments, will become an expectation for Data Product developers to simplify their working environment, increase the number of products they can configure and control directly from that CDE, and maximize their productivity per working hour
- Data Catalogs will move out of the Trough of Disillusionment (popularized by Gartner) as they move away from harvesting metadata from many disconnected sources, to the data catalog being “fully hydrated” by DataOps pipelines with high-quality, accurate and fresh metadata—allowing enterprises and Business users to 100% trust the accuracy of that metadata
- Federated domain/department-based deployment patterns (Data Mesh, etc.) will become the norm for all companies who want to enable high agility and business responsiveness through the adoption of Cloud Data Platforms
- DataOps will become widely recognized as a broader capability than Orchestration alone, and DataOps vendors will be recognized as either end-to-end DataOps platforms or DataOps point solutions
- Unified Metadata Observability will become recognized as a core capability within the “DataOps” market category, rather than a separate market in its own right
- More data engineering job positions will become active in 2023 than ever before due to the ability for organizations to now “create” productive data product developers from graduate hires. Organizations will enable these new data product developers to deliver production changes and releases within weeks rather than months due to highly controlled and automated DataOps processes
- More DataOps specific job descriptions such as DataOps Engineer will become widespread to define the team member who are the key DataOps experts acting as a force multiplier for the rest of the data teams
- AI and machine learning will start to influence and accelerate both a) the automatic generation of data pipeline & transformation code (SQL, Python, etc.) and b) the detection and prediction of anomalies and issues with production data pipelines
The DataOps market has made some significant strides in 2022—and the team at DataOps.live is thrilled to be a pivotal part of the tremendous value delivery we’ve seen with our customers. Dramatic increases in productivity, along with cost and time savings are benefits that every customer can appreciate—especially in challenging economic times. 2023 will likely see even more proof that the vision laid out in #TrueDataOps is a truly unique approach that is only possible with DataOps.live. We believe that there’s a bright future for DataOps.live and our ecosystem partners.
We hope you will find this article and its predictions useful. On behalf of both of us, we wish you all the most fulfilling and healthy 2023 that we can.
- Justin + Guy