CloudOps & DataOps Engineer Luke Crumley describes the lightbulb moment when he fully grasped the potential of DataOps to deliver new levels of speed, innovation and quality for business teams. Data engineering is about solving problems: helping organizations to make decisions based on facts rather than guesswork. As a result, businesses need increasingly powerful tools to help them to gather, sort, manage, manipulate and present insights to support those decisions—solutions like DataOps.live.
In short, DataOps makes it possible for developers and engineers to deliver more, do it faster, and more consistently.
I wasn’t originally a data engineer. Describing myself today as a ‘broad engineer’, I’ve been programming since childhood, LEGO® Mindstorms and websites, dabbling in amateur projects for friends and family. I developed a strong working knowledge of a variety of technologies and techniques. Around 2013, a close friend was working as a software engineer, and said I’d love it.
Looking back at earlier roles, people would use Excel and other rudimentary tools, and write their own scripts. They might have heard about source control and versioning but were not shown how to apply the concepts to their work. The focus was on getting data transformed and producing some business benefit. It was a ‘cottage industry’, and used other tools to try and stitch the separate parts together. People worked in silos. They simply weren’t benefiting from the knowledge and experience accumulated during decades of software engineering, in particular automation and what that enables.
In my previous role, I worked in a cloud engineering team supporting data engineers. We understood the tools being used in the data pipelines, but we didn’t understand the data in the same ways the data engineers did. I hadn’t really thought through the DataOps concept, and the benefits it could deliver: how it could help to solve our users’ problems. Until—true story—I was interviewed by Guy Adams, DataOps.live co-founder and CTO, and his team for this job. It all clicked into place. I realized the teams I’d been supporting would benefit massively from a DataOps approach, one that made the entire process more automated, collaborative, efficient and scalable.
A central problem with the homegrown cottage-industry approach is that when people have a great idea and want to innovate with their data, or scale up their activities, they can struggle, because that would impact on the value they are currently delivering in a production environment. This risk-averse way of working stifles creativity and innovation. By contrast, DataOps.live opens up a whole new world of opportunity for developers: you can create all these different branches, create and test new stuff, knowing you’re in control of the environment. You can feel safe innovating on a platform like this.
While there’s obviously a great deal of crossover, data engineers and software engineers do have different mindsets and approaches. For example, software engineers benefit from a fundamental understanding of how computers and operating systems work, and how you interact with them. Data engineers may not necessarily come with the same experiences and tend to have a more analytical approach, focusing on the data and getting to the value fast. They haven’t previously owned the data-to-operations process themselves. So this is where the automation, testing and orchestration in DataOps.live come in. You can develop things, test them, fix them, iterate faster, and deliver to a consistently high quality with a tool focused on data engineers.
What’s so special about DataOps.live is that it enables exponential improvements. I can’t think of any scenario where you wouldn’t want to use it, even smaller businesses if they have ambitions to grow. To put it another way, if you’re using Snowflake and have enough data to warrant a data engineer, you’ll want to look at DataOps.live.
It enables you to focus on the data. You’re not worrying if this change or that one affects production, or what else might happen. The risk is removed. It’s an end-to-end platform, so you can see what comes in one end, and what goes out the other—all fully observable, all fully tested. There are so many utilities and helpers baked-in, removing manual steps, that data engineers save significant time and effort. You gain the visibility and control you simply wouldn’t have with a ‘build your own’ mix of separate tools. Yet you still have the freedom and flexibility to incorporate specialized tools should you choose.
In my view, there are four imperatives when it comes to data engineering: to deliver what the customer wants; to automate as much as possible; to test (and test); and have the ability to store configuration as code in source control. DataOps.live enables you to do all this, and far more.
Based in the United Kingdom, Luke has been with DataOps.live since May 2022. Previously a Principal Cloud Engineer with Songtradr, Inc. and Principal Software Engineer at the UK Civil Service, he has a BSc from the University of West England Bristol. Connect with Luke here: https://www.linkedin.com/in/luke-crumley