DataOps develops open source webhooks for Snowflake

LONDON, United Kingdom, October 17th 2019—Over the last 2 years, DataOps has been developing Gallium (IoT Smart Data Compression Algorithm) technology, now launched as part of a new era of data services on Snowflake.

To deliver this, the DataOps team had to find new ways of triggering external data services from within Snowflake based on table updates without building external polling solutions.

DataOps has solved the challenge of triggering actions and processes in external systems from within Snowflake and has made the solution available to all Snowflake customers for free.

The Github repository for DataOps’ Snowflake webhooks can be found here.

By implementing webhooks, Snowflake users can now take advantage of powerful bidirectional integration from Snowflake to external systems like Talend and Databricks.

Benoit Dageville and Thierry Cruanes of Snowflake expressed their enthusiasm for this solution.

“Thierry and I have been working closely with the DataOps team for the last year. DataOps are one of the most experienced teams around the deep insides of Snowflake and we have been enjoying working with them. DataOps work closely with Snowflake as part of their data platform, but are also specialists in IoT timeseries data. They have produced Gallium which is a unique time series compression algorithm with limited loss of fidelity. Whether its 5:1 or 500:1 compression, the business value to companies using Snowflake and Gallium for IoT timeseries data compression will be significant.

As part of the process to integrate Gallium directly into Snowflake, DataOps have developed an excellent opensource approach to launching external data services, like Gallium, from within Snowflake using only the Snowflake capabilities available today.

This is an example of great solutions coming from the Snowflake ecosystem to help Snowflake users. It’s great to see innovative solutions from leading partners like DataOps. This Gallium integration is the first of a new age of data services called natively from within Snowflake, and with the generous open sourcing of this code, we look forward to seeing many more.”

The use cases for webhooks in Snowflake vary from triggering simple Email or SMS alerts based on events in Snowflake to complex integration and orchestration with external systems.

For example, a process in Talend can trigger long running jobs inside Snowflake without having to maintain a session or having to poll to find out if jobs have completed. When the job in Snowflake has finished, a webhook call can be made back into Talend to start the next step in the process.

Databricks users can also benefit from using webhooks in Snowflake. Many Databricks customers hold data inside Snowflake for training models. When a critical mass of new data has been added into Snowflake, a retraining of a model can be triggered within Databricks.

As a partner and service provider of Snowflake, Talend, Databricks, Azure and AWS, DataOps saw the need for a universally supported and well accepted method for triggering events in web-based systems from inside Snowflake. So they developed these webhooks and made them available on Github to help support the Snowflake customer community as well as its partner communities.

In the coming weeks, DataOps will be publishing more blogs detailing how they used these webhooks with Table Streams, how to use them for Data Services and more.