Skip to content
DataOps.live Professional EditionNEW
Purpose-built environment for small data teams and dbt Core developers.
DataOps.live Enterprise Edition
DataOps.live is the leading provider of Snowflake environment management, end-to-end orchestration, CI/CD, automated testing & observability, and code management, wrapped in an elegant developer interface.
Spendview for Snowflake FREE

An inexpensive, quick and easy way to build beautiful responsive website pages without coding knowledge.


Pricing and Edition

See whats included in our Professional and Enterprise Editions.

Getting Started
Docs- New to DataOps.liveStart learning by doing. Create your first project and set up your DataOps execution environment.
Join the Community
Join the CommunityFind answers to your DataOps questions, collaborate with your peers, share your knowledge!
#TrueDataOps Podcast
#TrueDataOps PodcastWelcome to the #TrueDataOps podcast with your host Kent Graziano, The Data Warrior!
Resource Hub
On-demand resources: eBooks, white papers, videos, webinars.

Customer Stories
Academy

Enroll in the DataOps.live Academy to take advantage of training courses. These courses will help you make the most out of DataOps.live.


Learning Resources
A collection of resources to support your learning journey.
Events
Connect with fellow professionals, expand your network, and gain knowledge from our esteemed product and industry experts.
Blogs

Stay updated with the latest insights and news from our DataOps team and community.


#TrueDataOps.org
#TrueDataOps is defined by seven key characteristics or pillars:
In The News

In The News

Stay up-to-date with the latest developments, press releases, and news.
About Us
About UsFounded in 2020 with a vision to enhance customer insights and value, our company has since developed technologies focused on DataOps.
Careers

Careers

Join the DataOps.live team today! We're looking for colleagues on our Sales, Marketing, Engineering, Product, and Support teams.
DataOps.liveMay 30, 2020 9:44:00 AM2 min read

5 Reasons APIs Don’t Work for Data Sharing

And 5 Reasons Snowflake’s Data Sharehouse Does

For the past decade the technology world has been obsessed with APIs. While the nature of the APIs themselves has changed over the years (e.g. from SOAP to REST), the premise has always been the sameAPIs are the way to allow different systems to share information. This doctrine is so well accepted, no one questions it. Certainly, for transactional processes, such as creating a user account, APIs work fine.

But what about a Service Provider wanting to share usage data with an Enterprise customer? Let’s say they want to share performance and operational time series data for a global network, which could easily run to many Terabytes.

It’s straight forward to provide an API to query usage data for a certain historical period. However standard BI tools don’t talk to APIs directly. In this case, the Enterprise would use the APIs to query the usage data from the Service Provider and then store it locally, typically in a SQL database. In this case APIs are used as an interface for creating a copy of the data. But there are problems with this model of data sharing.

Why APIs won’t work for data sharing

1. The data is being stored twice (probably more than twice in most cases)incurring additional cost.

2. Since APIs usually cannot handle massive quantities of data in single queries (and should have abuse protection built in to prevent people from trying), in order to pull a complete history will require scripting to import data in small chunks at a time creating a big processing overhead on extraction.

3. When new data is added, the Enterprise needs to poll for new data (which introduces delay) or use a streaming API like a WebSockets which requires a lot of sophisticated development to manage reconnections, collecting missing data, etc.

4. If the Service Provider needs to make a change or fix an error in historical data, it’s very hard for the Enterprise to discover this and make the associated updates.

5. There are enormous issues associated with trying to keep copies of a data set in sync and correct. This is especially problematic when monitoring SLAs or other contentious topicsif the Service Provider and the Enterprise have differences in their data, both will be adamant that they are rightthere is no single source of truth.

Snowflake Data Sharing eliminates ALL these issues. The Service Provider can provide a Secure View onto the data for the Enterprise who can then use this data in a form that is ready for consumption by BI tools. There are some clear benefits and advantages.

Why Snowflake’s Data Sharehouse works

1. Data is stored once and only once

2. There is no effort or processing required to enable access to a long history of data

3. There is no effort or processing required to get the most recent data as it is written

4. Any bulk updates to historical data are seen by the Enterprise instantly

5. The data as seen by both the Service Provider and the Enterprise is guaranteed to be identicalsince there is only one physical copy of the data in existence, there is now a single source of truth

APIs can now be relieved of the burden of analytics and data extraction and be used for what they are good attransactional processing. And data sharing can now live up to the promises being made with the use of Snowflake’s unique data sharehouse capability.

 

 

RELATED ARTICLES