The Data Warrior

Changing the world, one data model at a time. How can I help you?

Snowflake and Spark, Part 2: Pushing Spark Query Processing to Snowflake

Here is the latest post on using Spark and the Snowflake cloud-native data warehouse.

Welcome to the second post in our ongoing blog series describing Snowflake’s integration with Spark. In Part 1, we discussed the value of using Spark and Snowflake together to power an integrated data processing platform, with a particular focus on ETL scenarios.

In this post, we change perspective and focus on performing some of the more resource-intensive processing in Snowflake instead of Spark, which results in significant performance improvements. As part of this, we walk you through the details of Snowflake’s ability to push query processing down from Spark into Snowflake. We also touch on how this pushdown can help you transition from a traditional ETL process to a more flexible and powerful ELT model.

Read the rest: Snowflake and Spark, Part 2: Pushing Spark Query Processing to Snowflake

Enjoy!

Kent

The Data Warrior

Advertisements

Single Post Navigation

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: