Snowflake and Spark, Part 2: Pushing Spark Query Processing to Snowflake
Here is the latest post on using Spark and the Snowflake cloud-native data warehouse.
Welcome to the second post in our ongoing blog series describing Snowflake’s integration with Spark. In Part 1, we discussed the value of using Spark and Snowflake together to power an integrated data processing platform, with a particular focus on ETL scenarios.
In this post, we change perspective and focus on performing some of the more resource-intensive processing in Snowflake instead of Spark, which results in significant performance improvements. As part of this, we walk you through the details of Snowflake’s ability to push query processing down from Spark into Snowflake. We also touch on how this pushdown can help you transition from a traditional ETL process to a more flexible and powerful ELT model.
Read the rest: Snowflake and Spark, Part 2: Pushing Spark Query Processing to Snowflake
Enjoy!
Kent
The Data Warrior
The Data Warrior 