The Data Warrior

Changing the world, one data model at a time. How can I help you?

Archive for the tag “Data Vault”

Data Vault 2.0 Automation with erwin and Snowflake

I am seeing a HUGE uptick in interest in Data Vault around the globe. Part of the interest is the need for agility in building a modern data platform. One of the benefits of the Data Vault 2.0 method is the repeatable patterns which lend themselves to automation.  I am please to pass on this great new post with details on how to automate building your Data Vault 2.0 architecture on Snowflake using erwin! Thanks to my buddy John Carter at erwin for taking this project on.

The Data Vault methodology can be applied to almost any data store and populated by almost any ETL or ELT data integration tool. As Snowflake Chief Technical Evangelist Kent Graziano mentions in one of his many blog posts, “DV (Data Vault) was developed specifically to address agility, flexibility, and scalability issues found in the other mainstream data modeling approaches used in the data warehousing space.” In other words, it enables you to build a scalable data warehouse that can incorporate disparate data sources over time. Traditional data warehousing typically requires refactoring to integrate new sources, but when implemented correctly, Data Vault 2.0 requires no refactoring.

Successfully implementing a Data Vault solution requires skilled resources and traditionally entails a lot of manual effort to define the Data Vault pipeline and create ETL (or ELT) code from scratch. The entire process can take months or even years, and it is often riddled with errors, slowing down the data pipeline. Automating design changes and the code to process data movement ensures organizations can accelerate development and deployment in a timely and cost-effective manner, speeding the time to value of the data.

Snowflake’s Data Cloud contains all the necessary components for building, populating, and managing Data Vault 2.0 solutions. erwin’s toolset models, maps, and automates the creation, population, and maintenance of Data Vault solutions on Snowflake. The combination of Snowflake and erwin provides an end-to-end solution for a governed Data Vault with powerful performance.

Get the rest of the details here: Data Vault Automation with erwin and Snowflake

Vault away my friends!

Kent

The Data Warrior

Tips for Optimizing the #DataVault Architecture on Snowflake (Part 3) 

For this last post in my current Data Vault (DV) series, I will discuss two more cool features of Snowflake Cloud Data Platform (@Snowflakedb) that you can take advantage of when building a DV on our platform. If you are not familiar with the DV method, please read my introductory blog post and part 1 of this series before reading this post.

Get the details here: Tips for Optimizing the Data Vault Architecture on Snowflake (Part 3)

Happy Data Vaulting!

Kent

The Data Warrior

Tips for Optimizing the #DataVault Architecture on #Snowflake (Part 2)

SETTING UP FOR MAXIMAL PARALLEL LOADING!

In this post, I discuss how to engineer your Data Vault load in Snowflake Cloud Data Platform for maximum speed.

Because Snowflake separates compute from storage and allows the definition of multiple independent compute clusters, it provides some truly unique opportunities to configure virtual warehouses to support optimal throughput of DV loads.

Along with using larger “T-shirt size” warehouses to increase throughput, using multi-cluster warehouses during data loading increases concurrency for even faster loads at scale.

Get the details –  Tips for Optimizing the Data Vault Architecture on Snowflake (Part 2)

Enjoy!

Kent

The Data Warrior & Chief Technical Evangelist for Snowflake

Tips for Optimizing the #DataVault Architecture on #Snowflake

Data Vault is an architectural approach that includes a specific data model design pattern and methodology developed specifically to support a modern, agile approach to building an enterprise data warehouse and analytics repository.

Typical Data Vault Design with Hubs, Sats, and a Link

Snowflake Cloud Data Platform was built to be design pattern agnostic. That means you can use it with equal efficiency 3NF models, dimensional (star) schemas, DV, or any hybrid you might have.Snowflake supports DV designs and handles several DV design variations very well with excellent performance.

This series of blog posts will present some tips and recommendations that have evolved over the last few years for implementing a DV-style warehouse in Snowflake.

Here is the first set of tips:Tips for Optimizing the Data Vault Architecture on Snowflake (part 1)

I hope you find this helpful!

Kent

The Data Warrior and Chief Technical Evangelist for Snowflake

Better Data Modeling: Agile Data Engineering

You asked for it, you got it!

Ever since I wrote my Kindle book on Agile Data Engineering and Data Vault 2.0, many, many people have asked me to provide it in a hardcopy format. Well, I finally managed to find time to convert that ebook into a paperback book (I even corrected a few errors in the process).

If you forgot what the book was about, here is the description:

This book will give you a short introduction to Agile Data Engineering for Data Warehousing and Data Vault 2.0. I will explain why you should be trying to become Agile, some of the history and rationale for Data Vault 2.0, and then show you the basics for how to build a data warehouse model using the Data Vault 2.0 standards.In addition, I will cover some details about the Business Data Vault (what it is) and then how to build a virtual Information Mart off your Data Vault and Business Vault using the Data Vault 2.0 architecture.So if you want to start learning about Agile Data Engineering with Data Vault 2.0, this book is for you.

So here it is – Introduction to Agile Data Engineering – now available to purchase on Amazon.

Get your copy now. Next time you see me at an event, I will be happy to sign it for you. 🙂

Enjoy!

Kent

The Data Warrior

Post Navigation

%d bloggers like this: