The Data Warrior

Changing the world, one data model at a time. How can I help you?

Archive for the tag “data vault 2.0”

Data Vault 2.0 Automation with erwin and Snowflake

I am seeing a HUGE uptick in interest in Data Vault around the globe. Part of the interest is the need for agility in building a modern data platform. One of the benefits of the Data Vault 2.0 method is the repeatable patterns which lend themselves to automation.  I am please to pass on this great new post with details on how to automate building your Data Vault 2.0 architecture on Snowflake using erwin! Thanks to my buddy John Carter at erwin for taking this project on.

The Data Vault methodology can be applied to almost any data store and populated by almost any ETL or ELT data integration tool. As Snowflake Chief Technical Evangelist Kent Graziano mentions in one of his many blog posts, “DV (Data Vault) was developed specifically to address agility, flexibility, and scalability issues found in the other mainstream data modeling approaches used in the data warehousing space.” In other words, it enables you to build a scalable data warehouse that can incorporate disparate data sources over time. Traditional data warehousing typically requires refactoring to integrate new sources, but when implemented correctly, Data Vault 2.0 requires no refactoring.

Successfully implementing a Data Vault solution requires skilled resources and traditionally entails a lot of manual effort to define the Data Vault pipeline and create ETL (or ELT) code from scratch. The entire process can take months or even years, and it is often riddled with errors, slowing down the data pipeline. Automating design changes and the code to process data movement ensures organizations can accelerate development and deployment in a timely and cost-effective manner, speeding the time to value of the data.

Snowflake’s Data Cloud contains all the necessary components for building, populating, and managing Data Vault 2.0 solutions. erwin’s toolset models, maps, and automates the creation, population, and maintenance of Data Vault solutions on Snowflake. The combination of Snowflake and erwin provides an end-to-end solution for a governed Data Vault with powerful performance.

Get the rest of the details here: Data Vault Automation with erwin and Snowflake

Vault away my friends!

Kent

The Data Warrior

Tips for Optimizing the #DataVault Architecture on Snowflake (Part 3) 

For this last post in my current Data Vault (DV) series, I will discuss two more cool features of Snowflake Cloud Data Platform (@Snowflakedb) that you can take advantage of when building a DV on our platform. If you are not familiar with the DV method, please read my introductory blog post and part 1 of this series before reading this post.

Get the details here: Tips for Optimizing the Data Vault Architecture on Snowflake (Part 3)

Happy Data Vaulting!

Kent

The Data Warrior

Better Data Modeling: Agile Data Engineering

You asked for it, you got it!

Ever since I wrote my Kindle book on Agile Data Engineering and Data Vault 2.0, many, many people have asked me to provide it in a hardcopy format. Well, I finally managed to find time to convert that ebook into a paperback book (I even corrected a few errors in the process).

If you forgot what the book was about, here is the description:

This book will give you a short introduction to Agile Data Engineering for Data Warehousing and Data Vault 2.0. I will explain why you should be trying to become Agile, some of the history and rationale for Data Vault 2.0, and then show you the basics for how to build a data warehouse model using the Data Vault 2.0 standards.In addition, I will cover some details about the Business Data Vault (what it is) and then how to build a virtual Information Mart off your Data Vault and Business Vault using the Data Vault 2.0 architecture.So if you want to start learning about Agile Data Engineering with Data Vault 2.0, this book is for you.

So here it is – Introduction to Agile Data Engineering – now available to purchase on Amazon.

Get your copy now. Next time you see me at an event, I will be happy to sign it for you. 🙂

Enjoy!

Kent

The Data Warrior

Get Certified! #DataVault 2.0 Certification in the US

Quick update – if you have been waiting to get your Data Vault 2.0 certification there are three sessions coming in the new few months right in the USA.  If you already know you want to do that, just skip down to the links and sign up!

Why Data Vault?

The Data Vault 2.0 architecture gives you an entire systems based approach to developing a true enterprise data warehouse and analytics architecture. It is very structured, pattern based, and highly repeatable. In Data Vault, each component does it’s duty, and does it well. The engineering components are generally relegated to automation tools (because it is pattern based), so human effort is not wasted in the mundane and can be used in more interesting, intelligent and thinking tasks. It’s a much better use of intelligent beings as well as machines.

Separating the concerns makes design and development not just easy, but fast.

As a side effect projects using Data Vault 2.0 have always saved a lot of money and have been extremely successful with their predictable goals. Plus they are very resilient so they tend to stay in use for years to come with little or no re-engineering! One of my systems has been running for 14 years now – and was even successfully re-platformed in that time.

How do you get in on this innovative approach?

If you want to learn more (and why wouldn’t you?), there are many upcoming opportunities across the world to get more information about Data Vault 2.0 (just check Twitter or LinkedIn – look for #DataVault). If you understand it, and you want to use it to leverage your own successes, you can even get certified (That comes with a responsibility though).

Here’s a list of upcoming opportunities to get DV 2.0 certified in the US:

1. Sep 19-21, Chicago, IL – http://www.performanceg2.com/agile-bi-datavault-training/

2. Oct 2-4, New York City, NY – http://www.scalefree.com/2017/03/30/data-vault-2-0-boot-camp-and-certification-new-york-oct-2017/

3. Nov 27-29, Santa Clara, CA – http://www.scalefree.com/2017/03/29/data-vault-2-0-boot-camp-and-certification-santaclara-nov-2017/

Ready to challenge the status quo and become a data champion at your organization? Then sign up for one of these classes today!

Model on!

Kent

The Data Warrior

Snowflake at Stoweflake

20170519_073717

Every year the World Wide Data Vault Consortium (WWDVC) gets better and better! This year’s event was the 4th Annual and was again held at the lovely Stoweflake Mountain Lodge in Stowe, Vermont.

WWDVC_StoweflakeBalloon

And once again this year, my employer, Snowflake Computing, was a proud sponsor of the event. This year I even got to help with a hands on workshop with our ELT partner Talend as we walked folks through building a Data Vault in Snowflake using Talend!

WWDVC Sponsors

100 attendees got their minds filled and horizons broadened by an amazing slate of presentations given by great speakers from all over the world. Not only did we hear some real-life case studies from companies like Micron and Intact Financials (who have VERY large data vaults) but we even got to hear from someone at the US DoD (yes the Department of Defense!).

Then there were these mind-bending talks that challenged the most experienced in the audience:

– Measuring Data as an Asset (by Nols Ebersohn)
– How to get a DV project Approved (by Neil Strange)
– Uncertainty, Risk, & The Value of Information (by Brad Bergh)

And there were of course absolutely awesome keynotes from Tamara Dull (A Big Data Cheat Sheet for the Technically Savvy Data Professional) and from Scott Ambler we heard (very clearly!): Are You Agile or Are You Fragile?

What? You missed WWDVC 2017?

Well I guess you will have to wait for the 5th Annual WWDVC in 2018…

WWDVC Sunset

Can’t wait? You are in luck!

This year Dan hired professional videographers to record the entire event.

Yup, all the workshops, the keynotes, and all the presentations.

I have seen the videos and they came out great.

So, if you would like to join the elite group of 100 data vault aficionados that attended WWDVC17, you now have the chance to see and hear the same great content we all were exposed to. Then you can be the champion for brining Data Vault 2.0 to your organization.

You can purchase access to all the recordings right here.

NB: There are no refunds on this purchase out of consideration for those who spent the time and money to attend the event.

Here’s what you get:

Pre-Conference Sessions

  • Brainstorming with Dan Linstedt (Inventor of the Data Vault and DV 2.0), Michael Olschimke (Co-Author, Building a Scalable Data Warehouse with DV 2.0) and Sanjay Pande (co-founder, LearnDataVault)
  • Talend and Snowflake Hand’s On Session
  • WhereScape Hands On Session
  • Analytix DS Hands On Workshop

Conference Sessions

Day 1

  • Keynote I – A Big Data Cheat Sheet for the Technically Savvy Data Professional (Tamara Dull, Director of Emerging Technologies, SAS)
  • Implementing a Data Vault 2.0 in the DoD (Cynthia Meyersohn, Senior Technical Consultant, Quadrint)
  • Data Vault 2.0 and the Power of Metadata (Steven Mellare, Data and Information Architect and Strategist, Pepper Money)
  • Software Defined Data Warehouse Using Data Vault 2.0 (Tevje Olin, Data Architect and Consultant, Solita)
  • Big Data Vault at Micron (Mike Magalsky, Enterprise Data Architect, Micron and Chris Sundstrom, Principal Data Architect, IM Flash Technologies)
  • Talend in the world of Data Vault (Dale Anderson, Customer Success Architect, Talend)
  • Analytix DS (Sam Benedict, VP Strategic Accounts, Analytix DS)
  • Data Mining in the Data Vault (Michael Olschimke, CEO, ScaleFree)

Day 2

  • Keynote II – Are You Agile or Are You Fragile? (Scott Ambler, Senior Consulting Partner, Scott Ambler + Associates)
  • Agile Methods and Data Warehousing: How to Deliver Faster (Kent Graziano, Senior Technical Evangelist, Snowflake Computing)
  • No DV is an Island: What Lies Beyond (Nols Ebersohn, Principal Architect, Certus Solutions Limited)
  • Beyond a Hadoop DV 2.0 Data Warehouse (Sanjay Pande, Co-Founder, LearnDataVault.com)
  • Business Vault Creation using a Rules Engine (Bruce McCartney, Senior Information Architect, First4 Database Partners)
  • Getting a Data Vault Project Approved (Neil Strange, Founder and MD, Business Thinking)
  • A Data Modeler and Process Modeler Walk into a Data Vault (John Giles, Independent Consultant and author of “The Nimble Elephant”)
  • WhereScape Automation Enabling Data Vault 2.0 (Neil Barton, CTO, WhereScape and Paul Watson Gover, Senior Solution Architect, WhereScape)

Day 3

  • Data Vault Automation – An OnGoing Story at Intact Financial (Francois Trudeau, Application Architect for Enterprise Information Systems, Intact Financial)
  • Moving to the Cloud, Metadata Driven Automation at Yale (Robert Scott, CTO, EON Collective)
  • Uncertainty, Risk and the Value of Information (Brad Bergh, Enterprise Information Consultant)

The only thing you miss out on is the great food and of course the in-person networking. So put WWDVC18 on your calendar (May 2018) but in the meantime get started by purchasing the videos from WWDVC17 now.

wwdvc2017

Hopefully seeing these talk may even inspire you to not only attend next year but maybe even speak yourself!

Enjoy!

Kent

The Data Warrior & Data Vault Master

P.S. Of course if you have any questions or want to learn more about Snowflake, the 1st cloud-native data warehouse as a service, please reach out to me or follow me on twitter @kentgraziano.

Post Navigation

%d bloggers like this: