The Data Warrior

Changing the world, one data model at a time. How can I help you?

Archive for the tag “data warehouse design”

19 Apr 2021

Building a Real-time Data Vault in Snowflake?

Yes you can! The #DataCloud loves #DataVault!

In this day and age, with the ever-increasing availability and volume of data from many types of sources such as IoT, mobile devices, and weblogs, there is a growing need, and yes, demand, to go from batch load processes to streaming or “real-time” (RT) loading of data. Businesses are changing at an alarming rate and are becoming more competitive all the time. Those that can harness the value of their data faster to drive better business outcomes will be the ones to prevail.

One of the benefits of using the Data Vault 2.0 architecture is that it was designed from inception not only to accept data loaded using traditional batch mode (which was the prevailing mode in the early 2000s when Dan introduced Data Vault) but also to easily accept data loading in real or near-realtime (NRT). In the early 2000s, that was a nice-to-have aspect of the approach and meant the methodology was effectively future-proofed from that perspective. Still, few database systems had the capacity to support that kind of requirement. Today, RT or at least NRT loading is almost becoming a mandatory requirement for modern data platforms. Granted, not all loads or use cases need to be NRT, but most forward-thinking organizations need to onboard data for analytics in an NRT manner.

See all the details (and some code) in the full post over on Data Vault Alliance.

Happy Vaulting!

Kent

The Data Warrior

Posted in Data Vault, SnowflakeDB and tagged , , , , , , , , , , | Leave a comment
02 Sep 2017

Get Certified! #DataVault 2.0 Certification in the US

Quick update – if you have been waiting to get your Data Vault 2.0 certification there are three sessions coming in the new few months right in the USA.  If you already know you want to do that, just skip down to the links and sign up!

Why Data Vault?

The Data Vault 2.0 architecture gives you an entire systems based approach to developing a true enterprise data warehouse and analytics architecture. It is very structured, pattern based, and highly repeatable. In Data Vault, each component does it’s duty, and does it well. The engineering components are generally relegated to automation tools (because it is pattern based), so human effort is not wasted in the mundane and can be used in more interesting, intelligent and thinking tasks. It’s a much better use of intelligent beings as well as machines.

Separating the concerns makes design and development not just easy, but fast.

As a side effect projects using Data Vault 2.0 have always saved a lot of money and have been extremely successful with their predictable goals. Plus they are very resilient so they tend to stay in use for years to come with little or no re-engineering! One of my systems has been running for 14 years now – and was even successfully re-platformed in that time.

How do you get in on this innovative approach?

If you want to learn more (and why wouldn’t you?), there are many upcoming opportunities across the world to get more information about Data Vault 2.0 (just check Twitter or LinkedIn – look for #DataVault). If you understand it, and you want to use it to leverage your own successes, you can even get certified (That comes with a responsibility though).

Here’s a list of upcoming opportunities to get DV 2.0 certified in the US:

1. Sep 19-21, Chicago, IL – http://www.performanceg2.com/agile-bi-datavault-training/

2. Oct 2-4, New York City, NY – http://www.scalefree.com/2017/03/30/data-vault-2-0-boot-camp-and-certification-new-york-oct-2017/

3. Nov 27-29, Santa Clara, CA – http://www.scalefree.com/2017/03/29/data-vault-2-0-boot-camp-and-certification-santaclara-nov-2017/

Ready to challenge the status quo and become a data champion at your organization? Then sign up for one of these classes today!

Model on!

Kent

The Data Warrior

Posted in Data Vault, Training and tagged , , , , , , , , , , , | Leave a comment
20 Apr 2017

Meet me in St. Louie, Louie.

Next up on the Data Warrior speaking schedule is the St. Louis SilverLinings event on May 2nd. It will be held at the St. Charles Convention Center, St. Louis, MS.

SilverLinings

This promises to be a very exciting event boasting “edgy” and forward looking technical topics. It’s going to be a very busy day for me with three talks in total on some of my favorite topics.

Topic 1: Demystifying Data Warehousing as a Service: Top 10 Cool Features in Snowflake

Topic 2: Agile Methods and Data Warehousing: How to Deliver Faster

Topic 3: Agile Data Engineering: Introduction to Data Vault Data Modeling

So if you are in the St. Louis area, or fancy a trip to the Gateway to the West,  please join me there on May 2nd.

Special Discount for Data Warrior fans!

The organizers were kind enough to offer my followers a 50% discount. Wow!

Just use this code when you sign up: KGraz280790

So what are you waiting for – sign up register here.

See you soon!

Kent

The Data Warrior

Posted in Data Modeling, Data Vault, Data Warehouse, SnowflakeDB and tagged , , , , , , , , , , , , , , | Leave a comment
26 Jan 2017
1 Comment

RI (Referential Integrity) Constraints: 3 Reasons to Include Them in Your Data Warehouse

Over the years, I have had numerous conversations about the value of having referential integrity (RI) constraints, such as primary and foreign keys, in a relational data warehouse or data mart.

Many DBAs object that RI constraints slow the load process. This is a valid point if you are talking about enforced constraints that are checked in real-time during the load. But this is not an issue if you define the constraints as disabled.

Which then leads to this common question:

Is there any reason to maintain a permanently disabled FK in the data model?  If it is not going to be enabled, then from my perspective, it doesn’t make any sense to define the FK.  Instead, the relationship can be described in the comment of the child column.

So, why would I want RI constraints in my data warehouse?

Here are 3 reasons to consider…RI (Referential Integrity) Constraints: 3 Reasons to Include Them in Your Data Warehouse

Model on!

Kent

The Data Warrior

Posted in Data Modeling, Data Warehouse, SnowflakeDB and tagged , , , ,
20 Jul 2016

Where is the Data Warrior Now?

Hi folks. Time to update y’all on some upcoming speaking engagements for this summer and fall.

Here are a few talks I have scheduled:

Houston DAMA 

When: August 9, 2016 1:30 PM

Where: BP Westlake Campus, Helios Plaza, Houston, TX

Topic 1: Harnessing the Elasticity of the Cloud for Analytics with Snowflake and Tableau

Topic 2: Agile Data Warehousing – Building a Virtualized ODS

Register: Houston DAMA Meetup

Agile Denver: BI Meetup #1 

When: August 17, 2016 5:30 PM

Where: Charles Schwab, Lone Tree, CO (Denver Tech Center)

Topic: Agile Data Warehousing – Building a Virtualized ODS

Register: Agile BI Meetup

Agile Denver: BI Meetup #2

When: August 18, 2016 5:30 PM

Where: WebPT, Westminster, CO

Topic: Agile Data Warehousing – Building a Virtualized ODS

Register: Agile BI Meetup

Enterprise Dataversity

EDV2016-ImSpeakingWhen: September 21, 2016 8:30 AM

Where: The Drake Hotel, Chicago, Ill

Topic: Data Warehousing in 2016 and Beyond

Register: Enterprise Dataversity

 

Utah Oracle Users Group (UTOUG) Fall Symposium

When: October 26, 2016

Where: Salt Lake Community College Miller Campus, SLC, Utah

Session #1: Agile Data Warehousing: Using Oracle Data Modeler (SDDM) to build a Virtualized ODS

Session #2: Agile Methods and Data Warehousing: How to Deliver Faster

Session #3: Data Warehousing in the Real World

Register: UTOUG Registration – Coming Soon!

 

Southern Fried Agile

When: October 28, 2016

Where: Charlotte Convention Center, NC

Topic:  Agile Methods and Data Warehousing: How to Deliver Faster

Register: SFA Registration

 

East Coast Oracle (ECO) Users Conference

When: November 2 & 3, 2016

Where: Hilton North Raleigh/Midtown Hotel, Raleigh, NC

Topic #1: Agile Data Warehousing: Using Oracle Data Modeler (SDDM) to build a Virtualized ODS

Topic #2: Data Warehousing in 2016

Register: ECO Registration

 

There are more talks in the planning process so be sure to check back.

Hope to see you at one of these events!

Love your data!

Kent

The Data Warrior

Data Warrior LLC

Posted in Data Modeling, Data Warehouse, SnowflakeDB, User Groups and tagged , , , , , , , , , , , , | Leave a comment

Post Navigation

← Older Entries
%d bloggers like this: