The Data Warrior

Changing the world, one data model at a time. How can I help you?

Archive for the tag “Data Modeling”

Drill to Detail Podcast: Data Modeling, Data Vault, and Snowflake!

My good friend Mark Rittman has embarked on a new adventure as an independent analyst and consultant. As part of his new venture Mark started a new podcast series on iTunes that he calls Drill to Detail where he will feature interviews discussing a range of topics related to data warehousing, business intelligence, analytics, and big data.

I was honored to be asked to take part in this new venture and got to spend a hour with Mark a few weeks back recording what is now Episode 5 of the series. In this interview we talk about:

The podcast is about 60 minutes with each topic being about 20 minutes (so feel free to skip ahead if you are short on time). Please have a listen and let us know what you think in the comments below.

Cheers!

Kent

The Data Warrior

P.S. I will be speaking on these and related topics at a bunch of events over the next few weeks. Check out my speaking schedule and join me in person if you can!

Advertisements

Maintaining disabled FK’s, wisdom or farce?

A while back, I wrote a post about having FKs (foreign keys) in your data warehouse.

Well, a similar question came up recently on an Oracle forum with the above title. It is a fair question and it does surface fairly regularly in a variety of contexts (not just data warehousing).

Of course, as The Data Warrior, I felt is was my duty to respond.

The Question

Is there any reason to maintain a permanently disabled FK in the data model?  I’m not envisioning a reason to do it.  If it is not going to be enabled, then from my perspective, it would not make any sense to have it defined.  If anything, provide the definition of the relationship in the comment of the child column.

My Answer

Yes, by all means keep the FK please!

I see three good reasons for doing so:

  1. It is valuable metadata (& documentation). If somebody reverse engineers the database (say with ERWin or Oracle Data Modeler), the FK shows up in the diagram (way better than having to read a column comment to find out)
    Data Vault 2.0 Example

    A picture is worth a thousand words!

    .

  2. BI Metadata – If you want to use any sort of reporting or BI tool against the database, most tools will import the FK definition with the tables and build the proper join conditions. Way better than having someone guess what the join will be and then manually adding it to the metadata layer in the reporting tool. Examples that can read the Oracle data dictionary include OBIEE, Business Objects, COGNOS, Looker, and many others.(Note here that since the FK is not enforced on the remote databases, you might want to make sure these are treated as outer joins, lest you lose some transaction in the reports).
  3. The Oracle optimizer will use disabled constraints to improve query performance of joins. Again, this is metadata in the data dictionary which the optimizer can read. This is documented in the Oracle Data Warehouse guide and I have validated it on multiple occasions with Oracle product management.

While #3 applies specifically to Oracle, for other databases like MS SQL Server and Snowflake, #1 and #2 still apply.

Even if only one of the above is true for a given database, that, in my opinion, still justifies keeping the disabled constraint around.

Final Answer = Wisdom

What do you think? Feel free to comment below.

And please share on your favorite social media platform!

Model on!

Kent

The Data Warrior

 

One more time: Do we still need Data Modeling?

More specifically do we still need to worry about data modeling in the NoSQL, Hadoop, Big Data, Data Lake, world?

This keeps coming up. Today it was via email after a presentation I gave last week. This time the query was about the place of data modeling tools in this new world order.

Bottom line: YES, YES, YES! We still need to do data modeling and therefore need good data modeling tools and skills.

Snowflake with RI

A picture can say so much!

 

In order to get any business value out of the data, regardless of where or how it is stored, you have to understand the data, right?

That means you have to understand the model of the data. Even if the model (or schema) is not needed upfront to store the data (schema-on-write), you must discern the model in order to use it (schema-on-read).

It is (mostly) impossible to get repeatable, auditable metrics, KPIs, dashboard, or reports that bring value to the business without understanding the semantics of the data – which means you at least need a conceptual or logical model.

And if you want/need to join data from multiple source then you really have to understand each source or there is no way to properly join it all together to get meaningful results.

There are a few data cleansing, discovery,and “virtualization” tools out there that will help you figure out those relationships but they are expensive and mostly rely on standard data profiling techniques to find similar data objects across the sets and propose “relationships”. Some allow for the definition of fairly sophisticated matching rules including customizations. But a human still needs to figures those out, test, and validate the results.

In the end you still have to know your data.

One of the best ways to do that, in my opinion, is to model that data. Otherwise your data lake will likely become a data swamp!

So keep your data modeling tool and keep building your data dictionary with your business folks.

Final Stage Table

A good modeling tool can act as a visual data dictionary too!

If you agree with me, please share on social media!

#LoveYourData

Kent

The Data Warrior

P.S. If you need a good modeling tool, check out Oracle SQL Developer Data Modeler. And check out my books and training offering for SDDM on the blog sidebar.

Where is the Data Warrior Now?

Hi folks. Time to update y’all on some upcoming speaking engagements for this summer and fall.

Here are a few talks I have scheduled:

Houston DAMA 

When: August 9, 2016 1:30 PM

WhereBP Westlake Campus, Helios Plaza, Houston, TX

Topic 1: Harnessing the Elasticity of the Cloud for Analytics with Snowflake and Tableau

Topic 2: Agile Data Warehousing – Building a Virtualized ODS

Register: Houston DAMA Meetup

Agile Denver: BI Meetup #1 

When: August 17, 2016 5:30 PM

Where: Charles Schwab, Lone Tree, CO (Denver Tech Center)

Topic: Agile Data Warehousing – Building a Virtualized ODS

RegisterAgile BI Meetup

Agile Denver: BI Meetup #2

When: August 18, 2016 5:30 PM

Where: WebPT, Westminster, CO

Topic: Agile Data Warehousing – Building a Virtualized ODS

RegisterAgile BI Meetup

Enterprise Dataversity

EDV2016-ImSpeakingWhen: September 21, 2016 8:30 AM

Where: The Drake Hotel, Chicago, Ill

Topic: Data Warehousing in 2016 and Beyond

Register: Enterprise Dataversity

 

Utah Oracle Users Group (UTOUG) Fall Symposium

When: October 26, 2016

WhereSalt Lake Community College Miller Campus, SLC, Utah

Session #1Agile Data Warehousing: Using Oracle Data Modeler (SDDM) to build a Virtualized ODS

Session #2: Agile Methods and Data Warehousing: How to Deliver Faster

Session #3: Data Warehousing in the Real World

Register: UTOUG Registration – Coming Soon!

 

Southern Fried Agile

When: October 28, 2016

Where: Charlotte Convention Center, NC

Topic:  Agile Methods and Data Warehousing: How to Deliver Faster

Register: SFA Registration

 

East Coast Oracle (ECO) Users Conference

When: November 2 & 3, 2016

Where: Hilton North Raleigh/Midtown Hotel, Raleigh, NC

Topic #1Agile Data Warehousing: Using Oracle Data Modeler (SDDM) to build a Virtualized ODS

Topic #2: Data Warehousing in 2016

Register: ECO Registration

 

There are more talks in the planning process so be sure to check back.

Hope to see you at one of these events!

Love your data!

Kent

The Data Warrior

Data Warrior LLC

It WAS the #Best #DataVault Event Ever!

Last week I had the pleasure of spending a few days in lovely Stowe, Vermont at the Stoweflake Mountain Resort and Spa attending the 3rd Annual World Wide Data Vault Consortium (#WWDVC). Not only was the location picturesque, the weather was near perfect, the beer was tasty, and the learning and networking were outstanding.

We had 75 attendees coming from all over the world – Germany, Switzerland, Canada, England, Australia, New Zealand, The Netherlands, USA, Finland, and India. Quite a turnout!

20160523_190630.jpg

20160523_190617.jpg

Day 1- Data Vault Brainstorming

This year I arrived early enough to participate in what is arguably the best part of the event – a full day, open forum discussion with certified Data Vault modelers and practitioners, lead by the inventor of Data Vault, Dan Linstedt.

The brain power in the room was stunning. There were about 30 people in all and we all got to introduce ourselves and talk a bit about what we had been doing with Data Vault. It was great to hear the many and varied ways in which Data Vault is being used across multiple industries (including a US intelligence agency – but that is a secret). Everything from traditional data warehousing and BI, to realtime streaming IoT data, to virtual Data Vaults and virtualized information marts, to using Data Vault to help with Master Data Management (MDM). It was eye opening and exciting to hear all these applications and opportunities.

If you are not yet certified, get certified! Then you can attend this session at WWDVC 2017 (spoiler – at Stoweflake again!). And you are in luck as Dan just announced three classes later this year in St Albans, Vermont. Plus there are multiple classes coming up in Europe as well.

Day 2- Hands on Workshops

Another unprecedented day at WWDVC.  The three platinum sponsors, AnalytixDS, Talend, and Varigence, all ran 3-hour hands on workshops. These were a fantastic opportunity to see how these vendors have really stepped up to the plate to support quickly building Data Vault solutions with their tools.

These were great sessions, led but highly qualified folks. They showcased some great solutions and answered a lot of questions.

All three sessions were standing room only – with over 35 attendees. (We had to drag in chairs from other rooms!)

Be sure to make time to attend these next year as I am sure they will be on the agenda again.

Day 3 – The Main Event Begins

Yes, all the way to Day 3 before the official kickoff with keynote and speakers.

Dan of course got us started with welcome, thanks to all the sponsors, and housekeeping. Nicely this event only has one room and one track so no one has to pick between sessions!

20160526_080206.jpg

Keynote

The keynote this year was Swimming in the Data Lake by none other than the Father of Data Warehousing, Bill Inmon. I greatly enjoyed his somewhat irreverent look at our industry and his discussion on Big Data and the Data Lake concepts. It was quite a humorous talk (“I don’t mean to offend anybody, but….”). I would say it is one of the best talks I have ever heard Bill give over my 20+ years of knowing him (so I have heard a few).

WWDVC_BillInmon_DataLake

And being a prolific author, Bill of course has a new book out on Data Lakes (available now on Amazon here).

WWDVC_BillInmon_DataLakeBook

Being good geeks, several of us did manage to get our picture taken with Mr. Inmon as “social evidence” that we know him (well, I actually did co-author a book with him back in the day).

Kent with Bill 2016

Lots of Talks

Yes it was a full day with tons of stuff to fill our heads with ideas: new, useful and occasionally controversial. (stay tuned for videos on all these!)

Dan’s business partner, Sanjay Pande, came all the way from India to talk about Data Vault 2.0 on Hadoop. Roelant Vos came again from Australia to give us a business based view of a data vault project at his company (Allianz) about Customer Centric Analytics. Mary Mink and Sam Bendayan of Ultimate Software came for the 2nd year to talk about how their SaaS company is using Data Vault to provide customer value. This time they talked about their efforts to move to virtual information marts (very cool).

I did my presentation on Building a Virtualized ODS. This was a real life example from my consulting last year on doing an agile data warehouse project based on Data Vault architectural principles. It was a fun talk with lots of interaction. I love challenging the norm, then proving it works!

KentWWDVC16_VODS

Of course I did have to do a little intro promo about my employer, Snowflake Computing. I am happy to say there was quite a bit of interest in our cloud-native, elastic data warehouse offering.

KentWWDVC16_Snowflake

After my talk I did a drawing for a GoPro camera (courtesy of Snowflake). I am happy to say it went Russell Searle from Australia! This man loves Data Vault so much he has paid his own way to Vermont twice now to attend WWDVC. Now that is dedication!

KentWWDVC16_GoPro

Days 4 & 5

Sadly I had other commitments back in Texas and could not stay for these days (but did follow along a bit on twitter). If you want to see everything that happened, search Twitter for #WWDVC.

One fun thing on Day 4 was a few people got to go up in a tethered hot air balloon. Hopefully I can try that next year.

WWDVC_StoweflakeBalloon

Other Fun Stuff

Of course not everything happens in the sessions. Lots of good networking and information exchange happens informally at these events. I did several impromptu demonstrations of Snowflake. The German and Australian contingents were quite interested and can’t wait until Snowflake is available in their regions.

20160525_143828.jpg

Thanks to my friends Paul and Raphael at WhereScape for loaning me their big monitor!

I introduced a very international crowd to the best northern-style, southern BBQ at the Sunset Grille. We had good Data Vault, and non-DV, conversations along with finger licking ribs, brisket, and pulled pork (and beer of course).

20160524_193334.jpg

Take Aways

Every year, as he closes out the event, Dan tries to summarize key learnings for everyone to take home. Here they are for WWDVC 2016:

WWDVC_conclusions

Well that is it for this time around. With such a great event it is impossible to adequately cover everything but I hope this is enough to get you to put WWDVC 2017 on your event calendar. Ask for the time off now!

Safe travels to all the attendees. See you again soon.

Kent

The Data Warrior

Post Navigation

%d bloggers like this: