The Data Warrior

Changing the world, one data model at a time. How can I help you?

Archive for the tag “tuning”

28 Mar 2014
15 Comments

Better Data Modeling: My Top 3 Reasons why you should put Foreign Keys in your Data Warehouse

This question came up at the recent World Wide Data Vault Consortium. Seems there are still many folks who build a data warehouse (or data mart) that do not include FKs in the database.

The usual reason is that it “slows down” load performance.

No surprise there. Been hearing that for years.

And I say one of two things:

1. So what! I need my data to be correct and to come out fast too!

or

2. Show me! How slow is it really?

Keep in mind that while getting the data in quickly is important, so is getting the data out.

Who would you rather have complain – the ETL programmer or the business user trying to run a report?

Yes, it has to be a balance, but you should not immediately dismiss including FKs in your warehouse without considering the options and benefits of those options.

So here are my three main reasons why you should include FK constraints in your Oracle data warehouse database:

  1. The Oracle optimizer uses the constraints to make better decisions on join paths.
  2. Your Data Modeling and BI tools can read the FKs from the data dictionary to create correct joins in the meta data of the tool (SDDM, Erwin, OBIEE, Cognos, Bus Objects can all do this).
  3. It is a good QA check on your ETL. (Yeah, I know… the ETL code is perfect and checks all that stuff, bla, bla, bla)

Now of course there are compromise options. The three main ones are I know:

  1. Drop the constraints at the start of the load then add them back in after the load completes. If any fail to build, that tells you immediately where you may have some data quality problems or your model is wrong (or something else changed).
  2. Build all the constraints as DISABLE NOVALIDATE. This puts them in the database for the BI tools and data modeling tools to see and capture but, since they are not enforced, they put minimal overhead on the load process. And, so I am told by those that know, even a disabled constraint helps the optimizer make a smarter choice on the join path.
  3. (really 2a) Best of both – disable the constraints, load your data, then re-enable the constraints. You get optimization and quality checks.

So NOW what is your reason for not using FKs in your data warehouse?

Happy Modeling!

Kent

Posted in Data Modeling, Data Warehouse, Oracle, SQL Developer Data Modeler and tagged , , , , , , , , , , , , , , , , , ,
29 Sep 2013

Let’s Review #OOW13 and #OTW13 in Pictures

Yes I have been derelict in my duty and not posted about the sessions I attended at Oracle OpenWorld (#OOW13) and OakTable World (#OTW).

Well here are the high points with pictures!

Monday

Monday started off with the now annual Swim the Bay (so I missed the keynote). If you have Facebook, you can see pictures from the event here.

Most of the day I then spent at the alternate conference, OakTable World (#OTW13) seeing a few talk and giving one myself.

My good friend from Denver, Tim Gorman gave a nice talk about all the data compression options available in Oracle.

Tim Gorman: Oracle Compression Options

Tim Gorman: Oracle Compression Options

Next was a great session from the well known blogger and author Fabian Pascal. I have been reading his work for years but this was the first time I got to hear him speak in person. As with his writing, the talk was both intellectually stimulating and challenging!

Fabian Pascal: The Last Null

Fabian Pascal: The Last Null

It really is quite a debate in the database world about the meaning and use of NULL in an RDBMS. Fabian has a proposal on how we can (and should) represent data in a way where there will never be NULL attributes.

After a some scheduling issues. later in the day, I did my presentation on using Data Vault Modeling for Agile Data Warehouse Modeling. The room I got had a huge wall for me to project my session on. Definitely the biggest screen ever for one of my talks.

Biggest screen ever for me and my data vault presentation.

Biggest screen ever for me and my data vault presentation.

Tuesday

Started the morning with a few friends doing morning Chi Gung in Union Square, then followed by getting a quick survey of the exhibit hall in Moscone South and a trip to the Demo grounds.

The throng descends into the depths of Moscone West to hunt the exhibit hall for goodies.

The throng descends into the depths of Moscone West to hunt the exhibit hall for goodies.

The hall was of course HUGE as usual so some of the vendors who were tucked in back got creative on getting the foot traffic to come their way.

A clever gimmick one vendor did to get traffic to their booth in the gigantic hall

A clever gimmick one vendor did to get traffic to their booth in the gigantic hall

For sessions, I attend a road map session on Oracle’s Big Data strategy given by my friend JP Dijcks.

JP talks all things Big Data

JP talks all things Big Data

Mostly he painted a picture of the issues with figuring out how to collect and put all that data to real work. Of course Oracle has a ton of products to offer to help solve the problem.

How to shrink the gap between getting big data and actually using it!

How to shrink the gap between getting big data and actually using it!

Next up I attended Jeff Smith’s session on SQL Developer 4.0 and got to learn that there was a data mining extension available for the tool that makes doing some advanced analytics a lot easier.

Definition for Data Mining. An extension for Data Mining is available for SQL Developer.

Definition for Data Mining. An extension for Data Mining is available for SQL Developer.

Next on my agenda was the Cloud keynote with Microsoft. I wrote about that here.

Finally for the day, a late presentation by Maria Colgan and Jonathan Lewis giving us their top tuning tips in what they called the SQL Tuning Bootcamp.

Optimizer tips from a pro Jonathan Lewis. I am sure it means something to someone out there!

Optimizer tips from a pro Jonathan Lewis. I am sure it means something to someone out there!

As always with these type session, there was a ton of useful information that makes my brain hurt. I have to keep reviewing  my notes to make sure I can use at least 10% of what they taught.

Wednesday

This was mostly a work day for me at a client site. And a late lunch to see the final race of the America’s Cup.

In case you have been under a rock since last week, Team USA won! It was great to actually be there on Pier 27 during the final race. Not a great vantage point overall but with the big screen to watch and then seeing the boats right after they finished, it was worth the walk.

After the race and a little more data model work at my client’s office, I walked back to the conference to see a final session (for me) given by Gwen Shapira about using solid state disks with Exadata.

I really did not know much about SSDs before this session but feel really educated now. I actually had no idea that SSD and FLASH drives or FLASH memory were the same thing. Guess I was behind on the hardware buzzwords.

Gwen and Mark on Solid State Disk AKA Flash

Gwen and Mark on Solid State Disk AKA Flash

Then it was off to the annual blogger meetup then dinner on the town with friends at The Stinking Rose (thanks Tim!).

I decided to skip the appreciation event this year and take it easy, have a nice dinner, then pack up to head home. Thursday it was breakfast at Lori’s Diner then off to the airport and back home.

As a reminder if you want to see what the buzz was at the events, just check out the hashtags #OOW13 and #OTW13 on twitter (if you had a big data machine you might even be able to generate some insight from those feeds).

Well that’s a wrap for this years big show.

Next up, I will be speaking at the upcoming ECO conference in North Carolina. Should be fun.

Later.

Kent

P.S. If you want to see my OTW presentation, you can find them on Slideshare.

P.P.S. For another great review of OOW13 check out this post by my friend from Turkey, Gurcan. See if you can find my unlabeled cameo in the post.

Posted in Data Modeling, Data Vault, Data Warehouse, Oracle, SQL Developer Data Modeler and tagged , , , , , , , , , , , , , , | Leave a comment
27 Jun 2013

KScope13 Day Four: Agile, Big Data, and a Very Special Event

Mid-week. Hump day. The day of the BIG event for KScope13.

Lots of anticipation for the annual Special Event… (which I will write about in a minute or so)

Morning Chi Gung as usual, but with 24 people showing up. Biggest group this week. We even have a few locals joining us now. Everyone seems to be enjoying these sessions.

KScope attendees starting the day with Morning Chi Gung on the plaza in front of Harahs casino.

KScope attendees starting the day with Morning Chi Gung on the plaza in front of Harrahs casino.

In fact, the Chi Gung class at KScope may be the original cross over session! Attendees are from across the spectrum from DBAs, to developers, to Hyperion/EPM folks to spouses of attendees.

There is something for everyone in Morning Chi Gung.

Kanban and Scrum

Everyone wants to be “agile” these days. Stew Stryker of Dartmouth University came to KScope to share with us his experience in applying first Kanban then Scrum to the software development life cycle in his IT department.

Stew Stryker, from Dartmouth College, discusses how his team has use Kanban, and now SCRUM, to improve their software development process.

Stew Stryker, from Dartmouth College, discusses how his team has used Kanban, and now SCRUM, to improve their software development process.

One of Stew’s insights was that to effectively implement a change in methodology like this and get adoption it is first necessary for the powers-that-be to recognize the current approach (usually water fall) is failing.

If you do not know you have a problem, there is no motivation to fix it, right?

A key recommendation he had was to get a consultant that knows and has implemented Kanban for database projects to come in and work with you. Don’t try to do it by just reading articles and books or going to training. There are too many nuances and organizational dynamics to account for.

A simple comparison of aspects of a traditional waterfall methodology compared to the Kanban approach.

A simple comparison of aspects of a traditional waterfall methodology compared to the Kanban approach.

Another key to succes was to prevent context switching – that is keep everyone focused on the task at hand for the duration of the interval (or sprint). He did a great little exercise with us that really showed how task switching costs a lot of time. In some case up to 10 times longer.

It was great to hear real world experiences that we could all take back to our offices and implement and discuss. His team has experienced some great success but with lots of lessons learned, which he shared.

They have now switch to SCRUM with even more success.

Hands On Lab #2

I attended my second lab of the conference to learn from Maria Colgan (@SQLMaria) on how to prevent sub-optimal plans on SQL Statements.

Oracle Senior Product Manager Maria Colgan walks us through how to analyze and and tune some queries.

Oracle Senior Product Manager Maria Colgan walks us through how to analyze and tune some queries.

It was a great session using the Oracle Demo Days virtual box image again (from OTN). Maria walked us through several queries with Explain Plans that did not seem quite right and showed us how to diagnose and fix the potential problems.

It was a little tough for those of us who have not used Linux/Unix or command line in a few years but I did learn a lot and should be able to apply that knowledge when we have poor performing queries at my clients. Worse case, I can always start up the vm again and run through the lab.

Inside the Oracle 12c Opimizer

Another killer session from Maria showing us enhancements and new features to the query optimizer in the recently released Oracle 12c.

Overview of how adaptive query optimization works on Oracle 12c

Overview of how adaptive query optimization works on Oracle 12c

How the new Adaptive Execution Plans work in Oracle 12c

How the new Adaptive Execution Plans work in Oracle 12c

The key phrase for 12c “self-healing” and “adaptive”. Remember when there were just 17 rules for the optimizer that we could control with the syntax of the query?

Long ago.

I guess this is better, but there are still rules to know to make the optimizer work well.

And Maria definitely knows them!

Big Data

These days every tech event has to talk about big data. KScope13 is no different.

Alex Shlepakov, from Accenture’s Oracle BI practice, gave a nice talk about integrating Hadoop with OBIEE using ODI.

He did a really nice job explaining all the concepts and moving parts and how Oracle addressed these things.

Alex presented about doing big data analysis using Oracle BI tools.

Alex presented about doing big data analysis using Oracle BI tools.

All the Oracle products that support the analysis of data in a Hadoop environment

All the Oracle products that support the analysis of data in a Hadoop environment

Pretty sure these products cost lots of money too! But if you want to get value out of your big data, you may have to spend big money for the tools to help (unless you have a lot of programers with really big brains).

My main take away from this session is that the tools to support Hadoop and big data analysis are evolving to make it easier for most programmers to get to the data without having to be Map Reduce programmers.

But it will still be pretty hard, so you better have a good business case for digging into it.

Special Event (aka the big party)

As in past years, ODTUG really did it up right. This was truly a special event to remember – we went to Mardi Gras World!

The annual Special Event was held at Mardi Gras World where we got to see some of the big floats from the famous parade.

The annual Special Event was held at Mardi Gras World where we got to see some of the big floats from the famous parade.

What a treat to see some of the big floats used in the famous parade. I even found a full scale replica of the Bat Boat tucked away in the back. (There was a huge Batman statue as well)

The Oracle Data Warrior finds Batman's boat!

The Oracle Data Warrior finds Batman’s boat!

The tour of the Mardi Gras warehouse included plenty of bead throwing from the floats by the board of directors and the various KScope vendors. This was followed by a nice evening of drinks and a buffet dinner with lots of great food (even some gluten free and vegetarian options).   There was plenty of dancing to great cover band called The Mixed Nuts.

We finished the evening with a spectacular fire works display (which seems to becoming a standard at this event).

We had a spectacular fireworks display (shot off a barge) at the annual KScope Big Event

We had a spectacular fireworks display (shot off a barge) at the annual KScope Big Event

Over too soon, it was last call, last dance, then back to the buses and a short ride to the hotel.

And then there were the after parties….

Stay tuned for my notes on our final day in New Orleans.

Ciao!

Kent

Posted in Chi Gung, ODTUG, Oracle and tagged , , , , , , , , , , , , | Leave a comment
23 Jun 2013
1 Comment

Welcome to the Biggest KScope Conference Ever!

Day breaks over the Mighty Mississippi

Day breaks over the Mighty Mississippi

Day one at ODTUG’s KScope13 was awesome as always with a great set of symposiums and networking events.

At the speaker meeting we learned that there are 1,400 registered attendees, 30 countries represented and over 50 exhibitors making this the largest event and largest exhibit hall in the history of the Oracle Development Tools User Group (ODTUG).

So we must be doing something right to attract such a large crowd.

Part of what we do right is getting the top Oracle developers, consultants, and Oracle ACEs in the world to present. And we get stellar participation from the Product Management and Development teams at Oracle Corporation.

Even though it was Sunday, there was a tremendous turnout for our free Sunday Symposiums. In the Symposiums we had rooms dedicated to specific topics so attendees could stay in one place all day and get a series of related talks.

After conducting my now annual Morning Chi Gung class (had nine attendees!) I spent the day in the DB & Developer’s Toolbox Symposium. It had quite a line up.

First I attended Jeff Smith’s session  on SQL Developer 4.0 which will come out later this summer.

SQL Developer Product Manager, Jeff Smith of Oracle, introduces new features in the upcoming 4.0 release

SQL Developer Product Manager, Jeff Smith of Oracle, introduces new features in the upcoming 4.0 release

Got lots of great tips and trick which I am sure Jeff will be blogging about at ThatJeffSmith.com in the near future.

Next was Oracle Technologist Tom Kyte who spoke about many new features and enhancements to SQL And PL/SQL in the upcoming Oracle 12c release of the database.

Tom Kyte (of AskTom.com fame) introduces attendees to new SQL and PL/SQL features in Oracle 12c

Tom Kyte (of AskTom.com fame) introduces attendees to new SQL and PL/SQL features in Oracle 12c

And last for my day of learning was Maria Colgan discussing changes to the Oracle Optimizer in 12c.

Oracle Product Manager, Maria Colgan, discusses the Evolution of the Oracle Optimizer: Rules based to 12c

Oracle Product Manager, Maria Colgan, discusses the Evolution of the Oracle Optimizer: Rules based to 12c

As always, Maria provided so much information my head was ready to explode from information overload. And later in the week she has a second session on this topic with even more new features to discuss. If every Oracle DBA could attend one of her sessions, all of our databases would run better!

After the sessions it was time for formal and informal networking. There was the speakers meeting and the official welcome reception in the exhibit hall. I got to attend an official Oracle ACE dinner cruise on a steam powered paddle wheel river boat complete with live jazz band. Very much in the spirit of New Orleans.

Stay tuned for more reports as the week progresses.

Kent

The Oracle Data Warrior

P.S. I got to do a little touring as well and visited the oldest Catholic Cathedral in the United States. It is quite a beautiful church inside and out.

The beautiful St Louis Cathedral near Jackson Square in New Orleans

The beautiful St Louis Cathedral near Jackson Square in New Orleans

Posted in Chi Gung, ODTUG, Oracle, User Groups and tagged , , , , , , , , , , , , ,
13 Feb 2013

RMOUG Training Days 2013 – Day 2

So on this 2nd and final day of the annual RMOUG Training Days event, I started out by attending an excellent session on Exadata for Oracle DBAs.

Even though I am not a DBA these days I thought it would be good for me to get a better understanding of Oracle’s engineered Exadata machine.

I feel very luck to have attended this session given by Oracle Technologist of the Year, and ACE Director, Arup Nanda. He had some of the best graphics and clearest explanations of the basic anatomy of an Oracle database I have ever seen or heard.

Technologist of the Year, Arup Nanda, Database Machine Administrator

Technologist of the Year, Arup Nanda, Database Machine Administrator

He gave some pretty detailed explanations of what he called the “magic” of Exadata and why it works so well. Arup even coined a new job title, which he claims for himself, DMA – Database Machine Administrator. Because Exadata is an engineered system, it contains database, storage, and networking all in one rack. This requires some skills beyond what most dbas have or are expected to have.

He gave us a nice break down based on his experience using Exadata.

Break down of skills needed to be a successful Exadata DMA

Break down of skills needed to be a successful Exadata DMA

After this talk I can see why he was give the awards. He really knows his stuff and how to communicate it. You can follow him on Twitter @ArupNanda and see for yourself.

Next I went to see my friend,and ACE Director, Galo Balda from Austin, Texas. He gave a very informative talk about Regular Expressions.

ACE Director, Galo Balda, doing his very first presentation

ACE Director, Galo Balda, doing his very first presentation

His presentation was very informative with easy to understand examples of how to write and use regular expression and associated metacharacters to do some pretty neat things with SQL. If you attended the conference be sure to download his slides. They will make a great cheat sheet.

You can follow him on Twitter @GaloBalda or go to his blog.

After a nice vegetarian lunch, I went to see Maria Coglan talk about using (or not using) hints in SQL and how it affects the optimizer. Last year at Kscope12, I attended one of her optimizer sessions and felt like my head would explode becuase of all the information she gave. She assured me this talk would not be as bad.

She was right. It was a very informative talk.

A full house to see Maria Coglan discuss Hints and the Optimizer

A full house to see Maria Coglan discuss Hints and the Optimizer

Her main message was to always use caution when using hints. You really need to understand what you are or doing or you could make your application or reports run worse rather than better.

Maria even explained how to work with applications that already have hints embedded in them.

Approach to ignoring hints in an existing application

Approach to ignoring hints in an existing application

Get her slides and follow her on Twitter @SQLMaria

After Maria’s session I did my final session for the event. I talked about my Top 10 favorite cool tools in SQL Developer Data Modeler. There were 30 or so people in attendance. Most of them even stayed through the whole talk!

Which is pretty good since I ran over my time. There was just so many tips and tricks to show.  I will put it up on SlideShare in the next few days.

The final session for the event that I attended was done by RMOUG President, my long time friend, Tim Gorman.

Tim talked about the various options for data compression in the Oracle stack.

Tim Gorman (in the shadows) giving the last talk

Tim Gorman (in the shadows) giving the last talk

Tim gave some pretty detailed explanations and tried to depict how compression works with some nice graphics. He also told us which ones cost additional license fees.

Data Lifecycle when using Compress for OLTP

Data Lifecycle when using Compress for OLTPut how have

For me, the most useful part was his explanation about how having columns at the end of the table allows a default sort of compression to take place. I had heard this a long time ago. It was the reason so many of us were taught to put all mandatory columns at the beginning of the table – it saves space. In recent years I have been told by various DBAs that the rule no longer applies or made sense.

They were wrong! Tim gave us a real world example of how putting populated columns at the end of a table cost a lot of extra space to be used.

I will be taking that tidbit of information back to the office for sure.

You can follow Tim on Twitter @timothyjgorman.

A side note about RMOUG: At lunch, Tim shared with the attendees an agenda from 1991 for the 2nd RMOUG Training Days. We now realize that we started this event in 1990 and next year will be the 25th anniversary! (I say “we” because I was part of the planning committee back then and one of the early speakers too).

Another interesting notes was that 2/3 of the speakers came form out of town. Many, including me, paid there own way. Several speakers and attendees I know even had to take vacation time from their jobs to attend.

It is that important and that good an event!

So put it on your calendar to attend what is probably the most successful and longest running regional Oracle user conferences in the country. It will be in early February 2014. Watch www.rmoug.org for details.

And of ocurse count on me to post it here too.

Ciao for now! I am off to ski with some RMOUGers tomorrow.

Kent

Posted in Data Warehouse, Oracle, SQL Developer Data Modeler, User Groups and tagged , , , , , , , , , | Leave a comment

Post Navigation