The Data Warrior

Changing the world, one data model at a time. How can I help you?

Archive for the tag “agile development”

Agile Amped Interview

Last week I had a great time speaking and networking at the 1st every Agile Alliance Tech conference (#AATC2016), which was held at the newly renovated Marriott Crabtree.

The conference was a great success (IMHO), with some very interesting talks about TDD (Test Driven Development), Pair Programming, and Mob Programming (new to me!) among others. And of course my talk on Agile Data Engineering. You can catch up on all the action by reading the Tweet stream on #AATC2016.

While there I had the honor of being interviewed about my thoughts related to agile, data warehousing, and data vault. (And of course I mentioned Snowflake too!). The interview was hosted by SolutionsIQ and is now included in their online library of video podcasts about agile topics, called Agile Amped (a FREE resource).

Here is the interview:


Here is a direct link to the interview on the SolutionsIQ site.

Thanks to SolutionsIQ and Neville Poole for doing the interview and to Agile Alliance for inviting me to speak.

Agile On!

Kent

The Data Warrior

P.S. Keep your eyes open for #AATC2017 as I am pretty sure we will do this again next year!

P.P.S. Before the conference I gave another interview where I talked about why we need Agile Data Engineering.

4 Keys to Succeeding with Agile Data Warehousing in 2016

I have been out giving talks again on using agile methods for data warehouse and business intelligence projects, so I thought it was time for me to share my thoughts about the 4 key elements you need to be successful with an Agile DW project in 2016.

Adopt an Agile Methodology

By this I am talking about SCRUM, Kanban, ScrumBan, or DAD (Disciplined Agile Development), among others.

Go read the blogs, read the books, study these methods. Attend a conference (like Agile Tech in April). Figure out what will work for your organization’s culture and leverage the skills of your staff. One size does not fit all.

In past engagements I have used approaches primarily based on SCRUM and Kanban. Both have been very effective once we got our processes down.

If you need/want help, find a good agile coach.

Use an Agile Data Engineering Approach

If you want to develop your data warehouse in an agile, iterative manner, then you need a way to design your EDW repository that lends itself to this approach without causing huge re-engineering pains (known as refactoring) in future iterations.

The best way I have found is using the Data Vault modeling approach. It was designed specifically for building data warehouses in this manner. I have written much about this approach and give many talks showing examples of successful agile projects using Data Vault. And there is plenty of material available to help you learn how to do it (see the books on the sidebar of this blog).

Also keep an eye on Dan Linstedt’s twitter feed and blog for his training classes.

Use Data Warehouse Automation Software

No better way to get agile and deliver results fast, than to automate as much of your development work as possible. If you use repeatable patterns (like Data Vault) in your design methodology, then it is even easier to automate and greatly reduce your time to market.

There are two vendors in the market that I like a lot and have had some experience with. They are WhereScape and AnalytixDS. And both support not only “traditional” approaches to data warehousing (like automating the ETL for a Type 2 Slowly Changing Dimension) but they both also support Data Vault (and both will be at WWDVC 2016).

Which of these tools you might use depends on your approach, your current tools, and your skills.

If you are coming from a more traditional DW paradigm and use ETL tools like Informatica, Talend, or DataStage, then I would recommend you look at AnalytixDS Mapping Manager which allows you to generate your ETL code from source to target mappings.

If you are just getting started or are committed to more of a database-centric approach and want your ETL or ELT code to run in the database, then look at WhereScape’s products.

Both are great companies with knowledgable people and happy customers.

Your third option is to write your own automation routines. There are many shops doing that as well. Just be sure you have the appropriate skills in house and can allocate the upfront time to get going (a month or so at least).

Deploy on an Agile Data Warehouse Platform

So now that I have learned about Elastic Data Warehousing in the cloud, I can’t imagine trying to do an agile DW project any other way.

Of course I am referring to Snowflake Computing’s DWaaS (data warehouse as a service) offering. Yes, I might be a bit biased since I do work for them now, but…this tech is really good!

From a features perspective, what I am talking about is having a high powered, easily scalable database that supports BI and analytic workloads and does not require a ton of time to configure and tweak.

Why do I think that is a success criteria? Because I have spent way too many months on way too many “agile” projects waiting to get access to the hardware! Or I get access and we either run out of space (e.g., “we had no idea you need THAT much storage”) or we can’t properly test production level loads and queries because the development box does not have enough horsepower.

Taking advantage of the elasticity of the cloud solves both of these problems and the folks at Snowflake have successfully built an RDBMS in the cloud that specifically harnesses these features and leverages them for data warehouse and analytic workloads by providing the ability to scale up and scale down both storage and compute resources on demand.

That and its many other features, give me the infrastructure I need to get an agile data warehouse project off the ground almost instantly. And I can do a Data Vault on Snowflake too.

Very cool.

So what do you think? Are you ready to accelerate your team’s performance and adopt an agile approach to data warehousing?

I hope this post gives you a few ideas on how to make that happen.

Model on!

Kent

The Data Warrior

 

The 12 Steps to Faster Data Warehouse Success

Announcement!

I have exciting news!

With the help of my good friend Dan Linstedt (of LearnDataVault.com fame), we have just launched my first online training video based on my very popular white paper and talk: Agile Methods and Data Warehousing: How to Deliver Faster.

Most of you will agree that data warehousing and business intelligence projects take too long to deliver tangible results. I am sure all you project and program managers wish it was not true.

Often by the time a solution is in place, the business needs have changed.

With all the talk about Agile development methods, including SCRUM and Extreme Programming, the question arises as to how these approaches can be used to deliver data warehouse and business intelligence projects faster. This new online course will look at the 12 principles behind the Agile Manifesto and see how they can be applied in the context of a data warehouse project. Then I will talk about some of the specific agile techniques I have used with great success on my projects over the last 15 years. The goal is to determine a method or methods to get a more rapid (2-4 week) delivery of portions of an enterprise data warehouse architecture.

The last time I gave this talk, in Helsinki, Finland at Harmony 2014, I had standing room only and ended up being rated the 2nd best speaker at the event (pretty cool!). It was so popular that the UK Oracle Users Group asked me to write an article on the same topic for their international newsletter.

Since many of you don’t get the chance to travel to events like this (or may have missed my session), you can now see my talk online, at your convenience, for much less than the cost of a conference fee (and the airfare to get there!). We just filmed it last week, after I completed my most recent agile data warehouse engagement, so it contains some new insights and stories that even the folks in Helsinki did not get to hear.

As a bonus, once you have finished the course, you will be able to download a free copy of the detailed article I wrote for UKOUG.

If you have questions during or after the course, you can post them right there in the training portal where I will answer them. So in addition to the training course and the white paper, you also get interactive access to me!

How do I sign up?

So how do you sign up for this new class and how much does it cost?

Well, the full price for course will be $199, but for those of  you who read my blog, I have a special Valentines Day Special offer: if you are one of the first 50 people to purchase the class between now and midnight February 15, 2015, you get a full 50% off the retail price.

So that is $99.50 for over an hour of valuable content PLUS a copy of my white paper (and access to ask me your burning questions).

Use the coupon code: GRAZIANO50

You can buy it now by going to the all new Learn Data Vault training portal now.

On the site you see the class description, outline, and my introductory video, along with the “Buy Now!” button.

So hurry and cash in my special gift to you before the time is up (remember after 2/15/15 it will be $199).

Applying Agile

For those of you who had no idea there were 12 Principles behind the Agile Manifesto, let me tell you about one that I think is vitally important: Principle #6

The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.

This means the team works best when co-located so they can easily talk to each other during the day.

HINT: If not co-located, you need to be sure you have an adequate instant messaging system in place to facilitate their daily interaction. And that the team agrees to use it!

In addition, another best practice is to hold Team Huddles every morning. In the class, I give a lot of details about huddles and how they work, but the main point is that the team needs to meet briefly once a day (usually the morning) to make sure they are all one the same page as to what they are all working on.

I can tell you for a fact, that the daily huddles and ongoing interaction is definitely a critical success factor in adopting agile practices for your data warehouse team. I have seen great success where this was implemented properly and I have also seen lots of issues when the team did not communicate daily. There is no better recipe for disaster than to have your data architect building the wrong view when the report writer is trying to finalize the output with the user. Yikes!

So, if you want to learn how to apply the 12 Principle of Agile to become more successful in delivering usable results to your data warehouse and BI program, please go over to the training site and sign up from my class.

Here’s to your success!

Kent

The Data Warrior

P.S. Don’t forget to sign up before 2/15/15 with coupon code GRAZIANO50 to get 50% off the full price.

 

Live from the 1st Annual World Wide Data Vault Consortium: Day 3

Well it was the last day of the 1st annual WWDVC. What an event is has been. (See recap of Day 1 and Day 2, here and here)

The sign outside the meeting room

The sign outside the meeting room

Don’t forget you can see all the action by searching on #WWDVC on twitter.

Agility and Data Vault

Long time data vault advocate Tom Breur opened our closing day with a talk about how we should strive to build the right product and build it right without creating more technical debt in the process.

He said agile is about taking small steps not about being faster. If we do this right, the solution we deliver should generate more legitimate business requirements.

He encouraged us all to read the Theory of Constraints and The Goal (I have), and to learn about lean delivery. Our goal should be to deliver continuously and consistently. The shorter the sprint, the better.

Next he went on to tell us his conclusion that while building a Kimball based solution may appear to deliver more value to the business, it takes too long. And in the end it is a fragile and rigid solution subject to major re-work when requirements change after deployment.

We can deliver value, quicker, using the data vault method. And what we deliver can be done incrementally and more easily added to without re-work.

Tom drew this to show how much quicker we can deliver some value sooner

Tom drew this to show how much quicker we can deliver some value sooner

Data Vault Case Studies

John Sells and Josh Bartells from Data Blueprint shared with us their experiences implementing data vault solutions for their clients.

John and Josh several successful DV projects

John and Josh share several successful DV projects

Why Data Blueprint decided to use DV as their consulting

Why Data Blueprint decided to use DV as their consulting

The gujys from DataBlueprint discuss hurdles and objections they encountered selling DV to clients

The guys from DataBlueprint discuss hurdles and objections they encountered selling DV to clients

It was great to hear about their success stories from the field and see how they addressed the challenges many of us have faced.

Data Vault Modeling Tool

MID GmbH from Germany has a pretty nice modeling tool with built in capabilities to support modeling a data vault solution from stage tables through reporting (including using the new DV icons).

Nice ability to visually show relationships across diagram types

Nice ability to visually show relationships across diagram types

MID's modeler has nice diagram to show all sorts of metadata relationships

MID’s modeler has nice diagram to show all sorts of metadata relationships

This is a tool worth checking out if you are doing a lot of data vault modeling.

Concluding Remarks

Dan closed out this inaugural event with a few remarks, some memories, thanks, and talk about plans for doing this again next year.

Dan closes out the 1st Annual WWDVC and asks about next year - where and when?

Dan closes out the 1st Annual WWDVC and asks about next year – where and when?

Guess I need to be careful what I say! :-)

Guess I need to be careful what I say! 🙂

One last time I have to say I am glad I came and can’t wait to do it again next year!

Snowy St Albans

Snowy St Albans

So long for now from Vermont.

Don’t forget to check out LearnDataVault.com and get ready to join us next year!

Kent

KScope13 Day Five: C’est Fini!

Yes, the last day of the conference arrived. Many folks got a bit of a slow start as they recovered from the prior evenings festivities (but that is true almost any day in the French Quarter!).

Morning Chi Gung was smaller but still a respectable turnout of 14 people. Some of my participants had already started their trek home, others just could not quite get up… c’est la vie.

Morning Chi Gung participants practicing a  still stance meditation and breathing exercise.

Morning Chi Gung participants practicing a still stance meditation and breathing exercise.

The KScope Social Network

My first session on this final day was Charles Elliott from Rittman Mead, discussing how to do Social Network Analysis with Oracle Tools. He discussion centered around using R and a graphing tool called D2 (www.d3js.org).

Network analysis of KScope contributors and influencers across two tracks.

Network analysis of KScope contributors and influencers across two tracks.

Charles had mined some data (from twitter I think) and then charted some of the connections to determine who were the major influencers related to KScope13 and several of our tracks. My name is up there, but I am a very small dot. You might be able to see in the picture a pretty big dot which is Gwen Shapira, who was not even at the conference this year!

Looks like very cool tech. Not sure where I would use it (yet).

Agile Case Study?

One session I was not too happy with was this one. It was titled as an agile implementation success story but it was (IMO) a veiled pitch for a free addin to SQL Developer.

The entire presentation was a statement of issues in trying to do version control of database objects. And they were all legitimate issues. But there was no case study of a specific project and how they tried to solve the problem. The solution was a product from the speaker’s company that they have developed as an add in to SQL Developer to allow you to control database object check in and check out at the database level.

Nice idea. Looks like it will work.

But, the product is not even available today! It will be release later this summer and is free for up to five users (then I assume it will cost $$).

My beef is that this should have been clearly labeled as a vendor presentation not as a “success” story.

More Crossover

For my last session I went to Stewart Bryson and Edward Roske’s presentation on Innovation in BI: Oracle Business Intelligence Against Essbase & Relational.

This was a great collaborative effort between these two guys who work for companies that might be considered competitors. I am pretty sure this is a KScope first (in many ways).

A KScope 1st: Stewart Bryson and Edward Roske do a joint presentation on using OBIEE against both Essbase and a relation data warehouse at the same time.

A KScope 1st: Stewart Bryson and Edward Roske do a joint presentation on using OBIEE against both Essbase and a relation data warehouse at the same time.

In the end, after some demonstrations of how to do this, they guys left us with a really nice comparison chart on when to use which tool and its relative effectiveness in solving specific problems.

Edward and Stewart came up with this nice chart trying to compare the two technologies on a number of features and functions.

Edward and Stewart came up with this nice chart trying to compare the two technologies on a number of features and functions.

Nice job guys!

C’est Fini

A relatively new KScope tradition is to hold a final closing general session. Here we say farewell to the event and the city with some slides and humorous videos from the week.

We also get to learn who the top five speakers were for the event (as ranked by session evaluations) and who the speaker of the year winner is.

This year the Best Speaker Award went to Edward (don’t call me Ed!) Roske from InterRel.

There was also an award for the best Kscope Ambassador (the one who helped with the most sessions). This went to Mark Becerra.

Congratulations to both!

Attendees at the KScope13 closing session with their KScope14 travel mugs

Attendees at the KScope13 closing session with their KScope14 travel mugs

As a final note we got a great little video welcoming KScope14 to Seattle.

The even bigger news was that we already have sponsors for the 2014 event! Platinum, gold, and sliver level sponsors have already signed! Plus a boat load of exhibitors.

We must be doing something right that they have secured their spot a year in advance.

We have come a long way baby!

The website for Kscope14 is up and running. You can register, submit an abstract, and make your hotel reservation for next year.

So head on over there now while you are thinking of it!  KScope14.com

One Last Shot

I can’t end this series without at least one picture of the great food here in New Orleans. I did eat well!

Ending the conference on a high note with blueberry upside down cake at celebrity chef Emeril Lagasse's restaurant NOLA in the French Quarter.

Ending the conference on a high note with blueberry upside down cake at celebrity chef Emeril Lagasse’s restaurant NOLA in the French Quarter.

See you in Seattle (if not sooner)!

Kent

The Oracle Data Warrior

Post Navigation

%d bloggers like this: