I have data, but I need information

I have a confession:  I am a statistics and data junkie.  I’m not sure where it came from… perhaps it’s from my love of baseball growing up, from playing Fantasy Football (in which I never missed a Fantasy playoff in a league I’ve played in), or tracking the Rivals star status of Texas A&M’s football and basketball recruits to assess whether I should get excited about the upcoming seasons or start blogging in support of a coaching change.  Regardless of where it came from, I have built my post-MBA career on an ability to leverage data to decipher insights and recommend future strategy.  Especially working in the BI space, I have found that organizations that understand the importance of leveraging data for competitive advantage tend to be more successful than others.

This is a good time to be an analytics professional.  Today, like no other time in history, there is a plethora of data in the world.  From organizations siphoning terabytes (and even pedabytes) of operational business data into massive data warehouses, to all of the information you can search for on Google, to the clickstreams of your website visitors, to the massive amount of data aggregators providing syndicated lists & analyst reports, to all of the chatter on the blogosphere and Twitterverse, we are swimming in an ocean of data and organizations have an endless appetite for it. 

In fact, every company can probably say these two things:  1.) We have too much data and 2.) We don’t have enough data. 

Or perhaps this is what they are saying:

• “We have too much data”  We are having a difficult time getting value out of the data we have. 
• “We don’t have enough data”  We don’t know what we don’t know or need to know.

Both of these statements/issues are related.  I have heard multiple times, especially on consulting engagements, this term:  “We are data rich, but information poor.”  This is a good statement, because it recognizes the reality that a company has a lot of data at its disposal (like most organizations) but that’s all it is… data.  It means that the organization is spending a lot of energy collecting and delivering data but what it really needs is information… insights that can drive actions that get results. 

How do I do it?  How do I take all of the data I have, or better said want to collect, which has limited value and transform it into information and insights which are extremely valuable? 

At a high level, here’s how you make it happen

1.  Follow Steven Covey’s rule from “The Seven Habits of Highly Effective People” and start with the end in mind.  In order for your data to be effectively leveraged, you must first ask the question:  “What do I want to learn?” and evaluate the questions that you will answer through your data.  
2. Pre-define your data to support the questions you want to answer and develop systems to ensure that it is consistently collected and codified according to this definition.  Your data must support your analysis, rather than your analysis support your data if the quality of your information will drive the maximum value you want to achieve.  This is the hardest work, because in many cases you may not know what you don’t know yet.  As a result, this is fluid process.
3. Collect data consistently, accurately, and with proper governance.  This is especially important when your analytical data is coming from many sources in your organization.  If your organization is truly going to be data and insights driven, then data governance and data quality must be a high priority and governed at a high level in the organization. 
4. Review, analyze, and refine often.  Once you start gaining insights, you will start to ask deeper questions and create a market for intelligence in your company.  At this point, we would start over at #1 and once again ask the question:  “What do I want to learn and how do I need to collect and define my data to get me the answer I need?” 

This is key to truly becoming data-driven.  Good data is intelligently conceived and is supported by good processes.  You can’t have good processes without good data and you can’t have good data without good processes.  The two work hand-in-hand.  It is hard work, but the value that insights can give over mere data is well worth the effort to align an organization from being a data consumer to an insights creator.  

Self-Service and Real-Time Analytics

As I had written before, the consumer-facing market is migrating into more and more of a self-service model with the prevelent use of ATMs in banking, self-checkout and self-ordering devices in Retail and Hospitality, and the growth of eCommerce online merchants. This channel obviously provides the consumer and the merchant with some advantages (convienence, efficiency, etc.) but it has one notable disadvantage. This disadvantage is obviously the lack of person-to-person interaction in these exchanges. In such an environment, how does the consumer feel that she is "known" and the merchant able to conversely "know" the consumer?

In businesses where self-service works well, the merchant has to "know" the consumer based on her previous interactions with the company. What are her tendancies, history, affinities, etc? Based on the occasion that she is shopping within, what type of merchant offers will be relevant to her? This type of knowledge requires "real time analytics" in the sense that the shopper's behavior is analyzed and her tendancies are known... and then based on the predicted shopping occasion (fill-in, splurge, pantry filling, convienence trip, etc.) can the merchant offer her an incentive to make that visit more profitable for the merchant and more valuable for the consumer?

Obviously, in the eCommerce world the merchant will know through the contents of the consumers' shopping basket and her session clickstream behavior. Is this possible in a brick and mortar world? Is it possible without real-time basket information? Possibly... but this would require that the shopping and occasion data is available and the patterns are analyzed on every customer... something that requires sophisticated analytical CRM tools and an active data warehouse environment with the horsepower to support it (obviously the merchant needs to know in some way when the consumer is interacting with them). In the banking world, this can effectively be handled through the ATM device similarly to the way Amazon would offer a next best offer from a purchase.

However, what would the ROI on consumer marketing efforts be if, like eCommerce, you could analyze and dynamically offer value-added services in the midst of the shopping occassion... at the point of purchase instead of the point of sale? There are some mechanisms for doing that today, but they are by no means segemented. There is one retailer that is experimenting with self-scanning technology within the store... if this could be married to the offer and relationship management technology it could be a huge win for them, both in terms of understanding their shoppers but also in marketing effectively to them on their terms.

The Analytics ROI Question: "Which came first, the chicken or the egg?"

Everyone has likely heard the riddle: Which came first, the chicken or the egg? I do not know what the answer to that question is, and perhaps it is more a philosophical one to assess a person's position on origins...

Having spent a signifiant part of my life in the business development world in the Enterprise Data Warehousing market over the past couple of years, the question is also generally heard but in a different form: "Who takes credit for the ROI on my Business Intelligence project, the data warehouse infrastructure or the analytics/applications that make sense of all of this data?" In the sales world, this is an important question because it determines the strategic importance of a vendor's products. Of course, that strategic importance ultimately determines the vendors' share of wallet with that company and over the long term in the industry. Money talks. Therefore, like a good politician every technology is out there to take credit for the analytical ROI.

For an analytics or application solution like SAS, Retek, Siebel, TRM, i2, SPSS, Microsoft analysis tools, etc... the value is obvious. The deep analytics and decision-making capabilities enable companies to drive to decisions and answers they were not able to reach prior to having those tools and applications in place. Without the decision-making intelligence, no decision, no benefit, no ROI.

However, the database and infrastructure vendors also have a case to be made... especially the MPP (Massively Parallel Processing) database technology vendors like Teradata, DATAllegro, and Netezza. Without the ability that the database engines provide to crunch through terabytes worth of detailed data at the atomic level, the analytics engines that depend on this machinery would not be as effective. Therefore, they may say: "Not too fast, you're delivering that value on our nickel. We should take credit here."

Interestingly enough, only recently the BI market was relatively fragmented. Application vendors, BI vendors (Cognos, Business Objects, Hyperion, Microstrategy, etc.), and the database infrastructure vendors (Oracle, Teradata, IBM, etc.) were all in their own camps. And as such, each vendor sold on the merits of their own tools in this interdependent technical environment. Today, the industry is starting to consolidate where BI and application companies are being bought up by larger infrastructure players (IBM, Oracle, and SAP... who is taking the analytical market more seriously now). As such, it is now possible for a company to get all of their analytical needs met by a single vendor and gain a complete picture of the benefits and returns without necessarily having to ration the cost/benefit among tools offered by different vendors. Of course, this is being said as the ROI evaluation process provided by technology vendors is, at the end of the day, a marketing and sales function rather than a consulting or financial evaluation function. In this world, having a vendor that can offer a single package that will deliver all of the goods will be to your benefit.

However, while there is consolidation in the market, all of the tools are (as far as I know) compatible with all of the major infrastructure platforms. Additionally, the deep analytics powerhouses remain independent (I'm thinking of SAS here) so there will be tension in the ROI credit discussion. Yes, even with all of the consolidation this is a very competitive market... especially with the DWA (data warehouse appliance) significantly lowering the cost for performance for infrastructure which is a great thing for IT staffs with limited budgets.

For an industry poised for growth, and a market ready to leverage those capabiliites, that is a good thing.

Microsoft purchasing DATAllegro

On Thursday, July 24th, Microsoft and DATAllegro announced that Microsoft will purchase DATAllegro, one of the earlier data warehouse appliance startups. I arrived at home early this week to catch up with news in the States after a 10 day stint in Estonia working with High School students, and was intreagued with this news coming from the technology world.

My amateur first thoughts on this aquisition:

1.) From knowing how DATAllegro has chosen to go to market in the past year and a half, this purchase may make the most sense for Microsoft. DATAllegro's V3 appliance was the first major DW appliance product to significantly leverage third party hardware to move the market toward more of an open platform. Most (if not all) of the other major players in this space leverage a proprietary hardware platform for their solutions. Microsoft, not being in the server business, should be able to leverage and expand upon the DATAllegro partnerships for product development as they evolve the software and database components of this solution.

2.) While the final evolution of the Microsoft-DATAllegro purchase may be unknown, I imagine that Microsoft will be working over the next months to integrate their existing BI applications and development framework into the newly purchased appliance solution. This could be a huge win for companies that are looking to leverage skill sets from their application development groups into their BI initiatives. Adding Microsoft's vast development network of Professional Services firms to that mix will provide more incentives for companies to look at Microsoft as a contender. Time will tell whether they will deliver on these aspects of their brand in regards to Enterprise intelligence... be on the lookout. If Microsoft is able to fully integrate their existing solutions and .NET (including Excel and Access... still the leading BI tools used in the market) to the DWA platform, they should have a compelling value proposition especially for mid market companies that need rapid deployment and quick ROI wins to justify projects.

3.) This move further solidifies the data warehouse appliance's position in the Business Intelligence market. Microsoft would be the third major vendor to market an appliance product (HP's Neoview and the Teradata's 2500 being the others), which provides evidence to the market that these solutions are here to stay and have support. This should help Netezza's positioning as well as they were the first player in this market space and have a lot of mind share where data warehouse appliances are concerned.

4.) If successful, could this speed the rise of the open hardware platform in Business Intelligence? There are solutions out there today (Oracle's data warehouse platform being one of them), but the jury is still out as to whether they can economically scale. When the ability to economically scale data warehouses on commodity platforms is found, it should be game changing (in the same way that Netezza's first appliance changed the EDW game).

I'm interested to hear your thoughts.