Today’s enterprise data warehouse is undergoing significant technology changes as the business relies more heavily on the wealth of information it can provide. The next major step in data warehousing is to enable real-time business intelligence or in other words providing an active data warehouse which provides both historical analysis as well as active tactical look-ups.

The driving forces behind business intelligence today include savvier, demanding customers with a lower patience threshold as well faster moving business operations and tighter regulations on maintaining historical data about the business.

An “active” data warehouse essentially aligns business intelligence with operational intelligence, creating new “active” intelligence that allows for actionable information for the entire enterprise – every layer within the organizational hierarchy benefits from it, not just the business analysts and strategists. Consistent, well-informed decision making across the board drives higher levels of customer satisfaction and ultimately competitive advantage.

Forward-looking organizations understand that the data warehouse can play a prominent role in supporting overall business strategy. By extending the data warehouse to front-line operational users, suppliers and customers, data increases in value and enables users to make better-informed, timely decisions. When business intelligence is no longer limited to management for historical reporting and executives for long-term planning, it can have significant impact across the enterprise and beyond, reaching and benefiting a larger volume of users – from customer service personnel, call centers, sales and marketing to suppliers, partners and even customers.

In pursuit of active data warehousing, success depends largely on selecting the right data integration and acquisition approach in order to not only improve the quality of the data in the warehouse but to make sure it is the most timely and up-to-date. Fresh data delivered to the data warehouse drives improved business intelligence. Information is a time-perishable asset and its timeliness is directly correlated to its value for the business. An effective active data warehouse requires the continuous capture and delivery of real-time data from key transactional systems into the warehouse.

Making the Case for Real-Time Business Intelligence: Look to the Customer

Until recently most enterprise data warehouses were used exclusively for after-the-fact reporting, analytics and forecasting, but now organizations recognize that the data warehouse holds a wealth of untapped, valuable information which, if made more accessible, operational and active, could be leveraged in a number of new ways to meet ongoing business needs. Month, week, day, and even hour old data is often no longer adequate to serve the real-time enterprise. In order to gain true competitive advantage, organizations must consider new approaches to data warehousing that ensure more timely data to drive faster, more informed decisions. In the argument for real-time business intelligence, businesses need to look no further than their customers.

According to recent studies, the cost of acquiring new customers is 8 to 10 times greater than the cost of retaining existing ones. Focusing resources on identifying and prioritizing the most profitable and valuable customers through targeted marketing campaigns, improved service and an enriched customer experience yields a better return on investment.

Airline Customers

Let’s take the example of an airline that needs to have a reliable understanding of which customers are their most frequent, most loyal, and therefore most profitable. When this information is coupled with active intelligence when a high value customer is travelling, it can have a direct impact on the overall customer experience and ultimately their loyalty.

If their highly valued VIP customer is delayed on a long haul flight, reliable real-time data integration delivered to their operational data warehouse would ensure that the airline staff on the ground are proactively alerted so they can take care of re-routing on alternate routes in order to avoid further delays and frustrations. An active data warehouse would allow the ground crew to “look up” specific information about such key customers and take action against critical information, as it becomes available. As airlines and other customer-driven businesses consider the higher cost to acquire new customers, leveraging “active” intelligence like this to improve the customer experience, even as it is happening in real time, can increase loyalty and satisfaction among existing customers.

Online Retail

Another example, online retailers (or e-tailers) have the ongoing challenge of winning over today’s “no-loyalty” consumer. In addition to their online store availability and performance challenges, e-tailers are in danger of losing customers and revenue if they do not effectively cross-market and cross- or up-sell based on key customer intelligence. It is simply too easy for consumers to move on to the competition. Like the airline example above, e-tailers are starting to realize the benefit of accessing and leveraging transactional data in real-time about the customer for their benefit.

Imagine analyzing data about a customer while they are shopping such as: Did an email campaign drive them to the site? What areas of the site are they clicking and browsing? Is there an opportunity to cross-sell? What is their buying history? With an active data warehouse, it is not only possible, but profitable, to achieve a single view of the customer across multiple business lines, enabling e-tailers to identify the most successful marketing campaigns, tailor customer communications, and track behavior. As a result, e-tailers can understand shopping patterns, habits and preferences, and get closer to their customers than ever before.

With today’s savvy online buyer who has a wealth of choice at their fingertips, organizations need to understand how best to leverage critical up-to-date intelligence about them. The data warehouse can allow organizations to capitalize on real-time business intelligence and create new opportunities for retaining high value customers and building market share. Look no further than your customers to figure out the “why” behind real-time business intelligence. Now let’s move on to the “how.”

Wheelbarrows vs Conveyer Belts—Data Acquisition Approaches for the Data Warehouse

There are numerous data integration approaches that serve the data acquisition needs of a data warehouse. However, when you start to move towards active data warehousing, there is a limited choice of technologies that facilitate real-time data delivery. The challenge is to determine the right technology approach or combination of solutions that best meets the data delivery needs. Selection criteria should include considerations for frequency of data, acceptable latency, data volumes, data integrity, transformation requirements and processing overhead.

To discuss the different methods for feeding the data warehouse, we’ll set up an analogy which illustrates in very simple detail how the key technologies work – wheelbarrows and conveyer belts. Historically, the wheelbarrow and the conveyer belt were designed to improve the way heavy, physical loads were carried from one place to another, with one a vast technical improvement over the other. Despite the simplistic design of the wheelbarrow, its impact on the industry was undeniable. And while the wheelbarrow even today remains a useful tool, the conveyer belt revolutionized labor and efficiency by continuously and automatically speeding heavy loads from one place to another.

We can apply this analogy to the different methods for feeding data in to the warehouse. According to independent analyst research, about two years ago more than 60% of organizations surveyed were updating the data warehouse on a daily basis(1), but anticipated an increase in that frequency over the next 18 months. In fact in 2005, a subsequent survey(2) revealed that the frequency had moved along a continuum toward more real-time data integration (minutes, seconds or instantaneous), with the actual speed reported by organizations dependent upon the application.

The trend toward real-time data acquisition continues with more organizations moving away from previous methods, which includes batch processing using Extract, Transform and Load (ETL) technology solutions or in many cases, hand-coded scripts. The “wheelbarrow” of the data warehouse, batch processing is an intermittent data load system, feeding data on a daily and sometimes weekly basis. While adequate for some business intelligence needs – HR reporting, financial corporate performance management or regional sales breakdown, for example – the latency of batch processed data is too high for most enterprise intelligence needs. Batch processing also creates substantial overhead on the infrastructure and hence additional costs in maintenance.

Additionally, feeding data to the warehouse using the batch load method provides insight only into what happened in the past, and therefore limits an organization’s ability to glean intelligence for predictive analysis or any of the strategic business intelligence described above (airline and e-tailer examples). And even as organizations move away from nightly batch to mini-batch processing throughout the day, processing overheard on source systems can often burden the infrastructure causing performance degradation and even outages and still leaves gaps in the speed and currency of the data.

To reduce latency and drive greater value in data warehousing, organizations are starting to recognize that data acquisition and feeds need to approach real time. This simply can’t happen with batch processing – whether using hand-coded scripting or ETL. The time value of data(3) – data captured from a business event, delivered to the warehouse and a resulting action taken – is driving a move toward real-time feeds to the data warehouse.

Enterprise Application Integration (EAI)

Like the conveyer belt, real-time data acquisition to the warehouse delivers continuous data feeds, with sub-second latency. To solve the real-time challenge, businesses are turning to technologies like Enterprise Application Integration (EAI) and Transactional Data Management (TDM), which offer change data processing, delivering smaller “chunks” of data continuously with no overhead and lighter transformations. EAI has a greater implementation complexity and cost of maintenance, and handles smaller volumes of data. TDM provides the ability to capture transactions from OLTP systems, apply mapping, filtering and basic transformations and delivers to the data warehouse directly. As new transactions are committed at the source database(s), that data is immediately captured and delivered to other systems with only sub-second latency. TDM causes no system interruption or outage windows and can support the movement of thousands of transactions per second yet imposes negligible impact on the source and target systems. TDM’s greatest benefit is that it eliminates batch window dependency, is easy to implement and maintain, and delivers the lowest overhead of the technologies described above when designing a real-time, active data warehouse.

Real Time vs. Right Time

As discussed above, the pressures are both internal and external for reducing data latency and phasing out batch operations in favor of real-time data warehousing. When evaluating data acquisition technology solutions, it is important to architect and design your warehouse to accommodate the future needs of the business. Providing business users with accurate timely data in the warehouse is paramount to enabling real-time business operations. Once the data is available and accessible, the business will determine the “right time” to consume and analyze that data. Depending on the critical or non-critical nature of the varying uses for intelligence (HR, maintenance, customer inquiries, transaction lookups, marketing analysis, etc.) the user can determine right time vs. real time. Right time is then necessarily a component of user preference, or decision latency, and not a technical constraint (forced data latency).

For Improved Business Intelligence, Real Time is the Ticket

Active data warehousing is an important, growing initiative for any organization striving to gain a stronger understanding of their customer and business and excel in today’s competitive market. The resulting real-time business intelligence empowers forward-thinking organizations to not only analyze what happened in the past and what is happening now, but also to derive greater value from the business data collected to determine and influence what should happen next. Real time represents a powerful shift in business intelligence, and one that has the ability to influence overall business strategy toward increasing the bottom line.



_______________________________
(1)According to the Forrester-TDWI August 2004 Quarterly Technology Survey
(2)Gartner 2005 survey of 450 real-time data integration users
(3)“The Time Value of Data” is taken from TDWI, The Business Case for Real-Time BI, based on the concept developed by Richard Hackathorn, Bolder Technology