Avoiding the top mistakes of using an operational data store

Operational data stores enable real-time decision-making by bringing together multiple sources of transactional data in one place. Here’s how to optimize their full value.

Over the years, many types of data management technologies have emerged to provide an easy way to store, collect, process, analyze and visualize data. However, many organizations find it difficult to manage the overall data ecosystem. The problem is not about generating or storing the data; rather it is about extracting meaningful information from the data.

In most cases, the primary systems of record—like customer relationship management, enterprise resource planning, IT systems management platforms, etc.—provide highly valuable transactional data. However, they offer only limited reporting capabilities. To fill the gap, many business users import data from multiple sources and try to transform and aggregate the data using a spreadsheet program like Excel or a data analytics and visualization platform like Alteryx.

A better solution, however, would be to implement an operational data store (ODS). An ODS is different from other data management solutions because it is meant for transactional data. Transactional data is mostly stored in ERP, software-as-a-service and custom applications, whereas historical data is leveraged through a data warehouse, data lake, data mart and data lakehouse for analytical reporting.

An ODS platform, meanwhile, is a tactical solution that enables transactional data to be aggregated from multiple applications in real-time for a composite view of daily operations. The ODS ingests real-time transactions from multiple data sources and aggregates them for a specific business case. Real-time integration enables business users to make data-driven decisions based on the current snapshot of the operational data and status.

Figure 1

In our work with organizations to assess, design, build, implement and support their data platforms, we’ve encountered many instances where clients have misunderstood what is needed to make the ODS work to its fullest capacity. Based on these experiences, we’ve identified the five most common mistakes we find when managing operational data stores and how data leaders, chief data officers and chief information officers can work together to avoid them.

Operational data stores: top 5 mistakes to avoid

1.    Failing to clearly define the scope of the ODS

Businesses need to understand the main role of the ODS: It is a tactical solution that works in parallel with the front-line transactional platform, consolidating data from multiple data sources to provide a holistic view of the transactions in real-time. But CDOs and other data leaders need to ensure that the ODS has a clear objective and scope, as well as well-defined functional capabilities.

By doing so, they can ensure that the ODS platform collects accurate and precise data from the upstream applications, as well as identify the key data entities and business process for aggregating these entities for a specific use case.

2.    Inadequately incorporating governance management

Many organizations have guidelines in place to ensure data quality, reliability, accuracy, stewardship and security. But when these guidelines are not enforced in the ODS, the result is poor data quality, reliability and authenticity, which hinders business decision-making.

The CDO should work closely with the CIO to make sure every platform implements the data governance guidelines at every level—the data ingestion pipeline, data transformation, data storage and data consumption. Failure to implement the data governance framework on an ODS results in unreliable data for business decision-making.

3.    Underestimating the complexity of data transformation and ingestion

The process of ingesting data from multiple sources and performing transformation in real time is a complex process that can slow down data delivery and increase data quality risk.

CDOs need to work closely with their data architecture specialists, developers and business users to identify the volume, velocity and variety of the data transformation needs. Additionally, the overall architecture needs to enable the delivery of real- to near-real-time data with high quality and consistency.

4.    Ignoring real-time requirements

An ODS and an enterprise data warehouse share similar characteristics, as each performs data integrations, consolidation, transformation and aggregation. But the key difference is that an ODS is a tactical system that works with real-time data, while a data warehouse is an analytical platform that works with historical data. Organizations often make the mistake of ignoring the real-time data needs of the ODS and treating it more like an analytical and reporting platform.

The CDO needs to make sure the functional capabilities for the ODS and enterprise data warehouse are clearly defined. The tactical platform requires the continued flow of real-time data for better decision-making. Providing delayed transactional data will result in reduced operational efficiencies.

5.    Insufficiently planning for scalability

Because the ODS is a parallel system to the front-time transaction system, it needs to scale to peak data volumes. Otherwise, it can be a bottleneck for business operations. The CDO and data leadership team need to anticipate the data volume and platform usage and make sure the ODS can be scaled to meet business needs.

As this is a tactical system, the organization cannot afford to keep it offline for a long duration. A backup system or disaster recovery system needs to be implemented to make sure the business workflow is not disturbed.

Optimizing the use of an operational data store

CIOs, CTOs and CDOs are finding it difficult to manage the ever-growing volumes of data while also increasing data quality, managing data governance, reducing data silos and boosting operational efficiency.

ODS platforms can be a key part of the data management solution—as long as IT and data leaders understand these key mistakes and how they relate to each other, so they can take measures to avoid them. The key is to understand that an ODS is a tactical operation platform that provides real-time decision-making and a snapshot of the operational execution status.