Overview

The Operational Data Integration (ODI) product is designed to improve the performance and capabilities of moving data between “data providers” and “data consumers”. ODI is a middleware that serves as a flexible central bulk data-store delivered by data providers which in turn is made available to data consumers. What has historically become a time consuming and costly process of extracting and delivering bulk data to numerous data consumers is now offloaded to the ODI system which provides standardized data collection from data providers, significant reduction of extraction routines, and efficient data distribution and delivery to consumers. The diagram below illustrates this relationship.

As a middleware product, ODI is largely invisible to customers. Most of the processing of ODI is also invisible to consumers, except for those aspects that allow the consumers to access the data. These touch points of ODI with which consumers will interact are:

• Notification that provider data for which a consumer is authorized is ready for consumption
• A service for consumers to request credentials to access SQL View data
• Direct access to the operational data store (SQL Views) to fetch data
  -or-
• File extracts in standard CSV form

Process overview

Once data is received by ODI from the Data Provider, it is processed for consumption. When the data is ready for consumption a message is made available to the Data Consumer via the Enterprise Event System (event 50040). There are two distinct methods of data retrieval, Direct SQL data access and File based data access. Direct SQL data access requires the Data Consumer to acquire secure user credentials via the ODISvcCred message through jXchange Service Gateway. The user credentials are active for a limited time and cannot be used after expiration. Once expired, new credentials must be requested using ODISvcCred. The process repeats when the new Data Provider file is received by ODI. All existing data is overwritten with the new data and it is again presented for consumption

Direct SQL data access

Direct SQL Data Access allows the consumer to access the data store in SQL. The consumer can fetch data directly from SQL views. Once the data is retrieved to the consumer’s own data store or staging location, the consumer is free to manipulate the data as needed. This is the preferred data retrieval method as there is no processing required by ODI and the consumer can be flexible in when the data is retrieved.

Basic steps for using Direct SQL

As an example, here are the basic steps for using direct SQL access:

• Provider delivers data to ODI
• ODI stores the data in SQL
• ODI notifies the interested Consumers via EES
• Consumer reads the notification
• Consumer uses ODISvcCred to obtain temporary SQL credentials
• Consumer uses the SQL credentials to access their private view, using the connection info provided with the credentials.
• Consumer stores the Provider data in their own staging area, waiting for all data updates to complete before updating their working data store

ODISvcCred

There is a change regarding the ODISvcCred message response that will be implemented with the new version, ODI vNext. The message contract has not changed but information previously found in the response may no longer be provided. The <ServerIPAddr> and <SQLInstcName> elements may no longer contain IP addresses or SQL instance names. The primary reason for the change is due to the use of SQL Availability Groups (AG), which will provide High Availability services for the ODI data.
If SQL AG is present for the environment in use, the <SQLInstcName> element will be returned in the response but will be empty. If SQL AG is not used, the instance name value will be returned.

   <ServerName>OdiProdSqlAgLis.odi.edn,54278</ServerName>     
   <ServerIPAddr></ServerIPAddr>      
   <SQLInstcName></SQLInstcName>  
   <RsStat>Success</RsStat>  

Comprehensive information about the ODISvcCred API can be found here on the EI&S Development portal.

OFI (Operational File Integration)

The process described above covers most uses of the ODI system and the data it brokers. One special feature of ODI is OFI, or Operational File Integration. Like Operational Data Integration, it serves to provide access to other products’ data without having to contact that provider directly, but allows the provider and consumer to share data in a format not ingestible by ODI. Using OFI, a provider can make data available in any file format it chooses and pass that data file to ODI with the intent of the ODI system then making it available to one or more consumers. In this manner, ODI becomes a pass-through for data it cannot easily handle via the SQL database.

There are a few additional features of OFI that make it useful in a wide range of applications. These features are contained in the *.ofi control file that must accompany the actual provider data file. First, the provider has the option of naming the consumers that are to receive the data file. The consumers must already be configured to receive the file, but this allows the provider some ability to customize the data for specific consumers. Second, the control file can contain a ticket ID that can be used by the provider and consumer to correlate the data as a response to some external request process not made via ODI. The ticket ID is revealed to the consumer as part of the EES event sent as notification of the file’s availability. In this manner, ODI is operating as a bulk response mechanism.

Please keep in mind that the consumer must understand how to read the file structure sent by the provider, and any changes made in that format create a maintenance event for the consumer developers. As a rule, any data that can be made to fit the regular ODI contracts should, and only specially formatted information should be handled via OFI.

Next Steps

Please review the Getting Started with ODI section of this portal to further review the ODI product and see how to start working with ODI.