JH Wires
Introduction
The goal of this pipeline is to make the JH Wires dataset available in Google Big Query where it can be queried using the SQL interface for business purposes and to drive downstream business processes.
Pipelines
The JH Wires pipeline is made of of two different components.
The Wire-Search-Service receives history data for incoming and outgoing wires via the ‘WireDataUpdated’ event from the Incoming and Outgoing Wire Services. The history data includes details such as WireId, UniversalInstitutionId, CreationDate, ApplicationId, CssCorrelationId, InstitutionId, and other wire-specific state data with both old and new values, such as process step, state, and change user. This data is pushed to Data Hub on each raised ‘WireDataUpdated’ event. This data is written to big query in Data Hub via Rest API service and not Kafka PUSH. The configuration details include setting up ProjectId, DataSetId, and TableId in AppSettings on wires that match the configuration on the Data Hub project. The appropriate read-write permissions were granted to the service account configured for Wire-Search-Service that communicates with Data Hub.
The JH Wires ISO pipeline utilizes BigQuery External Connections in the Data Hub project, to the Publisher Spanner database in their project. Federated Queries are used with these External Connections to pull data from the Spanner Database into the BigQuery dataset / tables.
Topics in this section
- base_wires_fct_cdd (ISO)
- message_sanction_status_fct_cdd (ISO)
- messages_fct_cdd (ISO)
- pacs_004_messages_fct_cdd (ISO)
- pacs_008_messages_fct_cdd (ISO)
- pacs_009_messages_fct_cdd (ISO)
- wire_step_status_fct_cdd (ISO)
- wthist_fct_cdd