Data Flow from Source Systems to ICM

As part of Contextual Fusion Hub (CFH), adapters are created to connect to various source systems. These adapters connect to source systems to fetch data, and are configured on the network from where the source systems are accessible. For Edge devices, these adapters are deployed on docker and for cloud these are deployed in Azure. For on-premise, these adapters are deployed on any VM or Edge.

The following figure depicts the data flow from source systems to ICM.

Each docker will run seperately and will have its own schedule to trigger the respective scripts.
Azure data factory is used for scheduling.
Decompressing the zip files is done through ADF pipeline with source as input. Multiple sources can be configured in one pipeline.
Azure functions are triggered through each pipeline. Each Azure function triggers the underlying python scripts.