Skip to main content
Guides
Spaces

Bidirectional Data transfer

Overview​

To design and implement a solution that can subscribe to events from different external products like Customer Relationship Management (CRM) systems and route them to different internal products like CDP and MSC. The events may be in different formats, and there may be multiple tenants for each product. We need an architecture that can handle events from multiple sources and tenants, transform them into a common format, and route them to the appropriate products based on their configurations.

To address these requirements, we need to design a system with several layers, including a connector layer for each CRM system, a message transformation layer, a message queue system, and a subscriber layer for multiple tenants and products. The connector layer will communicate with the CRM systems' APIs and translate the events into a common format. The message transformation layer will translate the events from each CRM system into a common format that the subscriber layer can consume. The message queue system will store and distribute the events to the subscriber layer.

The subscriber layer will handle events from multiple sources and tenants and use filtering and routing logic to ensure that each tenant and product only receives the events they are interested in. The subscriber layer will route the events to different product systems based on their configurations. Each product system will consume the events and process them according to their requirements.

The system needs to be scalable, flexible, and reliable, and it should provide filtering and routing logic to ensure that each tenant and product only receives the events they are interested in. It should also be able to handle large volumes of events and be able to handle failures gracefully.

Architecture​

The architecture for subscribing to events from multiple external systems and routing them to different internal products is designed with several layers that work together to achieve the desired functionality.

Bidirectional data transfer architecture layers diagram

**Connector Layer: **The first layer in the architecture is the Connector Layer, which is responsible for communicating with external systems' APIs and translating the events into a common format that can be consumed by the rest of the system. Each external system will have its own Connector Layer, which is designed to handle the specific requirements of that system.

  • Salesforce connector: Salesforce API, Apex, Salesforce Connect
  • Dynamics connector: Dynamics API, Dynamics SDK, Dynamics Connect

Message Queue System: The Second layer is the Message Queue System, which is responsible for storing the events and distributing them to the subscriber layer. This layer provides a scalable and fault-tolerant mechanism for storing and distributing the events to the rest of the system.

  • Messaging queue: GCP Pub/Sub

Subscriber Layer: The Third layer is the Subscriber Layer, which is responsible for handling events from multiple sources and tenants and routing them to the appropriate products based on their configurations. This layer uses filtering and routing logic to ensure that each tenant and product only receives the events they are interested in.

  • Event-driven architecture: GCP Cloud Run
  • Routing and filtering: GCP Pub/Sub

Message Transformation Layer: The fourth layer is the Message Transformation Layer, which is responsible for translating the events from each CRM system into a common format that the subscriber layer can consume. This layer ensures that the events are consistent across all the CRM systems and that they are compatible with the rest of the system.

Product Systems: The final layer is the Product Systems, which are responsible for consuming the events and processing them according to their specific requirements. Each product system will have its own configuration for the events it needs to consume, and the subscriber layer will route the events to the appropriate product systems based on their configurations.

  • CDP, MSC endpoints

Overall, this architecture is designed to be flexible and scalable, allowing the system to handle events from multiple external systems and route them to different products based on their configurations. The system provides filtering and routing logic to ensure that each tenant and product only receives the events they are interested in, and it can handle large volumes of events and failures gracefully.