Data Transformations

Overview

This page provides information on the data transformation between connectors and adapters by using templates

Problem Statement

Response from adapters is not something that connectors can always process directly. Instead, they need to do one of the following actions on the fields in response.

• ignore
• rename
• replace
• order
• merge
• flatten out

Example: Consider the following adapter response

{
  "name": "Homer J. Simpson",
  "occupation": "Security inspector",
  "street": "Evergreen Terrace",
  "number": "742",
  "city": "Springfield",
  "state": "Kentucky",
  "parents": {
    "father": {
      "name": "Abraham",
      "age": "80"
    },
    "mother": {
      "name": "Mona",
      "age": "70"
    }
  }
}

Which requires to be transformed like this for the connector

{
  "fullName": "Homer J. Simpson",
  "profession": "Security inspector",
  "address": {
    "street": "Evergreen Terrace",
    "number": "742",
    "city": "Springfield"
  },
  "mother": "Mona",
  "father": "Abraham"
}

We need a template like this to transform

{
  "fullName": "{{name}}",
  "profession": "{{occupation}}",
  "address": {
    "street": "{{street}}",
    "number": "{{number}}",
    "city": "{{city}}"
  },
  "mother": "{{parents.mother.name}}",
  "father": "{{parents.father.name}}"
}

Whenever there are new fields being introduced then it requires connectors to have some changes in code to work with them.

This requires us to look at no-code solutions like templates that can be configured and customizable by the customer

We are trying to look at mapping with a structure like this

{
    "mapping" : {
        "user": {
            "first_name": "firstName",
            "address_obj": "address",
            "address_obj": {
                "postal_code": "postalCode"
            },
            "order_history": "orderHistory",
            "order_history": {
                "order_item": "item"
            }
        },
        "order": {
            "identifier": "id"
        }
    },
    "filter": {
        "user": [ "first_name, address_obj, order_history" ]
    }
}

Libraries with pros and cons

Library	Features	Pros and Cons
JDT https://github.com/microsoft/json-document-transformsWiki https://github.com/Microsoft/json-document-transforms/wiki Schema http://json.schemastore.org/jdt	It uses JSON Path to do transitions and one needs to aware of it to create templateshttps://goessner.net/articles/JsonPath/index.html It supports the following - Rename - Replace - Remove - Merge	Pros 1. This is the best suited for defining global level mappings 2. We need to write template for fields needs to be modified and rest can bring as is Cons 1. Flattening is not supported 2. Less features and flexibility 3. Template becomes too verbose
Dotliquid https://github.com/dotliquid/dotliquid Wiki https://github.com/dotliquid/dotliquid/wiki	It uses liquid templating language and used by many It supports many controls, tags, filters. Refer https://shopify.github.io/liquid/	Pros 1. This is the best suited for more customization of response and HTML. Not only for JSON 2. More inbuilt methods Cons 1. For global level mappings, templates can be too verbose 2. No way to write custom methods inside templates
Just.net https://github.com/WorkMaze/JUST.net	It uses JSON path by default and allows us to define custom language	Pros 1. This is the best suited for more customization of response 2. More inbuilt methods Cons 1. For global level mappings, templates can be too verbose 2. No way to write custom methods inside templates
JSLT https://github.com/schibsted/jslt	It is a complete query and transformation language for JSON. The language design is inspired by jq, XPath, and XQuery.	Pros 1. This is best suited for both global level and endpoint level templates 2. Language is designed for JSON 3. Turing complete Cons

Architecture

1. We are building JSLT service to do the following
   1. Return transformed json by taking JSON payload and JSLT template
   2. Validate JSLT template
2. We are building JSLT client with HTTP message handler implementation to do the following
   1. Get response from adapters
   2. Get a template based on configuration (customer id) and integration (Adapter id)
   3. Invoke JSLT service with template and response payload
   4. Override response payload from the adapter with JSLT Service response payload

Data Transformations using message handler

Attachments