Integration Layer Service (ILS)

Introduction

The Integration Layer Service receives data objects from one or several incoming flows, applies some business logic (such as a merge or split), validates them against a supplied schema, and provides the resulting valid objects as input to other flows. ILS can temporarily store these objects. Objects can be posted to the ILS through a REST-API, and be retrieved the same way.

API Reference Implementation

Technologies used

MongoDB: MongoDB is used as the ILS’s storage solution.

How it works

Use case: Merging objects

A core functionality is merging of incomplete objects from separate sources until all required data has been added. For this purpose, all incoming POST requests must supply a common identifier (cid) by which incoming objects can be matched against each other. Each time a new object comes in, the ILS first looks up its database to check whether another object with a matching cid is already stored. If one is found, the new data is merged into the existing object. If not, the incoming object is saved as-is.

In either case, the resulting object is then validated against a supplied definition, which includes a list of required fields. A valid object will be marked as such, and can then be retrieved by another component. If an object is invalid, it cannot be retrieved, and will instead be stored until the necessary data has been merged into it and it passes validation.

Use case: Splitting objects

ILS comes into play when the user would like to split an object into smaller or other objects containing properties from the main object. Let’s say that the user want to split the object containing employee’s firstName, lastName, salutation, organization, email, street and streetNumber into two separate objects. The first one should have the properties firstName, lastName and salutation ant the second one should contain the rest. In this case the user must provide splitSchema array containing two objects and each of them must have the properties meta and payload. Meta has the special property splitKey which servers as an identifier and payload consists of the fields which the new splitted object must contain. After successful splitting each new object should hold the splitKey and payload properties.

On the other hand the user would like to GET a new/splitted object. In such a case an ilaId and a splitKey must be provided. Then ILS will return an array of all objects which have the same splitKey.

Use case: Validating objects

ILS is able to validate an object against a certain schema. This means that the incoming object could be validated against a custom schema, which the user can provide in schema object or the object could be validated against a schema which is already stored in Meta Data Repository as well. Then the user should provide an IAM token, domainId and schemaUri. The last two properties could be requested from Meta Data Repository.

Integration Layer Service API

The ILS offers a REST API through which chunks can be stored, splitted, validated and retrieved. To interact with this API, the user must supply a valid bearer token generated by the Identity Management.

List of supported Methods and Routes

These operations ought to be conducted via the ILA, but can also be targeted manually for testing and development purposes.

For POST /chunks, the body format is expected to match this format:

{
  "ilaId": "string",
  "cid": "string",
  "def": {},
  "payload": {}
}

• ilaId: Identifies which combination of flows this object belongs to. Must match among all connected flows.
• cid: A common identifier designating which fields are used to match objects to one another. Must be a key within the supplied payload
• def: The definition against which objects are validated. Currently expected to be a JSON schema.
• payload: The actual data object, in JSON format.

For POST /chunks/split, the body format should have the following format:

{
  "ilaId": "string",
  "payload": {},
  "splitSchema": [
    {
      "meta": {
        "splitKey": "string"
      },
      "payload": {}
    }
  ]
}

• ilaId: Identifies which combination of flows this object belongs to. Must match among all connected flows.
• payload: The actual data object, in JSON format.
• splitSchema: A schema object which could contain more then one schema. The main purpose is each schema has a meta object with an unique splitKey. This identifies the splitting model. In payload is the actual schema with all properties.

For POST /chunks/validate, the body format should have the following format:

{
  "ilaId": "string",
  "token": "string",
  "cid": "string",
  "def": {
    "domainId": "string",
    "schemaUri": "string"
  },
  "payload": {}
}

• ilaId: Identifies which combination of flows this object belongs to. Must match among all connected flows.
• token: An IAM Bearer token
• cid: A common identifier designating which fields are used to match objects to one another. Must be a key within the supplied payload
• def: The definition against which objects are validated. Currently expected to be a JSON schema.
• domainId - A domainId from Meta Data Repository
• schemaUri - A schemaUri for a certain schema from Meta Data Repository
• payload: The actual data object, in JSON format.

To GET /chunks/${ilaId}, an ilaId must be supplied. This will return all objects marked as valid that have been saved with this ilaId.

To GET /chunks/${ilaId}?key=${splitKey} the user should provide the ilaId and key as a parameter for fetching objects which are splitted by a splitSchema with the certain splitKey.

Stored objects are endowed with a Time To Live of one hour, which is refreshed every time new data is merged into them.

Integration Layer Adapter (ILA)

The ILA is a generic component used to allow flows to communicate with the ILS. In posting mode, it automatically passes on any data objects received by other flow components, and endows them with the metadata listed above. In polling mode, it will automatically fetch all valid combined objects and pass them on to other components just like any other flow component. For further information about the ILA, see its GitHub Repository

Interaction with other Services

The Integration Layer Service interacts with this Service:

• Meta Data Repository: If the user wants to validate an object against a schema from Meta Data Repository, this schema must be fetched. For this purpose a bearertoken, domainId and schemaUri must be provided.