SFTP File Mediation

Version: 17.07

Supported Since: 17.01

Use case description

I want to mediate large JSON files via SFTP from one system to another after modifying the file name based on the value received as field in the input file. File name should be modified by attaching the received client ID as the prefix of the original file name by using the underscore as the separator. Client ID will be available at the pre-defined JSON path of the received input file content.

Proposed Solution

As the solution for this, we have decided to mediate files thorough an ESB as shown in the below diagram.

ESB will fetch file from the source SFTP server and extract the client ID by evaluating a pre-configured JSON path over the received file content. This client ID will append as the prefix to the original file name. So the final file name would be like <received_client_id>_<original_file_name>. With this file name modification, received file will be uploaded to the target SFTP server using the SFTP sender.

UltraStudio Configuration

UltraESB-X Configuration

Implementation of the Solution

Prerequisite

In order to implement above use case you must first select following dependencies when you are creating a new Ultra project.

SFTP Connector from the connector list
JSON Processors from the processor list
Message Logger from the processor list

If you have already created a project, you can add above dependencies via Component Registry. From Tools menu, select Ultra Studio → Component Registry and from the Connectors list and Processors list, select above dependencies.

Implementation

To implement above use case, first let’s create our integration flow named “sftp-file-mediation-flow”. Then let’s add required components by going through following steps in order.

Add an SFTP Ingress Connector from the Connectors → Ingress Connectors list, to fetch the files from the source SFTP server. SFTP Ingress Connector basic parameter should be filled as shown below to fetch the JSON files from the source system.
Add a JSON Path Extractor processing element from the Processors → JSON list to extract out the client ID from the input message content. As per the following structure of the message JSON path for the client ID will be "$.Client-ID". Let’s save the extracted client ID value as the variable with name "CLIENT_ID" by configuring the "Variable Name" parameter of the processing element.

{
  "First name": "Brian",
    "Last name": "Novik",
    "Client-ID" : "DC0-453291",
    "Organization": {
      "id": "100123H",
      "name" : "ABC Company"
    },
    "Orders": [
      ....
      orders
      ....
    ]
  ....
  rest of the content of the input JSON message
  ....
}

Add a Logger Processor from the Processors → Generic to log the received file name to make sure that we have information about the processed files from the SFTP Ingress Connector. Let’s configure "Log Template" as "Received file with name : @{message.header.ultra.file.name} for client ID : @{message.var.CLIENT_ID}" to log both received file name and extracted client ID value.
Add an SFTP Egress Connector element from the Connectors → Egress Connectors to the integration flow to send the file to the target SFTP server as per the requirement use case. SFTP EgressConnector can be configured by filling the Basic properties as shown below. In-order to modify the file name as described in the use case, "File Name" property should be configured as @{message.var.CLIENT_ID}_@{message.header.ultra.file.name} which contains the value of the variable with name CLIENT_ID and the original file name of the received input file.

The completed integration flow should look like below.

Configuration for each element is as below. The numbering corresponds to the numbers shown in above diagram.

Design View

Text View

.

1. SFTP Ingress Connector

If you are using key based authentication for SFTP connectivity, then you have to configure the key based authentication related configurations in the Key Authentication tab

2. JSON Path Extractor

3. Logger Processor

4. SFTP Egress Connector

.

1. SFTP Ingress Connector

SFTP Host

centosvm

SFTP Remote Port

22

Username

asankha

Password

mypassword

Fetch Path

input-x

File Name Pattern

If you are using key based authentication for SFTP connectivity, then you have to configure the key based authentication related configurations in the Key Authentication tab

2. JSON Path Extractor

Variable Name

CLIENT_ID

JSON Path

$.Client-ID

3. Logger Processor

Log Template

Received file with name \: @{message.headers.ultra.file.name} for client ID \: @{variable.CLIENT_ID}

Log Level

INFO

4. SFTP Egress Connector

Host Name

centosvm

Port

22

File Path

output-x

File Name

@{variable.CLIENT_ID}_@{message.headers.ultra.file.name}

Append Mode

false

User Name

asankha

Password

mypassword

.

Now you can run the Ultra Project and check the functionality of the integration flow. Create an UltraESB Server run configuration and start it.

Property Configuration

When running the sample in the UltraESB-X distribution, you need to override the following properties in-order for the sample to work. The properties file is located at $ULTRA_HOME/conf/projects/sftp-file-mediation/default.properties

Refer to Managing Project Properties documentation on how to override properties.

sftp-file-mediation-flow.sftp-ingress-connector.host

Hostname/ IP of the source SFTP sever (Default value is centosvm)

sftp-file-mediation-flow.sftp-ingress-connector.port

Port of the source SFTP sever (Default value is 22)

sftp-file-mediation-flow.sftp-ingress-connector.username

Username for the source SFTP sever (Default value is asankha)

sftp-file-mediation-flow.sftp-ingress-connector.password

Password for the source SFTP sever (Default value is mypassword)

sftp-file-mediation-flow.sftp-ingress-connector.fetchPath

File path fetch files from the source SFTP sever (Default value is input-x)

sftp-file-mediation-flow.sftp-egress-connector.host

Hostname/ IP of the destination SFTP sever (Default value is centosvm)

sftp-file-mediation-flow.sftp-egress-connector.port

Port of the destination SFTP sever (Default value is 22)

sftp-file-mediation-flow.sftp-egress-connector.username

Username for the destination SFTP sever (Default value is asankha)

sftp-file-mediation-flow.sftp-egress-connector.password

Password for the destination SFTP sever (Default value is mypassword)

sftp-file-mediation-flow.sftp-egress-connector.filePath

Path to upload file in the destination SFTP sever (Default value is output-x)

After that navigate to $ULTRA_HOME/bin directory. Next you can run the UltraESB-X distribution with following command to start the engine with this sample project deployed.

./ultraesbx.sh -sample sftp-file-mediation

Testing the Integration Project

Now you can run the integration flow and check the SFTP file mediating working as expected. If you put a input file with .json file name extension to the configured SFTP source location, your integration flow should fetch the file and upload it to the target SFTP server with the required file name modification.

Terms of Use