sftp ingress connector

SFTP Ingress Connector

Version: 17.07

Supported Since: 17.01

What is SFTP Ingress Connector?

The SFTP Ingress Connector can be used to obtain messages from a remote SFTP server and inject those messages to the UltraESB engine. Since UltraESB uses generic SFTP protocol to communicate with the SFTP servers, it should be able to communicate with almost all the existing SFTP server implementations.

When you are obtaining a message from a SFTP server, UltraESB server first download the input file to it’s local file system and construct a message by attaching the downloaded local file as the payload of that message. Then user can access the input file content as the message payload and do required modifications.

In order to use the SFTP Ingress Connector, you must first select the SFTP Connector dependency from the connector list when you are creating an empty Ultra project. If you have already created a project, you can add the SFTP Connector dependency via Component Registry. From Tools menu, select Ultra Studio → Component Registry and from the Connectors list, select the SFTP Connector dependency.
sftp ingress connector 4

Out Ports

Processor

The file picked form the SFTP location will be emitted from this out port as a Message

On Exception

The message will be emitted from this out port if the Ingress Connector fails to construct a message using the fetched file from the SFTP server

Parameters

* marked fields are mandatory

SFTP Host *

Basic

Host name/IP of the SFTP server that you are going to connect

SFTP Remote Port *

Basic

Port of the SFTP server that you are going to connect to fetch the files. By default value of this will be set to 22

Username *

Basic

Username that you are going to use to connect to the SFTP server

Password

Basic

Password for above user name if you are going to connect via password based authentication to the SFTP server

Fetch Path *

Basic

Path of the SFTP server that you are going to fetch files. This path should be relative to the home directory of the above given user

File Name Pattern *

Basic

Regex pattern to filter out the files from the above configured SFTP fetching location. All the files which are matched to this regex expression will be fetched from the above configured "Fetch Path" SFTP location

Wait After Modification

File Modifications

This is a time period in seconds which will be used to decide whether to fetch a file or not based on its last modified time of the file. Framework will not fetch a given fill until current time exceeds, the summation of the last modified time of the file and this value. By default, this value is zero.

Remove Original File

File Modifications

Specify whether to remove the original file after fetching from the SFTP server. If this is set to false, input file won’t be removed by the framework automatically and that file will be picked up in every iteration of the polling schedule. By default, this value is true.

If this is set to true and the Move Location After Process/Move Location After Failure parameters are not provided, file will be deleted after the processing. Otherwise it will be moved to the provided location based on the processing status.

Move Location After Process

File Modifications

File path to move input file in the SFTP server, after completing the processing successfully. This path will be considered only if the file was successfully processed without any errors.

Only if the Remove Original File parameter is set to true, the file will be moved into this location. Otherwise, only a copy of the file will be created in this location, leaving the original file intact.

Move Location After Failure

File Modifications

File path to move input file in the SFTP server, after completing the processing with failures. This path will be considered only if a failure was encountered while processing the file.

Only if the Remove Original File parameter is set to true, the file will be moved into this location. Otherwise, only a copy of the file will be created in this location, leaving the original file intact.

Move Timestamp Format

File Modifications

Set to attach the current timestamp in this format to the file name as a suffix, while moving files both in success and failure scenarios. If this is not set, original file name will be used as it’s while moving.

Key File Path

Key Authentication

Path of the private key of the user which will be used to communicate with SFTP server via key based authentication.

Key Password

Key Authentication

Password of the private key if it’s password protected.

Host Key Check

Key Authentication

Set to enable the host key checking while creating the communication channel with SFTP server. If this is not enabled, host key checking will be skipped while SFTP processing. By default, this will be false.

Scheduler Configuration

Scheduling

Bean reference of the scheduler configuration bean which should be declared as a resource in the project.xpml file. By default there is internal scheduler configuration within the framework which will be shared by all the polling connectors. If you need to configure higher level of concurrent processing threads which will fetch files from the SFTP server, you can configure your own thread pool configuration while declaring the parameters of the scheduler configuration bean as below.

<x:resources>
    <x:resource id="custom-scheduler-config">
        <bean id="schedulerConfigBean" class="org.adroitlogic.x.base.trp.SchedulerConfig">
            <constructor-arg name="name" value="my-custom-scheduler"/>
            <property name="schedulerThreadCount" value="4"/>
            <property name="pollingThreadCoreSize" value="4"/>
            <property name="pollingThreadMaxSize" value="10"/>
            <property name="pollingQueueSize" value="25"/>
            <property name="pollingKeepAliveTime" value="5000"/>
        </bean>
    </x:resource>
</x:resources>

In this configuration,

  • schedulerThreadCount - is the number of threads which will be used to schedule the number of threads to be used to handle scheduled polling tasks in this SFTP Ingress Connector. Generally this value should be smaller value since polling threads don’t do heavy task in this, rather it’s just initiating the polling task for the scheduling iteration and handover the file fetching and processing task to a separate executor service. This executor service can be configured by using next four parameters of above scheduling configuration bean.

  • pollingThreadCoreSize - is the core size of the SFTP file polling thread pool

  • pollingThreadMaxSize - is the maximum number of threads of the SFTP file polling thread pool

  • pollingQueueSize - is the queue size of the SFTP file polling thread pool

  • pollingKeepAliveTime - is the keep alive time of the SFTP file polling thread pool

Polling Cron Expression

Scheduling

Cron expression for the SFTP file polling schedule. Cron expression should be a valid Quartz cron expression since the Framework is underneath using Quartz to extract the schedule from the cron expression.

Polling Start Delay

Scheduling

Delay in milliseconds to start the polling schedule. Any iteration which comes within this time period from the startup time of the framework, won’t be considered as a valid file polling iteration.

Polling Repeat Interval

Scheduling

Interval in milliseconds for the next iteration of the polling schedule. This will be considered if there isn’t a configured cron schedule.

Polling Repeat Count

Scheduling

Number of iterations which should go through the polling schedule. If this is set to 1 which means only the first iteration of the polling schedule will be considered as a valid file polling iteration and all other iterations of the schedule will be ignored. By default, this value is set to -1 which means it will consider all the iterations of the polling schedule as a valid iteration.

Concurrent Polling Count

Scheduling

Maximum number of concurrent threads which can be used to poll the configured SFTP server to fetch files. By default, this value is 1.

Concurrent Execution Count

Scheduling

Maximum number of concurrent threads which can be used to process the fetched files from SFTP server. By default, this value is 4.

Local Temp Path

Advanced

File location to be used as the temporary directory while processing files in the SFTP server. Input files will be downloaded to this location before injecting it to the engine as a new message. If this is not set, then temporary directory of the framework will be used as the temporary directory of the SFTP processing.

Max Connections

Advanced

Maximum number of connections which will be cached within the SFTP IngressConnector for the configured SFTP server. By default, this value is 10.

Timeout

Advanced

Timeout in millis, to be used while connecting to the remote SFTP server. If it couldn’t establish a connection with the SFTP server within this time period, it will be considered as a connection failure and notify to the user. By default, this value is 60000.

Sample Use Case

In this use case scenario, there is a system which sends CSV files to its registered clients via SFTP protocol. Clients will connect to the SFTP server and fetch the relevant files to them based on the file name suffix. As an example, if the client’s registered name is Adroitlogic, then the file name will be ended as *_Adroitlogic.csv.

Let’s say as a client of the this company, you want to integrate your system to fetch the CSV files from the SFTP server and convert the file content to XML and then send it to an existing web service.

Further, you can create your own scheduling configuration as a resource in the project.xpml. For this use case I’ll use default scheduling configuration within the framework. If you want to poll the files from the SFTP server at a higher concurrency level, then you have to create your own scheduling configuration by specifying the required number of threads for the polling thread pool. Check the example scheduler configuration in the parameters table.

Implementation

To implement above use case, first let’s create our integration flow named “integrate-to-fetch-via-sftp” and add a SFTP IngressConnector to that. Then you need to configure your SFTP IngressConnector by configuring it’s Basic parameters as below.

sftp ingress connector 1

.*_Adroitlogic.csv will be used to filter-out files generated for my company - Company Name will be appended to the file name by the source system

Then we should add the CSV to XML Converter processing element to transform the CSV content to XML before sending to the web service. This processing element should be able to find in the Processors → Transformation section.

As the final step, we should send this converted message to the service via HTTP protocol. For that, let’s add the NIO HTTP Sender egress connector to our configuration. In the NIO HTTP Sender egress connector, we have to configure the Destination Host, Port and Service Path from our internal web service information.

After filling out all the required parameters of above three components, we can add a Logger processing element to log the completion of the web service call and to mark the end of the integration flow, we should add the Successful Flow End processing element as below.

sftp ingress connector 2

Now if you run the flow, and put an JSON file to with name ending with _Adroitlogic.csv, to the "$SFTP_USER_HOME/inventory-information/end-of-day" directory of the SFTP server, then that file content should be transformed to XML and send to the internal web service URL via HTTP

NOTE: In order to debug any connectivity issues (e.g. troubleshoot authentication failures), consider setting TRACE log level for the org.adroitlogic.x.transport.sftp package. This will display a sftp -v compatible trace during connection and disconnection, indicating where the problem occurs.

In this topic
In this topic
(c) Copyright 2016-2017 AdroitLogic. All rights reserved.
Terms of Use
Contact Us