Version: 17.07
Supported Since: 17.01
The SFTP Ingress Connector can be used to obtain messages from a remote SFTP server and inject those messages to the UltraESB engine. Since UltraESB uses generic SFTP protocol to communicate with the SFTP servers, it should be able to communicate with almost all the existing SFTP server implementations.
When you are obtaining a message from a SFTP server, UltraESB server first download the input file to it’s local file system and construct a message by attaching the downloaded local file as the payload of that message. Then user can access the input file content as the message payload and do required modifications.
In order to use the SFTP Ingress Connector, you must first select the SFTP Connector dependency from the connector list when you are creating an empty Ultra project. If you have already created a project, you can add the SFTP Connector dependency via Component Registry. From Tools menu, select Ultra Studio → Component Registry and from the Connectors list, select the SFTP Connector dependency. |
Processor |
The file picked form the SFTP location will be emitted from this out port as a Message |
On Exception |
The message will be emitted from this out port if the Ingress Connector fails to construct a message using the fetched file from the SFTP server |
* marked fields are mandatory
SFTP Host * |
Basic |
Host name/IP of the SFTP server that you are going to connect |
SFTP Remote Port * |
Basic |
Port of the SFTP server that you are going to connect to fetch the files. By default value of this will be set to 22 |
Username * |
Basic |
Username that you are going to use to connect to the SFTP server |
Password |
Basic |
Password for above user name if you are going to connect via password based authentication to the SFTP server |
Fetch Path * |
Basic |
Path of the SFTP server that you are going to fetch files. This path should be relative to the home directory of the above given user |
File Name Pattern * |
Basic |
Regex pattern to filter out the files from the above configured SFTP fetching location. All the files which are matched to this regex expression will be fetched from the above configured "Fetch Path" SFTP location |
Wait After Modification |
File Modifications |
This is a time period in seconds which will be used to decide whether to fetch a file or not based on its last modified time of the file. Framework will not fetch a given fill until current time exceeds, the summation of the last modified time of the file and this value. By default, this value is zero. |
Remove Original File |
File Modifications |
Specify whether to remove original file after fetching from the SFTP server. If this is set to false, input file won’t be removed by the framework automatically and that file will be picked up in every iteration of the polling schedule. By default, this value is true. |
Move Location After Process |
File Modifications |
File path to move input file in the SFTP server, after completing the processing successfully. This path will be considered only if it’s successfully processed without any errors |
Move Location After Failure |
File Modifications |
File path to move input file in the SFTP server, after completing the processing with failures. This path will be considered only if it’s couldn’t process the file in the SFTP server |
Move Timestamp Format |
File Modifications |
Set to attach the current timestamp in this format to the file name as a suffix, while moving files both in success and failure scenarios. If this is not set, original file name will be used as it’s while moving. |
Key File Path |
Key Authentication |
Path of the private key of the user which will be used to communicate with SFTP server via key based authentication. |
Key Password |
Key Authentication |
Password of the private key if it’s password protected. |
Host Key Check |
Key Authentication |
Set to enable the host key checking while creating the communication channel with SFTP server. If this is not enabled, host key checking will be skipped while SFTP processing. By default, this will be false. |
Scheduler Configuration |
Scheduling |
Bean reference of the scheduler configuration bean which should be declared as a resource in the project.xpml file. By default there is internal scheduler configuration within the framework which will be shared by all the polling connectors. If you need to configure higher level of concurrent processing threads which will fetch files from the SFTP server, you can configure your own thread pool configuration while declaring the parameters of the scheduler configuration bean as below.
In this configuration,
|
Polling Cron Expression |
Scheduling |
Cron expression for the SFTP file polling schedule. Cron expression should be a valid Quartz cron expression since the Framework is underneath using Quartz to extract the schedule from the cron expression. |
Polling Start Delay |
Scheduling |
Delay in milliseconds to start the polling schedule. Any iteration which comes within this time period from the startup time of the framework, won’t be considered as a valid file polling iteration. |
Polling Repeat Interval |
Scheduling |
Interval in milliseconds for the next iteration of the polling schedule. This will be considered if there isn’t a configured cron schedule. |
Polling Repeat Count |
Scheduling |
Number of iterations which should go through the polling schedule. If this is set to 1 which means only the first iteration of the polling schedule will be considered as a valid file polling iteration and all other iterations of the schedule will be ignored. By default, this value is set to -1 which means it will consider all the iterations of the polling schedule as a valid iteration. |
Concurrent Polling Count |
Scheduling |
Maximum number of concurrent threads which can be used to poll the configured SFTP server to fetch files. By default, this value is 1. |
Concurrent Execution Count |
Scheduling |
Maximum number of concurrent threads which can be used to process the fetched files from SFTP server. By default, this value is 4. |
Local Temp Path |
Advanced |
File location to be used as the temporary directory while processing files in the SFTP server. Input files will be downloaded to this location before injecting it to the engine as a new message. If this is not set, then temporary directory of the framework will be used as the temporary directory of the SFTP processing. |
Max Connections |
Advanced |
Maximum number of connections which will be cached within the SFTP IngressConnector for the configured SFTP server. By default, this value is 10. |
Timeout |
Advanced |
Timeout in millis, to be used while connecting to the remote SFTP server. If it couldn’t establish a connection with the SFTP server within this time period, it will be considered as a connection failure and notify to the user. By default, this value is 60000. |
In this use case scenario, there is a system which sends CSV files to its registered clients via SFTP protocol. Clients
will connect to the SFTP server and fetch the relevant files to them based on the file name suffix. As an example, if
the client’s registered name is Adroitlogic, then the file name will be ended as *_Adroitlogic.csv
.
Let’s say as a client of the this company, you want to integrate your system to fetch the CSV files from the SFTP server and convert the file content to XML and then send it to an existing web service.
Further, you can create your own scheduling configuration as a resource in the project.xpml. For this use case I’ll use default scheduling configuration within the framework. If you want to poll the files from the SFTP server at a higher concurrency level, then you have to create your own scheduling configuration by specifying the required number of threads for the polling thread pool. Check the example scheduler configuration in the parameters table.
To implement above use case, first let’s create our integration flow named “integrate-to-fetch-via-sftp” and add a SFTP IngressConnector to that. Then you need to configure your SFTP IngressConnector by configuring it’s Basic parameters as below.
.*_Adroitlogic.csv
will be used to filter-out files generated for my company - Company Name will be appended to the
file name by the source system
Then we should add the CSV to XML Converter processing element to transform the CSV content to XML before sending to the web service. This processing element should be able to find in the Processors → Transformation section.
As the final step, we should send this converted message to the service via HTTP protocol. For that, let’s add the NIO HTTP Sender egress connector to our configuration. In the NIO HTTP Sender egress connector, we have to configure the Destination Host, Port and Service Path from our internal web service information.
After filling out all the required parameters of above three components, we can add a Logger processing element to log the completion of the web service call and to mark the end of the integration flow, we should add the Successful Flow End processing element as below.
Now if you run the flow, and put an JSON file to with name ending with _Adroitlogic.csv
, to the
"$SFTP_USER_HOME/inventory-information/end-of-day" directory of the SFTP server, then that file content should be
transformed to XML and send to the internal web service URL via HTTP