file ingress connector

File Ingress Connector

Version: 17.01

Supported Since: 17.01

What is File Ingress Connector?

The File Ingress Connector can be used to obtain messages from a local file system and inject those messages to the UltraESB-X engine. The connector has tested on Linux, Windows and OS-X environments and it is supposed to be able to operate in almost all the existing file systems.

When receiving a file to the UltraESB-X through the local file transport, the file will be set as the payload of a message and that message will be injected to the engine. Therefore users should be able to access the received file by accessing the message payload within the configured message flow and do required modifications.

The sample use cases below depict an integration flow with File ingress connector.

Sample Use Case

Use case description

In this use case scenario, there is a system that picks files from a local directory and send them to appropriate external web services via HTTP protocol. The external web service will be selected based upon the extension of the file name. This filtering part will be handled inside a custom processing element and it will forward the message to appropriate external web service.

For an example the files put into the directory with the extension .xml should be sent out to an external web service that processes XML documents while files put into the directory with the extension .csv should be sent out to the external web service that processes CSV document.

Prerequisite

In order to implement above described use case user should first add the following dependencies while creating the Ultra Project.

  • File Connector and HTTP NIO dependencies from the connector list.

  • Message Transformation and Flow Control dependencies from the processor list.

If you have already created a project, you can add above dependencies via Component Registry. From Tools menu, select Ultra Studio → Component Registry and from the Connectors list and Processors list, select above dependencies

After that, optionally you can create your own scheduling configuration as a resource in the project.xpml or else wise the default scheduling configuration of the framework will be used instead. If the requirement is to poll the files from the local file system at a higher concurrency level, then it will be required to create an appropriate scheduling configuration manually by specifying the required number of threads for the polling thread pool. Check the example scheduler configuration in the parameters table.

Implementation

To implement above use case first it is required to create a new integration flow and add a File IngressConnector to that. After that it is required to provide the Basic parameters to the File IngressConnector and configure it.

config file ingress connector

User should define the directory path to be used by the File connector to pick the files for the File Path input field, that is /tmp/files in this example. Here the user will only expect files with the extension .xml or .csv, therefore in the file ingress connector it can be configured inorder to pick files only with one of those two extensions. It can be achieved by specifying the File Name Pattern property to be .*\.(xml|csv).

After filling out the above mentioned parameters and other required parameters, this configuration can be saved by clicking the Save button at the bottom of the configuration pane of the File IngressConnector.

Then it is required to add the custom processing element that will do the filtering of input files by examining the file’s extension before sending to the appropriate web service.

As the final step, it is required to send the message with file payload to the appropriate web service via HTTP protocol using the properly configured NIO HTTP egress connectors.

After filling out all the required parameters of above three components, inorder to log the response received from the external web service it is required to add a Logger Processing element and finally to mark the end of the integration flow, user should add the "Successful Flow End" processing element as below.

flow file ingress connector

Parameters

* marked fields are mandatory

File Path *

Basic

Absolute file path to the base directory that the file ingress connector is supposed to pick files

File Name Pattern

Basic

Regex pattern to filter out the files from the above configured file fetching location. All the files which are matched to this regex expression will be fetched from the above configured File Path file location.

File Path Pattern

Basic

Other than the absolute location provided as the File Path property user can alternatively define a regex pattern for this property. The file paths defined by this property will be relative to the provided absolute file path. When the File Name Pattern property is provided it will get the priority over this property and this will be ignored for the file fetching logic of the file ingress connector.

Wait After Modification

Advanced

This is a time period in seconds which will be used to decide whether to fetch a file or not based on it’s last modified time of the file. Framework will not fetch a given fill until current time exceeds, the summation of the last modified time of the file and this value. By default, this value is zero.

Remove Original File

Advanced

Specify whether to remove original file after fetching it for processing. If this is set to false, input file won’t be removed by the framework automatically and that file will be picked up in every iteration of the polling schedule. By default, this value is true.

Move After Process

Advanced

File path to move input file in the file system, after completing the processing successfully. This path will be considered only if it’s successfully processed without any errors.

Move After Failure

Advanced

File path to move the fetched input file in the file system, after completing the processing with failures. This path will be considered only if it’s couldn’t process the file.

Move Timestamp Format

Advanced

Set to attach the current timestamp in this format to the file name as a suffix, while moving files both in success and failure scenarios. If this is not set, original file name will be used as it’s while moving.

Response Timeout

Advanced

Timeout in millis, to be used until receiving the response. If it didn’t receive the success response within this time period, it will be considered as a failure and notify to the user. By default, this value is 60000.

Scheduler Configuration

Scheduling

Bean reference of the scheduler configuration bean which should be declared as a resource in the project.xpml file. By default there is internal scheduler configuration within the framework which will be shared by all the polling connectors. If you need to configure higher level of concurrent processing threads which will fetch the files from the SFTP server, you can configure your own thread pool configuration while declaring the parameters of the scheduler configuration bean as below.

<x:resources>
    <x:resource id="custom-scheduler-config">
        <bean id="schedulerConfigBean" class="org.adroitlogic.x.base.trp.SchedulerConfig">
            <constructor-arg name="name" value="my-custom-scheduler"/>
            <property name="schedulerThreadCount" value="4"/>
            <property name="pollingThreadCoreSize" value="4"/>
            <property name="pollingThreadMaxSize" value="10"/>
            <property name="pollingQueueSize" value="25"/>
            <property name="pollingKeepAliveTime" value="5000"/>
        </bean>
    </x:resource>
</x:resources>

In this configuration, - schedulerThreadCount - is the number of threads which will be used to schedule the number of threads to be used to handle scheduled polling tasks in this SFTP Ingress Connector. Generally this value should be smaller value since polling threads don’t do heavy task in this, rather it’s just initiating the polling task for the scheduling iteration and handover the file fetching and processing task to a separate executor service. This executor service can be configured by using next four parameters of above scheduling configuration bean. - pollingThreadCoreSize - is the core size of the SFTP file fetching thread pool - pollingThreadMaxSize - is the maximum number of threads of the SFTP file fetching thread pool - pollingQueueSize - is the queue size of the SFTP file fetching thread pool - pollingKeepAliveTime - is the keep alive time of the SFTP file fetching thread pool

Polling Cron Expression

Scheduling

Cron expression for the file polling schedule. Cron expression should be a valid Quartz cron expression since the Framework is underneath using Quartz to extract the schedule from the cron expression.

Polling Start Delay

Scheduling

Delay in milli seconds to start the polling schedule. Any iteration which comes within this time period from the startup time of the framework, won’t be considered as a valid file polling iteration.

Polling Repeat Interval

Scheduling

Interval in milli seconds for the next iteration of the polling schedule. This will be considered if there isn’t a configured cron schedule.

Polling Repeat Count

Scheduling

Number of iterations which should go through the polling schedule. If this is set to 1 which means only the first iteration of the polling schedule will be considered as a valid file polling iteration and all other iterations of the schedule will be ignored. By default, this value is set to -1 which means it will consider all the iterations of the polling schedule as a valid iteration.

Concurrent Polling Count *

Scheduling

Maximum number of concurrent threads which can be used to poll the configured SFTP path to fetch files from the SFTP server. By default, this value is 1.

Concurrent Execution Limit *

Scheduling

Maximum number of concurrent threads which can be used process the fetched files from the SFTP server. By default, this value is 4.
In this topic
In this topic
(c) Copyright 2016-2017 AdroitLogic. All rights reserved.
Terms of Use
Contact Us