file ingress connector

File Ingress Connector

Version: 17.07

Supported Since: 17.01

What is File Ingress Connector?

The File Ingress Connector can be used to obtain messages from a local file system and inject those messages to the UltraESB-X engine. The connector has tested on Linux, Windows and OS-X environments and it is supposed to be able to operate in almost all the existing file systems.

When receiving a file to the UltraESB-X through the local file transport, the file will be set as the payload of a message and that message will be injected to the engine. Therefore users should be able to access the received file by accessing the message payload within the configured message flow and do required modifications.

In order to use the File Ingress Connector, you must first select the Local File Connector dependency from the connector list when you are creating an empty Ultra project. If you have already created a project, you can add the Local File Connector dependency via Component Registry. From Tools menu, select Ultra Studio → Component Registry and from the Connectors list, select the Local File Connector dependency.
file ingress connector 1

Out Ports

Processor

Message will be emitted from this out port with the content of the file as the payload of the message

On Exception

The message will be emitted from this out port, if an error occurred while creating the payload from the file.

Parameters

* marked fields are mandatory

File Path *

Basic

Absolute file path to the base directory that the file ingress connector is supposed to pick files

File Name Pattern

Basic

Regex pattern to filter out the files from the above configured file fetching location. All the files which are matched to this regex expression will be fetched from the above configured File Path file location.

File Path Pattern

Basic

Other than the absolute location provided as the File Path property, user can alternatively define a regex pattern for this property. The file paths defined by this property will be relative to the provided absolute file path. When the File Name Pattern property is provided it will get the priority over this property and this will be ignored for the file fetching logic of the file ingress connector. This can be used in cases where you want to recursively fetch files within subdirectories of your root File Path - which File Name Pattern does not facilitate. Note that this should also qualify as a valid file path (e.g. the regex cannot include | characters when on Windows)

Wait After Modification

File Modifications

This is a time period in seconds which will be used to decide whether to fetch a file or not based on its last modified time of the file. Framework will not fetch a given fill until current time exceeds, the summation of the last modified time of the file and this value. By default, this value is zero.

Remove Original File

File Modifications

Specify whether to remove the original file after fetching it for processing. If this is set to false, input file won’t be removed by the framework automatically and that file will be picked up in every iteration of the polling schedule. By default, this value is true.

If this is set to true and the Move After Process/Move After Failure parameters are not provided, file will be deleted after the processing. Otherwise it will be moved to the provided location based on the processing status.

Move After Process

File Modifications

File path to move input file in the file system, after completing the processing successfully. This path will be considered only if the file was successfully processed without any errors. Unless this value is an absolute path, it will be computed relative to the directory from which the file was picked.

Only if the Remove Original File parameter is set to true, the file will be moved into this location. Otherwise, only a copy of the file will be created in this location, leaving the original file intact.

Move After Failure

File Modifications

File path to move the fetched input file in the file system, after completing the processing with failures. This path will be considered only if a failure was encountered while processing the file. Unless this value is an absolute path, it will be computed relative to the directory from which the file was picked.

Only if the Remove Original File parameter is set to true, the file will be moved into this location. Otherwise, only a copy of the file will be created in this location, leaving the original file intact.

Move Timestamp Format

File Modifications

Set to attach the current timestamp in this format to the file name as a suffix, while moving files both in success and failure scenarios. If this is not set, original file name will be used as it is while moving.

Scheduler Configuration

Scheduling

Bean reference of the scheduler configuration bean which should be declared as a resource in the project.xpml file. By default there is internal scheduler configuration within the framework which will be shared by all the polling connectors. If you need to configure higher level of concurrent processing threads which will fetch the files from local file system, you can configure your own thread pool configuration while declaring the parameters of the scheduler configuration bean as below.

<x:resources>
    <x:resource id="custom-scheduler-config">
        <bean id="schedulerConfigBean" class="org.adroitlogic.x.base.trp.SchedulerConfig">
            <constructor-arg name="name" value="my-custom-scheduler"/>
            <property name="schedulerThreadCount" value="4"/>
            <property name="pollingThreadCoreSize" value="4"/>
            <property name="pollingThreadMaxSize" value="10"/>
            <property name="pollingQueueSize" value="25"/>
            <property name="pollingKeepAliveTime" value="5000"/>
        </bean>
    </x:resource>
</x:resources>

In this configuration,

  • schedulerThreadCount - is the number of threads which will be used to schedule the number of threads to be used to handle scheduled polling tasks in this File Ingress Connector. Generally this value should be a smaller value since polling threads don’t do heavy task in this, rather it’s just initiating the polling task for the scheduling iteration and handover the file fetching and processing task to a separate executor service. This executor service can be configured by using next four parameters of above scheduling configuration bean.

  • pollingThreadCoreSize - is the core size of the file fetching thread pool

  • pollingThreadMaxSize - is the maximum number of threads of the file fetching thread pool

  • pollingQueueSize - is the queue size of the file fetching thread pool

  • pollingKeepAliveTime - is the keep alive time of the file fetching thread pool

Polling Cron Expression

Scheduling

Cron expression for the database table querying schedule. Cron expression should be a valid Quartz cron expression since the Framework is underneath using Quartz to extract the schedule from the cron expression.

Polling Start Delay

Scheduling

Delay in milliseconds to start the polling schedule. Any iteration which comes within this time period from the startup time of the framework, won’t be considered as a valid file polling iteration.

Polling Repeat Interval

Scheduling

Interval in milliseconds for the next iteration of the polling schedule. This will be considered if there isn’t a configured cron schedule.

Polling Repeat Count

Scheduling

Number of iterations which should go through the polling schedule. If this is set to 1 which means only the first iteration of the polling schedule will be considered as a valid file polling iteration and all other iterations of the schedule will be ignored. By default, this value is set to -1 which means it will consider all the iterations of the polling schedule as a valid iteration.

Concurrent Polling Count

Scheduling

Maximum number of concurrent threads which can be used to poll the configured path to fetch files from the local file system. By default, this value is 1.

Concurrent Execution Limit

Scheduling

Maximum number of concurrent threads which can be used process the fetched files from the local file system. By default, this value is 4.

Sample Use Case

Use case description

In this sample use case, let’s use File Ingress Connector to pull files with .csv and .xml extensions and then log the message message payload to the console and then send the payload to an external HTTP server. The sample integration flow should look as below.

file ingress connector 2

The files will be picked up from /tmp/files directory and it is specified as the File Path. The 'File Name Pattern' should be .*\.(xml|csv), because we onl want files wilt .csv or .xml extensions. Below image shows the property configuration of the File Ingress Connector.

file ingress connector 3

Further, in the Log processing element we configure to log the message payload and the HTTP Egress Connector is configured to send the message to an Echo Server. The Successful flow end processing element is added to complete the integration flow.

In this topic
In this topic
(c) Copyright 2016-2017 AdroitLogic. All rights reserved.
Terms of Use
Contact Us