scp ingress connector

SCP Ingress Connector

Version: 17.07

Supported Since: 17.01

What is SCP Ingress Connector?

SCP (Secure Copy) protocol allows files in the local file system or a remote file system to be copied to the local files system or to a remote file system. This uses SSH for data transfer and provides the same authentication and same level of security as SSH. Authentication between two hosts can be done using the private and public key pair or username and password. This Ingress Connector will scan for the files which matches the given pattern in a given directory and executes according the parameters set by the user.

The SCP Ingress Connector can be used to copy files from a remote machine and inject those messages to the UltraESB engine. When you are copying a file using SCP, UltraESB first download the input file to the local file system and constructs a message by attaching the downloaded local file as the payload of that message. Then user can access the input file content as the message payload and do required modifications.

We use JSch library which is a pure Java implementation of SSH2. JSch allows you to connect to an sshd server and use port forwarding, X11 forwarding, file transfer, etc., and you can integrate its functionality into your own Java programs. JSch is licensed under BSD style license.

The remote file system can be a local file system as well. If you configure localhost system as the remote system files will be copied from local system to local system.
In order to use the SCP Ingress Connector, you must first select the SCP Connector dependency from the connector list when you are creating an empty Ultra project. If you have already created a project, you can add the SCP Connector dependency via Component Registry. From Tools menu, select Ultra Studio → Component Registry and from the Connectors list, select the SCP Connector dependency.
scp ingress connector 3

Out Ports

Processor

The message will be sent to this outport for the next processing element

On Exception

The message will be sent to this outport if the processing element failed to transform the payload due to some reason

Parameters

* marked fields are mandatory

SCP Host *

Basic

Host name/IP of the remote machine that you are going to connect for SCP

SCP Remote Port

Basic

Port of the remote machine that you are going to connect to copy the files using SCP. By default value of this will be set to 22

Fetch Path *

Basic

Path of the remote machine that you are going to copy files. This path should be relative to the home directory of the above given user

File Name Pattern *

Basic

Regex pattern to filter out the files from the above configured SCP location. All the files which are matched to this regex expression will be copied from the above configured "Fetch Path" location using SCP

Username *

Basic

Username that you are going to use to connect to the remote machine

Password

Basic

Password for above user name if you are going to connect via password based authentication to the remote machine

Key File Path

Key Authentication

Path of the private key of the user which will be used to communicate with remote machine via key based authentication.

Key Password

Key Authentication

Password of the private key if it’s password protected.

Host Key Check

Key Authentication

Set to enable the host key checking while creating the communication channel with remote machine. If this is not enabled, host key checking will be skipped while SCP processing. By default, this will be false.

Local Temp Path

Advanced

File location to be used as the temporary directory while processing files in the remote machine. Input files will be downloaded to this location before injecting it to the engine as a new message. If this is not set, then temporary directory of the framework will be used as the temporary directory of the SCP processing.

Max Connections

Advanced

Maximum number of connections which will be cached within the SCP Ingress Connector. By default, this value is 10.

Timeout

Advanced

Timeout in millis, to be used while connecting to the remote machine. If it couldn’t establish a connection with the remote machine within this time period, it will be considered as a connection failure and notify to the user. By default, this value is 60000.

Remove Original File

File Modifications

Specify whether to remove original file after fetching from the remote machine. If this is set to false, input file won’t be removed by the framework automatically and that file will be picked up in every iteration of the polling schedule. By default, this value is true.

Move Location After Process

File Modifications

File path to move input file, after completing the processing successfully. This path will be considered only if it’s successfully processed without any errors

Move Location After Failure

File Modifications

File path to move input file, after completing the processing with failures. This path will be considered only if it couldn’t process the file in the remote machine

Move Timestamp Format

File Modifications

Set to attach the current timestamp in this format to the file name as a suffix, while moving files both in success and failure scenarios. If this is not set, original file name will be used as it is while moving.

Scheduler Configuration

Scheduling

Bean reference of the scheduler configuration bean which should be declared as a resource in the project.xpml file. By default there is internal scheduler configuration within the framework which will be shared by all the polling connectors. If you need to configure higher level of concurrent processing threads which will fetch the objects from remote location, you can configure your own thread pool configuration while declaring the parameters of the scheduler configuration bean as below.

<x:resources>
    <x:resource id="custom-scheduler-config">
        <bean id="schedulerConfigBean" class="org.adroitlogic.x.base.trp.SchedulerConfig">
            <constructor-arg name="name" value="my-custom-scheduler"/>
            <property name="schedulerThreadCount" value="4"/>
            <property name="pollingThreadCoreSize" value="4"/>
            <property name="pollingThreadMaxSize" value="10"/>
            <property name="pollingQueueSize" value="25"/>
            <property name="pollingKeepAliveTime" value="5000"/>
        </bean>
    </x:resource>
</x:resources>

In this configuration,

  • schedulerThreadCount - is the number of threads which will be used to schedule the number of threads to be used to handle scheduled polling tasks in this SCP Ingress Connector. Generally this value should be smaller value since polling threads don’t do heavy task in this, rather it’s just initiating the polling task for the scheduling iteration and handover the file fetching and processing task to a separate executor service. This executor service can be configured by using next four parameters of above scheduling configuration bean.

  • pollingThreadCoreSize - is the core size of the remote file fetching thread pool

  • pollingThreadMaxSize - is the maximum number of threads of the remote file fetching thread pool

  • pollingQueueSize - is the queue size of the remote file fetching thread pool

  • pollingKeepAliveTime - is the keep alive time of the remote file fetching thread pool

Polling Cron Expression

Scheduling

Cron expression for the remote file polling schedule. Cron expression should be a valid Quartz cron expression since the Framework is underneath using Quartz to extract the schedule from the cron expression.

Polling Start Delay

Scheduling

Delay in milliseconds to start the polling schedule. Any iteration which comes within this time period from the startup time of the framework, won’t be considered as a valid file polling iteration.

Polling Repeat Interval

Scheduling

Interval in milliseconds for the next iteration of the polling schedule. This will be considered if there isn’t a configured cron schedule.

Polling Repeat Count

Scheduling

Number of iterations which should go through the polling schedule. If this is set to 1 which means only the first iteration of the polling schedule will be considered as a valid file polling iteration and all other iterations of the schedule will be ignored. By default, this value is set to -1 which means it will consider all the iterations of the polling schedule as a valid iteration.

Concurrent Polling Count

Scheduling

Maximum number of concurrent threads which can be used to poll the configured remote location to fetch files. By default, this value is 1.

Concurrent Execution Count

Scheduling

Maximum number of concurrent threads which can be used to process the fetched files from the remote location. By default, this value is 4.

Sample Use Case

In this use case scenario, there is a system which scans and copy files from a remote server and copy files to another remote server using SCP. This use both SCP Ingress Connector and SCP Egress Connector. The overall Integration flow would look as below.

scp ingress connector 2

The configuration of SCP Ingress Connector would look as below

scp ingress connector 1

Then we should add the SCP Egress Connector which will copy the files from local file system to a remote file system. Now if you run the flow, and put any file which matches the file pattern you configured into the directory you specified that will be copied to your local file system and then it will be copied to your remote file system mentioned at the egress connector.

In this topic
SCP Ingress Connector
What is SCP Ingress Connector?
Out Ports
Parameters
Sample Use Case
In this topic
SCP Ingress Connector
What is SCP Ingress Connector?
Out Ports
Parameters
Sample Use Case
(c) Copyright 2016-2017 AdroitLogic. All rights reserved.
Terms of Use
Contact Us