mail ingress connector

Mail Ingress Connector

Version: 17.07

Supported Since: 17.01

What is Mail Ingress Connector?

The Mail Ingress Connector can be used to fetch emails from a mail server and inject them into UltraESB-X for further processing.

In order to use the Mail Ingress Connector, you must first select the Mail Connector dependency from the connector list when you are creating an empty Ultra project. If you have already created a project, you can add the Mail Connector dependency via Component Registry. From Tools menu, select Ultra Studio → Component Registry and from the Connectors list, select the Mail Connector dependency.
mail ingress connector 1

Out Ports

Processor

The received message will be emitted through this port to the next processing element in the flow.

On Exception

The message will be sent to this out port if the Ingress Connector fails to process the message due to some reason.

Parameters

* marked fields are mandatory

Protocol *

Basic

Protocol to be used for accessing mail. IMAP (imap) is recommended over POP3 (pop3) due to inherent limitations of the POP3 protocol (such as lack of support for flags).

Host *

Basic

Hostname (or IP address) of the mail server

Port *

Basic

Service port on the mail server (usually dependent on the protocol)

User *

Basic

Username for authenticating to the mail server

Password *

Basic

Password for authenticating to the mail server

Folder

Basic

Mail folder to be accessed for fetching mail. Check with your email provider for the folder names available under the selected protocol. (For example, Gmail IMAP offers multiple folders such as [Gmail]/Sent Mail and separate folder for each Gmail label you have created, whereas Gmail POP3 offers only a single folder INBOX.)

Other Settings

Basic

Map of additional transport-level properties for mail server access, such as standard JavaMail properties (e.g. enabling SSL as in the above example)

From

Filters

A comma-separated list of strings that should be matched against the list of From addresses in the email (see Address Matching for details)

To

Filters

A comma-separated list of strings that should be matched against the list of To addresses in the email (see Address Matching for details)

CC

Filters

A comma-separated list of strings that should be matched against the list of CC addresses in the email (see Address Matching for details)

Subject

Filters

A simple string that should be present in the email subject

Older Than

Filters

Minimum age of the email in days. Note that locally calculated time intervals will be compared against the mail server time; hence time mismatches between the local system and mail server should be take into account.

Newer Than

Filters

Maximum age of the email in days (subjected to the same timing concern as the Newer Than field)

Address Combiner

Advanced

Decides the mode of comparison in cases where more than one email address filter is included in the From, To or CC field. For example, if the value alice@mail.com, bob@mail.com is specified in the To field, using AND as the address combiner will match only the emails where the To field contains both alice@mail.com and bob@mail.com.

Search Headers

Advanced

Allows searching for emails based on their headers. A Map of email header name-value pairs (defined as a project resource can be specified here. Partial matches of the value are accepted, hence specifying MsgId-1 will match X-MsgId-10361 as well as the plain MsgId-1.

Search Header Combiner

Advanced

If multiple search header matchers are specified, this field determines whether all of them or a subset of them should be matched for the email to be considered as a match.

Search Flags

Advanced

Email flags that are required to be present/absent in the email for it to be considered a match. Note that SEEN and DELETED flags are reserved for internal use and cannot be used for filtering.

Search Flag Combiner

Advanced

Similar to Search Header Combiner, this field determines whether all or a subset of the specified flags should match for the email to be considered as a match

Server-side Filtering

Advanced

If enabled, the email search query would be sent to and processed on the server rather than the local system. This may significantly improve processing speed and reduce network bandwidth usage because, when the option is disabled, metadata of each email in the folder has to be downloaded before the matching logic can be executed on it. Note, however, that server-side search may not work as expected in all cases, and availability and compatibility of search options offered by the email provider should be validated if this option is to be used. (For example, subject line search in Gmail does not accept special characters, so a subject query for UltraESB-X may not match the subject AdroitLogic UltraESB-X Released.)

Remove Headers

Processing

A comma-separated list of header names to be excluded from the original email when generating the message to be injected. All headers not specified in this list would be extracted by default, unless this field is left empty and Preserve Headers are specified.

Preserve Headers

Processing

A comma-separated list of header names to be extracted from the original email into the injected message (as transport headers). Value specified in this field will be ignored if Remove Headers field is non-empty.

Lazy-fetch Payload

Processing

Enabling this option delays the fetching (downloading) of the message payload (email body) until it is requested by some component later in the message flow. Once downloaded, the payload is cached for further use. Enable this option if your message processing logic does not require the message body (e.g. in a notification processor, where only the presence of an email with a particular header or subject line would be used as the trigger).

Lazy-fetch Attachments

Processing

Similar to Lazy-fetch Payload option, but applies for the attachments available in the message

Copy after Process

Completion

Specifies a mail folder to which the email should be copied (on server side) after successful completion of processing of the injected message

Copy on Failure

Completion

Specifies a mail folder to which the email should be copied (on server side) in case of processing (message flow) failure for the injected message

Delete after Process

Completion

Specifies whether the original email should be deleted from the mail server (marked with DELETED flag) after successful completion of processing of the injected message. If disabled, the message would be marked as read (with the SEEN flag) in order to prevent it from being picked in the next email check (polling) cycle. Note that in mail check protocols like POP3 where labels are not supported by default, this implementation may not work properly.

Delete on Failure

Completion

Similar to Delete after Process option but applies for cases where processing (flow) of the injected message has failed

Scheduler Configuration

Scheduling

Bean reference of the scheduler configuration bean which should be declared as a resource in the project.xpml file. By default there is internal scheduler configuration within the framework which will be shared by all the polling connectors. If you need to configure higher level of concurrent processing threads which will fetch the emails, you can configure your own thread pool configuration while declaring the parameters of the scheduler configuration bean as below.

<x:resources>
    <x:resource id="custom-scheduler-config">
        <bean id="schedulerConfigBean" class="org.adroitlogic.x.base.trp.SchedulerConfig">
            <constructor-arg name="name" value="my-custom-scheduler"/>
            <property name="schedulerThreadCount" value="4"/>
            <property name="pollingThreadCoreSize" value="4"/>
            <property name="pollingThreadMaxSize" value="10"/>
            <property name="pollingQueueSize" value="25"/>
            <property name="pollingKeepAliveTime" value="5000"/>
        </bean>
    </x:resource>
</x:resources>

In this configuration,

  • schedulerThreadCount - is the number of threads which will be used to schedule the number of threads to be used to handle scheduled polling tasks in this Mail Ingress Connector. Generally this value should be smaller value since polling threads don’t do heavy task in this, rather it’s just initiating the polling task for the scheduling iteration and handover the file fetching and processing task to a separate executor service. This executor service can be configured by using next four parameters of above scheduling configuration bean.

  • pollingThreadCoreSize - is the core size of the mail fetching thread pool

  • pollingThreadMaxSize - is the maximum number of threads of the mail fetching thread pool

  • pollingQueueSize - is the queue size of the mail fetching thread pool

  • pollingKeepAliveTime - is the keep alive time of the mail fetching thread pool

Polling Cron Expression

Scheduling

Cron expression for the mail polling schedule. Cron expression should be a valid Quartz cron expression since the Framework is underneath using Quartz to extract the schedule from the cron expression.

Polling Start Delay

Scheduling

Delay in milliseconds to start the polling schedule. Any iteration which comes within this time period from the startup time of the framework, won’t be considered as a valid file polling iteration.

Polling Repeat Interval

Scheduling

Interval in milliseconds for the next iteration of the polling schedule. This will be considered if there isn’t a configured cron schedule.

Polling Repeat Count

Scheduling

Number of iterations which should go through the polling schedule. If this is set to 1 which means only the first iteration of the polling schedule will be considered as a valid file polling iteration and all other iterations of the schedule will be ignored. By default, this value is set to -1 which means it will consider all the iterations of the polling schedule as a valid iteration.

Concurrent Polling Count

Scheduling

Maximum number of concurrent threads which can be used to poll the configured mail server for emails. By default, this value is 1.

Concurrent Execution Count

Scheduling

Maximum number of concurrent threads which can be used to process the fetched emails from the mail server. By default, this value is 4.

Address Matching

Individual comma-separated elements in the queries for From, To and CC fields under Filters, are separately applied as partial match ("contains") queries against the respective fields in the original email. Whether any or all of the elements should be matched, is determined based on the Address Combiner value under Advanced category.

E.g. in the example we have tried out above, the From query sender1@mail1.com, sender2@mail2.com, sender3 along with the OR Address Combiner, will match emails with any of the following From fields:

  • sender1@mail1.com (exact match with first element)

  • anon-sender3@mail.ru (partial match with third element)

  • sender2@mail2.com, sender3@mail3.com (exact match with second element, partial match with third)

If we instead used the same query with AND as the Address Combiner, none of the above would be matched (as none matches all 3 fragments in the query). But an email with a From field anon-sender1@mail1.com, anon-sender2@mail2.com, verified-sender3@mail.com.au will still be matched.

Sample Use case

A minimal configuration for the Mail Ingress Connector consists of the following:

  • Protocol

  • Host

  • Port

  • User

  • Password

In addition, in most mail server implementations it will also be necessary to define a Folder for polling mail.

If the mailbox is accessible only over SSL (as is the case with mail providers like GMail or Yahoo), an Other Settings Map containing mail.<protocol>.ssl.enable set to true will also have to be specified. The Other Settings Map can also be useful for specifying other custom transport-related parameters, such as standard JavaMail parameters.

When accessing a mailbox with flag support, the connector will initially filter out all read and/or deleted emails (having flag SEEN and/or DELETED). Hence only unread, non-deleted emails will be considered as candidates for being fetched, which will then be filtered against the other specified filtering criteria. Flags SEEN and DELETED are used internally by the connector to mark the original email as already processed, so that it would not be picked repeatedly during subsequent checking cycles. In cases where flags are not supported by the mailbox, behaviour of the connector cannot be guaranteed.

First, let’s add a Mail Ingress Connector with the following configurations:

mail basic
Figure 1. Basic Tab configuration

For Other Settings add a String map as shown below

imap map
Figure 2. Other Settings Map
mail advance
Figure 3. Advanced Tab configuration

For Search Flags add a String map as shown below

flagged map
Figure 4. Search Flags Map