mail ingress connector

Mail Ingress Connector

Version: 17.07

Supported Since: 17.01

What is Mail Ingress Connector?

The Mail Ingress Connector can be used to fetch emails from a mail server and inject them into UltraESB-X for further processing.

In order to use the Mail Ingress Connector, you must first select the Mail Connector dependency from the connector list when you are creating an empty Ultra project. If you have already created a project, you can add the Mail Connector dependency via Component Registry. From Tools menu, select Ultra Studio → Component Registry and from the Connectors list, select the Mail Connector dependency.
mail ingress connector 1

Out Ports

Processor

The received message will be emitted through this port to the next processing element in the flow.

On Exception

The message will be sent to this out port if the Ingress Connector fails to process the message due to some reason.

Parameters

* marked fields are mandatory

Protocol *

Basic

Protocol to be used for accessing mail. IMAP (imap) is recommended over POP3 (pop3) due to inherent limitations of the POP3 protocol (such as lack of support for flags).

Host *

Basic

Hostname (or IP address) of the mail server

Port *

Basic

Service port on the mail server (usually dependent on the protocol)

User *

Basic

Username for authenticating to the mail server

Password *

Basic

Password for authenticating to the mail server

Folder

Basic

Mail folder to be accessed for fetching mail. Check with your email provider for the folder names available under the selected protocol. (For example, Gmail IMAP offers multiple folders such as [Gmail]/Sent Mail and separate folder for each Gmail label you have created, whereas Gmail POP3 offers only a single folder INBOX.)

Other Settings

Basic

Map of additional transport-level properties for mail server access, such as standard JavaMail properties (e.g. enabling SSL as in the above example)

From

Filters

A comma-separated list of strings that should be matched against the list of From addresses in the email (see Address Matching for details)

To

Filters

A comma-separated list of strings that should be matched against the list of To addresses in the email (see Address Matching for details)

CC

Filters

A comma-separated list of strings that should be matched against the list of CC addresses in the email (see Address Matching for details)

Subject

Filters

A simple string that should be present in the email subject

Older Than

Filters

Minimum age of the email in days. Note that locally calculated time intervals will be compared against the mail server time; hence time mismatches between the local system and mail server should be take into account.

Newer Than

Filters

Maximum age of the email in days (subjected to the same timing concern as the Newer Than field)

Address Combiner

Advanced

Decides the mode of comparison in cases where more than one email address filter is included in the From, To or CC field. For example, if the value alice@mail.com, bob@mail.com is specified in the To field, using AND as the address combiner will match only the emails where the To field contains both alice@mail.com and bob@mail.com.

Search Headers

Advanced

Allows searching for emails based on their headers. A Map of email header name-value pairs (defined as a project resource can be specified here. Partial matches of the value are accepted, hence specifying MsgId-1 will match X-MsgId-10361 as well as the plain MsgId-1.

Search Header Combiner

Advanced

If multiple search header matchers are specified, this field determines whether all of them or a subset of them should be matched for the email to be considered as a match.

Search Flags

Advanced

Email flags that are required to be present/absent in the email for it to be considered a match. Note that SEEN and DELETED flags are reserved for internal use and cannot be used for filtering.

Search Flag Combiner

Advanced

Similar to Search Header Combiner, this field determines whether all or a subset of the specified flags should match for the email to be considered as a match

Server-side Filtering

Advanced

If enabled, the email search query would be sent to and processed on the server rather than the local system. This may significantly improve processing speed and reduce network bandwidth usage because, when the option is disabled, metadata of each email in the folder has to be downloaded before the matching logic can be executed on it. Note, however, that server-side search may not work as expected in all cases, and availability and compatibility of search options offered by the email provider should be validated if this option is to be used. (For example, subject line search in Gmail does not accept special characters, so a subject query for UltraESB-X may not match the subject AdroitLogic UltraESB-X Released.)

Remove Headers

Processing

A comma-separated list of header names to be excluded from the original email when generating the message to be injected. All headers not specified in this list would be extracted by default, unless this field is left empty and Preserve Headers are specified.

Preserve Headers

Processing

A comma-separated list of header names to be extracted from the original email into the injected message (as transport headers). Value specified in this field will be ignored if Remove Headers field is non-empty.

Lazy-fetch Payload

Processing

Enabling this option delays the fetching (downloading) of the message payload (email body) until it is requested by some component later in the message flow. Once downloaded, the payload is cached for further use. Enable this option if your message processing logic does not require the message body (e.g. in a notification processor, where only the presence of an email with a particular header or subject line would be used as the trigger).

Lazy-fetch Attachments

Processing

Similar to Lazy-fetch Payload option, but applies for the attachments available in the message

Copy after Process

Completion

Specifies a mail folder to which the email should be copied (on server side) after successful completion of processing of the injected message

Copy on Failure

Completion

Specifies a mail folder to which the email should be copied (on server side) in case of processing (message flow) failure for the injected message

Delete after Process

Completion

Specifies whether the original email should be deleted from the mail server (marked with DELETED flag) after successful completion of processing of the injected message. If disabled, the message would be marked as read (with the SEEN flag) in order to prevent it from being picked in the next email check (polling) cycle. Note that in mail check protocols like POP3 where labels are not supported by default, this implementation may not work properly.

Delete on Failure

Completion

Similar to Delete after Process option but applies for cases where processing (flow) of the injected message has failed

Scheduler Configuration

Scheduling

Bean reference of the scheduler configuration bean which should be declared as a resource in the project.xpml file. By default there is internal scheduler configuration within the framework which will be shared by all the polling connectors. If you need to configure higher level of concurrent processing threads which will fetch the emails, you can configure your own thread pool configuration while declaring the parameters of the scheduler configuration bean as below.

<x:resources>
    <x:resource id="custom-scheduler-config">
        <bean id="schedulerConfigBean" class="org.adroitlogic.x.base.trp.SchedulerConfig">
            <constructor-arg name="name" value="my-custom-scheduler"/>
            <property name="schedulerThreadCount" value="4"/>
            <property name="pollingThreadCoreSize" value="4"/>
            <property name="pollingThreadMaxSize" value="10"/>
            <property name="pollingQueueSize" value="25"/>
            <property name="pollingKeepAliveTime" value="5000"/>
        </bean>
    </x:resource>
</x:resources>

In this configuration,

  • schedulerThreadCount - is the number of threads which will be used to schedule the number of threads to be used to handle scheduled polling tasks in this Mail Ingress Connector. Generally this value should be smaller value since polling threads don’t do heavy task in this, rather it’s just initiating the polling task for the scheduling iteration and handover the file fetching and processing task to a separate executor service. This executor service can be configured by using next four parameters of above scheduling configuration bean.

  • pollingThreadCoreSize - is the core size of the mail fetching thread pool

  • pollingThreadMaxSize - is the maximum number of threads of the mail fetching thread pool

  • pollingQueueSize - is the queue size of the mail fetching thread pool

  • pollingKeepAliveTime - is the keep alive time of the mail fetching thread pool

Polling Cron Expression

Scheduling

Cron expression for the mail polling schedule. Cron expression should be a valid Quartz cron expression since the Framework is underneath using Quartz to extract the schedule from the cron expression.

Polling Start Delay

Scheduling

Delay in milliseconds to start the polling schedule. Any iteration which comes within this time period from the startup time of the framework, won’t be considered as a valid file polling iteration.

Polling Repeat Interval

Scheduling

Interval in milliseconds for the next iteration of the polling schedule. This will be considered if there isn’t a configured cron schedule.

Polling Repeat Count

Scheduling

Number of iterations which should go through the polling schedule. If this is set to 1 which means only the first iteration of the polling schedule will be considered as a valid file polling iteration and all other iterations of the schedule will be ignored. By default, this value is set to -1 which means it will consider all the iterations of the polling schedule as a valid iteration.

Concurrent Polling Count

Scheduling

Maximum number of concurrent threads which can be used to poll the configured mail server for emails. By default, this value is 1.

Concurrent Execution Count

Scheduling

Maximum number of concurrent threads which can be used to process the fetched emails from the mail server. By default, this value is 4.

Address Matching

Individual comma-separated elements in the queries for From, To and CC fields under Filters, are separately applied as partial match ("contains") queries against the respective fields in the original email. Whether any or all of the elements should be matched, is determined based on the Address Combiner value under Advanced category.

E.g. in the example we have tried out above, the From query sender1@mail1.com, sender2@mail2.com, sender3 along with the OR Address Combiner, will match emails with any of the following From fields:

  • sender1@mail1.com (exact match with first element)

  • anon-sender3@mail.ru (partial match with third element)

  • sender2@mail2.com, sender3@mail3.com (exact match with second element, partial match with third)

If we instead used the same query with AND as the Address Combiner, none of the above would be matched (as none matches all 3 fragments in the query). But an email with a From field anon-sender1@mail1.com, anon-sender2@mail2.com, verified-sender3@mail.com.au will still be matched.

Sample Use case

A minimal configuration for the Mail Ingress Connector consists of the following:

  • Protocol

  • Host

  • Port

  • User

  • Password

In addition, in most mail server implementations it will also be necessary to define a Folder for polling mail.

If the mailbox is accessible only over SSL (as is the case with mail providers like GMail or Yahoo), an Other Settings Map containing mail.<protocol>.ssl.enable set to true will also have to be specified. The Other Settings Map can also be useful for specifying other custom transport-related parameters, such as standard JavaMail parameters.

When accessing a mailbox with flag support, the connector will initially filter out all read and/or deleted emails (having flag SEEN and/or DELETED). Hence only unread, non-deleted emails will be considered as candidates for being fetched, which will then be filtered against the other specified filtering criteria. Flags SEEN and DELETED are used internally by the connector to mark the original email as already processed, so that it would not be picked repeatedly during subsequent checking cycles. In cases where flags are not supported by the mailbox, behaviour of the connector cannot be guaranteed.

First, let’s add a Mail Ingress Connector with the following configurations:

mail basic
Figure 1. Basic Tab configuration

For Other Settings add a String map as shown below

imap map
Figure 2. Other Settings Map
mail advance
Figure 3. Advanced Tab configuration

For Search Flags add a String map as shown below

flagged map
Figure 4. Search Flags Map
mail completion
Figure 5. Completion Tab configuration
mail filter
Figure 6. Filters Tab configuration
mail processing
Figure 7. Processing Tab configuration
mail scheduling
Figure 8. Scheduling Tab configuration

(Replace the email account user, password and the From and To filter values as per your choice. If you intend to use a provider other than GMail, you will also have to change the Copy after Process folder accordingly.)

This configuration will search for emails matching the following criteria:

  • inside INBOX

  • any derivative of sender1@mail1.com, sender2@mail2.com or sender3 on the From field (e.g. mails from sender1@mail1.com as well as distant-sender3@mail.com.au will be matched; see Address Matching for details)

  • myemailaddress@gmail.com on the To field

  • received during the last 2 weeks

  • not having the FLAGGED flag (analogous to not being Starred in providers like Gmail)

When the email is processed,

  • all email headers except the Subject header will be extracted

  • the email body will be immediately available (pre-fetched) as the UltraESB-X message payload

  • attachments would be added to the message, but will not be fetched in advance (will only become available if the ESB tries to read their content at some point)

After message processing completes, the original email will be

  • marked with SEEN flag (by the framework, upon which the integration flow logic has no control)

  • moved to the [Gmail]/Starred folder on the mail server (later we will notice a side effect of this action)

After the mail is fetched, its body will be available as the payload of the message. Let’s add a Console Logger to log this payload to the console and successfully complete the flow with a Successful Flow End element:

Log Template

@{message.payload}

Log Level

INFO

The final flow would resemble the following:

mail ingress flow

For this exercise, in order to see the actual email fetch process in action, let’s increase the log level of the mail transport by updating src/test/resources/log4j2.xml:

log4j2.xml (src/test/resources/)

<Configuration ...>
    ...
    <Loggers>
        ...
        <AsyncLogger name="org.adroitlogic.x.transport.mail" level="TRACE"/>

Now, after creating a run configuration and running the project, you will be able to see the specified email account being accessed, the matching emails being fetched from the mailbox, and their bodies being logged in the console:

2017-01-24T07:29:19,435 [127.0.1.1-janaka-ENVY] [mail-default-poll1] [system-] [000001D000] DEBUG MailMessageInjector Checking mail of aliceandbob2016@gmail.com
2017-01-24T07:29:19,441 [127.0.1.1-janaka-ENVY] [mail-default-poll1] [system-] [000000D000] DEBUG MailTransportListener Building imap mail store for aliceandbob2016@gmail.com
DEBUG: JavaMail version 1.5.6
...
DEBUG IMAP: trying to connect to host "imap.gmail.com", port 993, isSSL true
2017-01-24T07:29:19,469 [127.0.1.1-janaka-ENVY] [mail-default-poll1] [system-] [000000D000] DEBUG MailTransportListener Connecting to imap mail store at imap.gmail.com:993 as aliceandbob2016@gmail.com
...
DEBUG IMAP: protocolConnect login, host=imap.gmail.com, user=aliceandbob2016@gmail.com, password=<non-null>
...
2017-01-24T07:29:24,636 [127.0.1.1-janaka-ENVY] [mail-default-poll1] [system-] [000000D000] DEBUG MailTransportListener Accessing folder INBOX on mail store of aliceandbob2016@gmail.com
...
A4 OK [READ-WRITE] INBOX selected. (Success)
2017-01-24T07:29:24,936 [127.0.1.1-janaka-ENVY] [mail-default-poll1] [system-] [000001D000] DEBUG MailMessageInjector Listing INBOX mails for aliceandbob2016@gmail.com
2017-01-24T07:29:24,936 [127.0.1.1-janaka-ENVY] [mail-default-poll1] [system-] [000000D000] DEBUG MailSearchUtil Building mail search query for INBOX
2017-01-24T07:29:24,943 [127.0.1.1-janaka-ENVY] [mail-default-poll1] [system-] [000000D000] DEBUG MailSearchUtil Searching folder INBOX using server-side search
A5 SEARCH UNSEEN UNDELETED UNFLAGGED OR OR FROM janakaud@gmail.com FROM rajindruparathna@gmail.com FROM thuvvashan90 TO aliceandbob2016@gmail.com SINCE 10-Jan-2017 ALL
* SEARCH 4 7
A5 OK SEARCH completed (Success)
2017-01-24T07:29:25,426 [127.0.1.1-janaka-ENVY] [mail-default-poll1] [system-] [000000D000] DEBUG MailSearchUtil 2 matches in folder INBOX
...
A7 FETCH 4 (UID)
...
2017-01-24T07:29:26,591 [127.0.1.1-janaka-ENVY] [mail-default-poll1] [system-] [000001D000] DEBUG MailMessageInjector Composing message with mail #4
A10 FETCH 4 (ENVELOPE INTERNALDATE RFC822.SIZE)
...
A11 FETCH 4 (BODY.PEEK[HEADER])
* 4 FETCH (BODY[HEADER] {2938}
Delivered-To: aliceandbob2016@gmail.com
Received: by 10.237.55.232 with SMTP id j95csp286327qtb; Mon, 16 Jan 2017
 22:47:04 -0800 (PST)
...
2017-01-24T07:29:27,198 [127.0.1.1-janaka-ENVY] [mail-default-poll1] [system-] [000001D000] DEBUG MailMessageInjector Parsing mail #4 as a multipart message
...
2017-01-24T07:29:27,498 [127.0.1.1-janaka-ENVY] [mail-default-poll1] [system-] [000000D000] DEBUG CachingDataHandlerFormat Composed payload for message 7aefc66a-6059-7034-0000-000000000001
A13 FETCH 4 (BODY[1]<0.8>)
* 4 FETCH (FLAGS (\Seen) BODY[1]<0> {8}
qwerty
)
...
2017-01-24T07:29:27,879 [127.0.1.1-janaka-ENVY] [mail-default-poll1] [system-] [000000D000] DEBUG CachingDataHandlerFormat Successfully fetched content from DataHandler to cache file x-file199.tmp
2017-01-24T07:29:27,882 [127.0.1.1-janaka-ENVY] [mail-default-poll1] [system-] [000001D000] DEBUG MailMessageInjector Composed message 7aefc66a-6059-7034-0000-000000000001 with mail #4
...
2017-01-24T07:29:27,916 [127.0.1.1-janaka-ENVY] [pool-2-thread-1] [mail-1.0-SNAPSHOT] [000000I000]  INFO LoggerProcessingElement qwerty
2017-01-24T07:29:27,917 [127.0.1.1-janaka-ENVY] [pool-2-thread-1] [mail-1.0-SNAPSHOT] [000001D000] DEBUG MailMessageInjector Moving processed mail #4 to folder [Gmail]/Starred
...
2017-01-24T07:29:28,191 [127.0.1.1-janaka-ENVY] [mail-default-poll1] [system-] [000001D000] DEBUG MailMessageInjector Composing message with mail #7
...
2017-01-24T07:29:29,549 [127.0.1.1-janaka-ENVY] [mail-default-poll1] [system-] [000001D000] DEBUG MailMessageInjector Composed message 7aefc66a-6059-7034-0000-000000000002 with mail #7
2017-01-24T07:29:29,552 [127.0.1.1-janaka-ENVY] [pool-2-thread-2] [mail-1.0-SNAPSHOT] [000000I000]  INFO LoggerProcessingElement
2017-01-24T07:29:29,553 [127.0.1.1-janaka-ENVY] [pool-2-thread-2] [mail-1.0-SNAPSHOT] [000001D000] DEBUG MailMessageInjector Moving processed mail #7 to folder [Gmail]/Starred
...
2017-01-24T07:29:31,510 [127.0.1.1-janaka-ENVY] [pool-2-thread-2] [mail-1.0-SNAPSHOT] [000000D000] DEBUG MailTransportListener Accessing folder [Gmail]/Starred on mail store of aliceandbob2016@gmail.com
...
A19 COPY 4 [Gmail]/Starred
...
A19 OK [COPYUID 4 36 5] (Success)
A20 STORE 4 +FLAGS (\Seen)
...
A20 OK Success
...
2017-01-24T07:29:32,507 [127.0.1.1-janaka-ENVY] [pool-2-thread-1] [mail-1.0-SNAPSHOT] [000001D000] DEBUG MailMessageInjector Marking mail #4 as SEEN
...
2017-01-24T07:29:34,426 [127.0.1.1-janaka-ENVY] [mail-default-poll2] [system-] [000001D000] DEBUG MailMessageInjector Checking mail of aliceandbob2016@gmail.com
...
2017-01-24T07:29:34,587 [127.0.1.1-janaka-ENVY] [mail-default-poll2] [system-] [000001D000] DEBUG MailMessageInjector Listing INBOX mails for aliceandbob2016@gmail.com
2017-01-24T07:29:34,587 [127.0.1.1-janaka-ENVY] [pool-2-thread-2] [mail-1.0-SNAPSHOT] [000001D000] DEBUG MailMessageInjector Marking mail #7 as SEEN
2017-01-24T07:29:34,587 [127.0.1.1-janaka-ENVY] [mail-default-poll2] [system-] [000000D000] DEBUG MailSearchUtil Searching folder INBOX using server-side search
...
A23 SEARCH UNSEEN UNDELETED UNFLAGGED OR OR FROM janakaud@gmail.com FROM rajindruparathna@gmail.com FROM thuvvashan90 TO aliceandbob2016@gmail.com SINCE 10-Jan-2017 ALL
* SEARCH
A23 OK SEARCH completed (Success)
2017-01-24T07:29:35,187 [127.0.1.1-janaka-ENVY] [mail-default-poll2] [system-] [000000D000] DEBUG MailSearchUtil 0 matches in folder INBOX

Based on the above logs, 2 matching emails were found and processed during the first cycle (starting at 2017-01-24T07:29:19,435) and they were not picked again because they were marked as SEEN after processing. But even if we try to mark them as unread again (via an email client) they would not be picked for processing again. This is because Gmail marks the email as FLAGGED when it is moved to the [Gmail]/Starred folder (i.e. "starred", from Gmail terminology), so that it no longer matches our query. If we invoke the "Remove Star" action on the emails, they would become eligible for processing again.

In this topic
In this topic
Contact Us