Version: 17.07
Supported Since: 17.01
The Mail Ingress Connector can be used to fetch emails from a mail server and inject them into UltraESB-X for further processing.
In order to use the Mail Ingress Connector, you must first select the Mail Connector dependency from the connector list when you are creating an empty Ultra project. If you have already created a project, you can add the Mail Connector dependency via Component Registry. From Tools menu, select Ultra Studio → Component Registry and from the Connectors list, select the Mail Connector dependency. |
Processor |
The received message will be emitted through this port to the next processing element in the flow. |
On Exception |
The message will be sent to this out port if the Ingress Connector fails to process the message due to some reason. |
* marked fields are mandatory
Protocol * |
Basic |
Protocol to be used for accessing mail. IMAP ( |
Host * |
Basic |
Hostname (or IP address) of the mail server |
Port * |
Basic |
Service port on the mail server (usually dependent on the protocol) |
User * |
Basic |
Username for authenticating to the mail server |
Password * |
Basic |
Password for authenticating to the mail server |
Folder |
Basic |
Mail folder to be accessed for fetching mail. Check with your email provider for the folder names available under the
selected protocol. (For example, Gmail IMAP offers multiple folders such as |
Other Settings |
Basic |
Map of additional transport-level properties for mail server access, such as standard JavaMail properties (e.g. enabling SSL as in the above example) |
From |
Filters |
A comma-separated list of strings that should be matched against the list of From addresses in the email (see Address Matching for details) |
To |
Filters |
A comma-separated list of strings that should be matched against the list of To addresses in the email (see Address Matching for details) |
CC |
Filters |
A comma-separated list of strings that should be matched against the list of CC addresses in the email (see Address Matching for details) |
Subject |
Filters |
A simple string that should be present in the email subject |
Older Than |
Filters |
Minimum age of the email in days. Note that locally calculated time intervals will be compared against the mail server time; hence time mismatches between the local system and mail server should be take into account. |
Newer Than |
Filters |
Maximum age of the email in days (subjected to the same timing concern as the |
Address Combiner |
Advanced |
Decides the mode of comparison in cases where more than one email address filter is included in the |
Search Headers |
Advanced |
Allows searching for emails based on their headers.
A Map of email header name-value pairs (defined as a project resource can be specified here.
Partial matches of the value are accepted, hence specifying |
Search Header Combiner |
Advanced |
If multiple search header matchers are specified, this field determines whether all of them or a subset of them should be matched for the email to be considered as a match. |
Search Flags |
Advanced |
Email flags that are required to be present/absent in the email for it to be considered a match.
Note that |
Search Flag Combiner |
Advanced |
Similar to |
Server-side Filtering |
Advanced |
If enabled, the email search query would be sent to and processed on the server rather than the local system.
This may significantly improve processing speed and reduce network bandwidth usage because, when the option is disabled,
metadata of each email in the folder has to be downloaded before the matching logic can be executed on it.
Note, however, that server-side search may not work as expected in all cases, and availability and compatibility of
search options offered by the email provider should be validated if this option is to be used.
(For example, subject line search in Gmail does not accept special characters, so a subject query for |
Remove Headers |
Processing |
A comma-separated list of header names to be excluded from the original email when generating the message to be injected.
All headers not specified in this list would be extracted by default, unless this field is left empty and
|
Preserve Headers |
Processing |
A comma-separated list of header names to be extracted from the original email into the injected message (as transport headers).
Value specified in this field will be ignored if |
Lazy-fetch Payload |
Processing |
Enabling this option delays the fetching (downloading) of the message payload (email body) until it is requested by some component later in the message flow. Once downloaded, the payload is cached for further use. Enable this option if your message processing logic does not require the message body (e.g. in a notification processor, where only the presence of an email with a particular header or subject line would be used as the trigger). |
Lazy-fetch Attachments |
Processing |
Similar to |
Copy after Process |
Completion |
Specifies a mail folder to which the email should be copied (on server side) after successful completion of processing of the injected message |
Copy on Failure |
Completion |
Specifies a mail folder to which the email should be copied (on server side) in case of processing (message flow) failure for the injected message |
Delete after Process |
Completion |
Specifies whether the original email should be deleted from the mail server (marked with |
Delete on Failure |
Completion |
Similar to |
Scheduler Configuration |
Scheduling |
Bean reference of the scheduler configuration bean which should be declared as a resource in the project.xpml file. By default there is internal scheduler configuration within the framework which will be shared by all the polling connectors. If you need to configure higher level of concurrent processing threads which will fetch the emails, you can configure your own thread pool configuration while declaring the parameters of the scheduler configuration bean as below.
In this configuration,
|
Polling Cron Expression |
Scheduling |
Cron expression for the mail polling schedule. Cron expression should be a valid Quartz cron expression since the Framework is underneath using Quartz to extract the schedule from the cron expression. |
Polling Start Delay |
Scheduling |
Delay in milliseconds to start the polling schedule. Any iteration which comes within this time period from the startup time of the framework, won’t be considered as a valid file polling iteration. |
Polling Repeat Interval |
Scheduling |
Interval in milliseconds for the next iteration of the polling schedule. This will be considered if there isn’t a configured cron schedule. |
Polling Repeat Count |
Scheduling |
Number of iterations which should go through the polling schedule. If this is set to 1 which means only the first iteration of the polling schedule will be considered as a valid file polling iteration and all other iterations of the schedule will be ignored. By default, this value is set to -1 which means it will consider all the iterations of the polling schedule as a valid iteration. |
Concurrent Polling Count |
Scheduling |
Maximum number of concurrent threads which can be used to poll the configured mail server for emails. By default, this value is 1. |
Concurrent Execution Count |
Scheduling |
Maximum number of concurrent threads which can be used to process the fetched emails from the mail server. By default, this value is 4. |
Individual comma-separated elements in the queries for From
, To
and CC
fields under Filters, are separately
applied as partial match ("contains") queries against the respective fields in the original email.
Whether any or all of the elements should be matched, is determined based on the Address Combiner
value under
Advanced category.
E.g. in the example we have tried out above, the From
query sender1@mail1.com
, sender2@mail2.com
, sender3
along with
the OR
Address Combiner
, will match emails with any of the following From fields:
sender1@mail1.com (exact match with first element)
anon-sender3@mail.ru (partial match with third element)
sender2@mail2.com, sender3@mail3.com (exact match with second element, partial match with third)
If we instead used the same query with AND
as the Address Combiner
, none of the above would
be matched (as none matches all 3 fragments in the query). But an email with a From field
anon-sender1@mail1.com, anon-sender2@mail2.com, verified-sender3@mail.com.au will still be matched.
A minimal configuration for the Mail Ingress Connector consists of the following:
Protocol
Host
Port
User
Password
In addition, in most mail server implementations it will also be necessary to define a Folder
for polling mail.
If the mailbox is accessible only over SSL (as is the case with mail providers like GMail or Yahoo), an
Other Settings
Map containing mail.<protocol>.ssl.enable
set to true
will also have to be specified.
The Other Settings
Map can also be useful for specifying other custom transport-related parameters, such as
standard JavaMail parameters.
When accessing a mailbox with flag support, the connector will initially filter out all read and/or deleted emails
(having flag SEEN
and/or DELETED
).
Hence only unread, non-deleted emails will be considered as candidates for being fetched, which will then be filtered
against the other specified filtering criteria.
Flags SEEN
and DELETED
are used internally by the connector to mark the original email as already processed,
so that it would not be picked repeatedly during subsequent checking cycles.
In cases where flags are not supported by the mailbox, behaviour of the connector cannot be guaranteed.
First, let’s add a Mail Ingress Connector with the following configurations:
For Other Settings
add a String map as shown below
For Search Flags
add a String map as shown below
(Replace the email account user, password and the From
and To
filter values as per your choice.
If you intend to use a provider other than GMail, you will also have to change the Copy after Process
folder accordingly.)
This configuration will search for emails matching the following criteria:
inside INBOX
any derivative of sender1@mail1.com, sender2@mail2.com or sender3 on the From field
(e.g. mails from sender1@mail1.com
as well as distant-sender3@mail.com.au
will be matched;
see Address Matching for details)
myemailaddress@gmail.com on the To field
received during the last 2 weeks
not having the FLAGGED
flag (analogous to not being Starred in providers like Gmail)
When the email is processed,
all email headers except the Subject
header will be extracted
the email body will be immediately available (pre-fetched) as the UltraESB-X message payload
attachments would be added to the message, but will not be fetched in advance (will only become available if the ESB tries to read their content at some point)
After message processing completes, the original email will be
marked with SEEN
flag (by the framework, upon which the integration flow logic has no control)
moved to the [Gmail]/Starred
folder on the mail server (later we will notice a side effect of this action)
After the mail is fetched, its body will be available as the payload of the message. Let’s add a Console Logger to log this payload to the console and successfully complete the flow with a Successful Flow End element:
Log Template |
|
Log Level |
|
The final flow would resemble the following:
For this exercise, in order to see the actual email fetch process in action, let’s increase the log level of the mail
transport by updating src/test/resources/log4j2.xml
:
log4j2.xml (src/test/resources/)
<Configuration ...>
...
<Loggers>
...
<AsyncLogger name="org.adroitlogic.x.transport.mail" level="TRACE"/>
Now, after creating a run configuration and running the project, you will be able to see the specified email account being accessed, the matching emails being fetched from the mailbox, and their bodies being logged in the console:
2017-01-24T07:29:19,435 [127.0.1.1-janaka-ENVY] [mail-default-poll1] [system-] [000001D000] DEBUG MailMessageInjector Checking mail of aliceandbob2016@gmail.com
2017-01-24T07:29:19,441 [127.0.1.1-janaka-ENVY] [mail-default-poll1] [system-] [000000D000] DEBUG MailTransportListener Building imap mail store for aliceandbob2016@gmail.com
DEBUG: JavaMail version 1.5.6
...
DEBUG IMAP: trying to connect to host "imap.gmail.com", port 993, isSSL true
2017-01-24T07:29:19,469 [127.0.1.1-janaka-ENVY] [mail-default-poll1] [system-] [000000D000] DEBUG MailTransportListener Connecting to imap mail store at imap.gmail.com:993 as aliceandbob2016@gmail.com
...
DEBUG IMAP: protocolConnect login, host=imap.gmail.com, user=aliceandbob2016@gmail.com, password=<non-null>
...
2017-01-24T07:29:24,636 [127.0.1.1-janaka-ENVY] [mail-default-poll1] [system-] [000000D000] DEBUG MailTransportListener Accessing folder INBOX on mail store of aliceandbob2016@gmail.com
...
A4 OK [READ-WRITE] INBOX selected. (Success)
2017-01-24T07:29:24,936 [127.0.1.1-janaka-ENVY] [mail-default-poll1] [system-] [000001D000] DEBUG MailMessageInjector Listing INBOX mails for aliceandbob2016@gmail.com
2017-01-24T07:29:24,936 [127.0.1.1-janaka-ENVY] [mail-default-poll1] [system-] [000000D000] DEBUG MailSearchUtil Building mail search query for INBOX
2017-01-24T07:29:24,943 [127.0.1.1-janaka-ENVY] [mail-default-poll1] [system-] [000000D000] DEBUG MailSearchUtil Searching folder INBOX using server-side search
A5 SEARCH UNSEEN UNDELETED UNFLAGGED OR OR FROM janakaud@gmail.com FROM rajindruparathna@gmail.com FROM thuvvashan90 TO aliceandbob2016@gmail.com SINCE 10-Jan-2017 ALL
* SEARCH 4 7
A5 OK SEARCH completed (Success)
2017-01-24T07:29:25,426 [127.0.1.1-janaka-ENVY] [mail-default-poll1] [system-] [000000D000] DEBUG MailSearchUtil 2 matches in folder INBOX
...
A7 FETCH 4 (UID)
...
2017-01-24T07:29:26,591 [127.0.1.1-janaka-ENVY] [mail-default-poll1] [system-] [000001D000] DEBUG MailMessageInjector Composing message with mail #4
A10 FETCH 4 (ENVELOPE INTERNALDATE RFC822.SIZE)
...
A11 FETCH 4 (BODY.PEEK[HEADER])
* 4 FETCH (BODY[HEADER] {2938}
Delivered-To: aliceandbob2016@gmail.com
Received: by 10.237.55.232 with SMTP id j95csp286327qtb; Mon, 16 Jan 2017
22:47:04 -0800 (PST)
...
2017-01-24T07:29:27,198 [127.0.1.1-janaka-ENVY] [mail-default-poll1] [system-] [000001D000] DEBUG MailMessageInjector Parsing mail #4 as a multipart message
...
2017-01-24T07:29:27,498 [127.0.1.1-janaka-ENVY] [mail-default-poll1] [system-] [000000D000] DEBUG CachingDataHandlerFormat Composed payload for message 7aefc66a-6059-7034-0000-000000000001
A13 FETCH 4 (BODY[1]<0.8>)
* 4 FETCH (FLAGS (\Seen) BODY[1]<0> {8}
qwerty
)
...
2017-01-24T07:29:27,879 [127.0.1.1-janaka-ENVY] [mail-default-poll1] [system-] [000000D000] DEBUG CachingDataHandlerFormat Successfully fetched content from DataHandler to cache file x-file199.tmp
2017-01-24T07:29:27,882 [127.0.1.1-janaka-ENVY] [mail-default-poll1] [system-] [000001D000] DEBUG MailMessageInjector Composed message 7aefc66a-6059-7034-0000-000000000001 with mail #4
...
2017-01-24T07:29:27,916 [127.0.1.1-janaka-ENVY] [pool-2-thread-1] [mail-1.0-SNAPSHOT] [000000I000] INFO LoggerProcessingElement qwerty
2017-01-24T07:29:27,917 [127.0.1.1-janaka-ENVY] [pool-2-thread-1] [mail-1.0-SNAPSHOT] [000001D000] DEBUG MailMessageInjector Moving processed mail #4 to folder [Gmail]/Starred
...
2017-01-24T07:29:28,191 [127.0.1.1-janaka-ENVY] [mail-default-poll1] [system-] [000001D000] DEBUG MailMessageInjector Composing message with mail #7
...
2017-01-24T07:29:29,549 [127.0.1.1-janaka-ENVY] [mail-default-poll1] [system-] [000001D000] DEBUG MailMessageInjector Composed message 7aefc66a-6059-7034-0000-000000000002 with mail #7
2017-01-24T07:29:29,552 [127.0.1.1-janaka-ENVY] [pool-2-thread-2] [mail-1.0-SNAPSHOT] [000000I000] INFO LoggerProcessingElement
2017-01-24T07:29:29,553 [127.0.1.1-janaka-ENVY] [pool-2-thread-2] [mail-1.0-SNAPSHOT] [000001D000] DEBUG MailMessageInjector Moving processed mail #7 to folder [Gmail]/Starred
...
2017-01-24T07:29:31,510 [127.0.1.1-janaka-ENVY] [pool-2-thread-2] [mail-1.0-SNAPSHOT] [000000D000] DEBUG MailTransportListener Accessing folder [Gmail]/Starred on mail store of aliceandbob2016@gmail.com
...
A19 COPY 4 [Gmail]/Starred
...
A19 OK [COPYUID 4 36 5] (Success)
A20 STORE 4 +FLAGS (\Seen)
...
A20 OK Success
...
2017-01-24T07:29:32,507 [127.0.1.1-janaka-ENVY] [pool-2-thread-1] [mail-1.0-SNAPSHOT] [000001D000] DEBUG MailMessageInjector Marking mail #4 as SEEN
...
2017-01-24T07:29:34,426 [127.0.1.1-janaka-ENVY] [mail-default-poll2] [system-] [000001D000] DEBUG MailMessageInjector Checking mail of aliceandbob2016@gmail.com
...
2017-01-24T07:29:34,587 [127.0.1.1-janaka-ENVY] [mail-default-poll2] [system-] [000001D000] DEBUG MailMessageInjector Listing INBOX mails for aliceandbob2016@gmail.com
2017-01-24T07:29:34,587 [127.0.1.1-janaka-ENVY] [pool-2-thread-2] [mail-1.0-SNAPSHOT] [000001D000] DEBUG MailMessageInjector Marking mail #7 as SEEN
2017-01-24T07:29:34,587 [127.0.1.1-janaka-ENVY] [mail-default-poll2] [system-] [000000D000] DEBUG MailSearchUtil Searching folder INBOX using server-side search
...
A23 SEARCH UNSEEN UNDELETED UNFLAGGED OR OR FROM janakaud@gmail.com FROM rajindruparathna@gmail.com FROM thuvvashan90 TO aliceandbob2016@gmail.com SINCE 10-Jan-2017 ALL
* SEARCH
A23 OK SEARCH completed (Success)
2017-01-24T07:29:35,187 [127.0.1.1-janaka-ENVY] [mail-default-poll2] [system-] [000000D000] DEBUG MailSearchUtil 0 matches in folder INBOX
Based on the above logs, 2 matching emails were found and processed during the first cycle (starting at
2017-01-24T07:29:19,435
) and they were not picked again because they were marked as SEEN
after processing.
But even if we try to mark them as unread again (via an email client) they would not be picked for processing again.
This is because Gmail marks the email as FLAGGED
when it is moved to the [Gmail]/Starred
folder (i.e. "starred",
from Gmail terminology), so that it no longer matches our query.
If we invoke the "Remove Star" action on the emails, they would become eligible for processing again.