File Polling

Version: 17.07

Supported Since: 17.01

Use Case Description

Gringotts is one of the major banks in Western Europe. They provide financial support for large number of small time business institutions. For these institutions they are required to send monthly business invoices in a secure manner. Currently they do have two separate systems to utilize for this purpose. The invoice generation needs to be done using one of their legacy backend processing systems while the generated invoices needs to be sent out to relevant parties using a separate secure file gateway. The requirement is after generating the invoices it is required to notify the file gateway about the generated invoices and the relevant information about the recipient through a JMS queue.

The legacy invoice generation system will generate the invoice file and a XML file that contains the required meta information about the invoice recipient. The generated files will be placed into a predefined locations of local file system. The secure file gateway is supposed to utilize the meta information file to extract the relevant information and send the invoices securely to the relevant parties.

Proposed Solution

Gringotts bank has decided to integrate the secure file gateway with the legacy invoice generator using an ESB. Their requirement of ESB is to poll the predefined local file system directories in-order to pick up the generated meta information XML files and validate the XML content before forwarding it to the file gateway through a JMS queue. For this implementation they have decided to use Apache ActiveMQ as the JSM provider.

file polling 1

UltraStudio Configuration

UltraESB-X Configuration

Implementation of the Solution

Prerequisite

In order to implement above use case you must first select following dependencies when you are creating a new Ultra project.

  • Local File Connector from the connector list

  • Spring JMS Connector from the connector list

  • XML Operation from the processor list

If you have already created a project, you can add above dependencies via Component Registry. From Tools menu, select Ultra Studio → Component Registry and from the Connectors list and Processors list, select above dependencies.

For this sample let’s use ActiveMQ and if you have not already installed ActiveMQ please follow the installation guide.

First open the project.xpml file of the project and right click on it (on the XML content of the file). From the context menu, select Resource Template as shown in below figure

jms egress connector 5

After that from the shown dialog box, select ActiveMQ JMS template.

jms egress connector 6

Next you need to specify the required parameters as shown below. It is mandatory to specify a bean prefix and for that you can specify any value. The only other mandatory field is the ActiveMQ Broker URL.

jms egress connector 7
Implementation

To implement above use case, first create an integration flow named “file-polling-with-xml-validation-flow”, and then add required components by going through following steps in order.

  1. Add a File Ingress Connector to obtain the generated invoice files from the predefined directory as request messages to the ESB. For the input field File Path it is required to enter a base directory location whether the invoice meta-information files will be put into by the legacy invoice generator. For the input field File Name Pattern user should provide a regular expression to filter the invoice meta-information files by their name. Instead of using this property user can use the "File Path Pattern* property by inserting a regex pattern for sub directory path to access meta-information files in the sub directories within the base directory.

There are some advanced properties that the user will be able to configure for the file ingress connector element of the ESB message flow. To make sure that the meta information file has been completely written to the file system before injecting it to the ESB, user can define the property Wait After Modification (in seconds). It defines the minimum time that the File Ingress Connector will wait after the last modified time of the file before the processing is started. By marking the property Remove Original File user will be able to command the ESB to delete the original file after the processing has been completed. Otherwise the same file will be processed repeatedly. The properties Move After Process and Move After Failure respectively defines the file paths where the original file should be copied after processing in the success case and in the failure case.

User can define the file polling schedule of the File Ingress Connector under the Scheduling configurations section. The property Polling Start Delay defines the time that the File Ingress Connector should wait after the ESB is started before it starts processing files. Polling Repeat Interval defines the time between two polling cycles for the File Listener in milli seconds. User can define the number of repetitions that the File Ingress Connector should poll by defining the property Polling Repeat Count. By entering a negative value for this property, it will imply that the polling process should continue indefinitely.

  1. Add an XSD Validator to validate the XML content of the meta-information file and select the metaValidator.xsd file as the XSD file path.

<?xml version="1.0" encoding="UTF-8"?>
<xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns="http://www.adroitlogic.org/gringotts"
            targetNamespace="http://www.adroitlogic.org/gringotts" elementFormDefault="qualified">
  <xsd:element name="invoice-meta-info">
    <xsd:complexType>
      <xsd:sequence>
        <xsd:element name="subject" type="xsd:string" minOccurs="0" maxOccurs="1" />
        <xsd:element name="receiver-id" type="xsd:string" minOccurs="0" maxOccurs="1" />
        <xsd:element name="receiver-address" type="xsd:string" minOccurs="0" maxOccurs="1" />
        <xsd:element name="payload" type="Payload" minOccurs="1" maxOccurs="1" />
      </xsd:sequence>
    </xsd:complexType>
  </xsd:element>
  <xsd:complexType name="Payload">
    <xsd:sequence>
      <xsd:element name="path" type="xsd:string" minOccurs="1" maxOccurs="unbounded" />
    </xsd:sequence>
  </xsd:complexType>
</xsd:schema>
  1. After that send the validated message to destination JMS queue using a Spring JMS Egress Connector. Specify ActiveMq-jmsTemplate as the JMS template and the destination queue name as well.

  2. In case of XSD validation failure of the XML payload the meta information file should be moved to an error directory location. To handle that scenario it is required to add File Egress Connector to the XSD Validator elements on failed validation out port.

  3. After that add an Exception flow end element to exceptionally complete the flow

The completed integration flow should look like below.

file polling complete flow

Configuration for each element is as below. The numbering corresponds to the numbers shown in above diagram.

Design View

Text View

.

1. File Ingress Connector (Basic Tab)

file polling component 1

1. File Ingress Connector (File Modifications Tab)

file polling component 1 1

1. File Ingress Connector (Scheduling Tab)

file polling component 1 2

2. XSD Validator

file polling component 2

3. Exceptional Flow End

file polling component 3

4. JMS Egress Connector

file polling component 4

5. File Egress Connector

file polling component 5
.

1. File Ingress Connector (Basic Tab)

File Path

resources/meta-files

File Name Pattern

1. File Ingress Connector (File Modifications Tab)

Wait After Modification

2

Remove Original File

true

Move After Process

/tmp/processed

Move After Failure

/tmp/failure

1. File Ingress Connector (Scheduling Tab)

Polling CRON Expression

0 11 12 ? * MON-FRI

Polling Start Delay

1000

Polling Repeat Interval

1000

Polling Repeat Count

-1

Concurrent Polling Count

1

Concurrent Execution Limit

4

2. XSD Validator

XSD file Paths

metaValidator.xsd

3. Exceptional Flow End

Exception Message

file polling validation error

4. JMS Egress Connector

JMS Template

activeMq-jmsTemplate

JMS Destination

invoiceMetaDestQueue

5. File Egress Connector

File Path

/tmp/failed

.

Now you can run the Ultra Project and check the functionality of the integration flow. Create an UltraESB Server run configuration and start it.

Property Configuration

When running the sample in the UltraESB-X distribution, you need to override the following properties in-order for the sample to work. The properties file is located at $ULTRA_HOME/conf/projects/file-polling-with-xml-validation/default.properties

Refer to Managing Project Properties documentation on how to override properties.

file-polling-with-xml-validation-flow.fileListener.filePath

Directory path to listen and poll files from (Default value is resources/meta-files)

file-polling-with-xml-validation-flow.fileListener.cronExpression

Schedule expression to poll for files from configured directory (Default value is 0 11 12 ? * MON-FRI)

file-polling-with-xml-validation-flow.fileListener.fileNamePattern

File name pattern to be used in polling files (Default value is ..xml*)

file-polling-with-xml-validation-flow.fileListener.moveAfterProcess

Directory path to copy input files after complete processing (Default value is /tmp/processed)

file-polling-with-xml-validation-flow.fileListener.moveAfterFailure

Directory path to copy input files after processing failure (Default value is /tmp/failure)

file-polling-with-xml-validation-flow.fileListener.removeOriginalFile

Whether to keep or remove input file after processing from the input directory (Default value is TRUE)

file-polling-with-xml-validation-flow.xsd.xsdPaths

File path to the XSD file to validate the input XML (Default value is metaValidator.xsd)

activeMq.fileListener.brokerURL

Active MQ broker URL (Default value is tcp://localhost:61616)

activeMq.fileListener.userName

Active MQ username (Default value is admin)

activeMq.fileListener.password

Active MQ password (Default value is admin)

file-polling-with-xml-validation-flow.fileSender.filePath

File path to move the file upon validation failure (Default value is /tmp/failed)

file-polling-with-xml-validation-flow.jmsForFileListener.destination

JMS queue name to be used at egress end (Default value is invoiceMetaDestQueue)

In order to run this sample it is mandatory to place the required files in your local file system and configure their file paths as defined in the following table

file-polling-with-xml-validation-flow.fileListener.filePath

Meta information XML file with the content

<?xml version="1.0" encoding="UTF-8"?>
<invoice-meta-info xmlns="http://www.adroitlogic.org/gringotts" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                   xsi:schemaLocation="http://www.adroitlogic.org/gringotts_metainfo.xsd">
    <subject>monthly receipt</subject>
    <receiver-id>Hogsmeade</receiver-id>
    <receiver-address>https://hogsmeade.com/service/finance-receipt</receiver-address>
    <payload>
        <path>{file-path-to-input-file01}</path>
        <path>{file-path-to-input-file02}</path>
    </payload>
</invoice-meta-info>

file-polling-with-xml-validation-flow.fileListener.moveAfterProcess

Directory path to copy input files after complete processing

file-polling-with-xml-validation-flow.fileListener.moveAfterFailure

Directory path to copy input files after processing failure

file-polling-with-xml-validation-flow.fileSender.filePath

File path to move the file upon validation failure

file-polling-with-xml-validation-flow.validate.xsdPaths

File path to the XSD file to validate the input XML

After that navigate to $ULTRA_HOME/bin directory. Next you can run the UltraESB-X distribution with following command to start the engine with this sample project deployed.

./ultraesbx.sh -sample file-polling-with-xml-validation

Testing the Integration Project

Now after running the project, send the sample message by placing a meta information file into the defined location. A sample content of a meta information file is provided below. The payload will be sent to the JMS queue after successful validation.

<?xml version="1.0" encoding="UTF-8"?>
<invoice-meta-info xmlns="http://www.adroitlogic.org/gringotts" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                   xsi:schemaLocation="http://www.adroitlogic.org/gringotts_metainfo.xsd">
    <subject>monthly receipt</subject>
    <receiver-id>Hogsmeade</receiver-id>
    <receiver-address>https://hogsmeade.com/service/finance-receipt</receiver-address>
    <payload>
        <path>{file-path-to-input-file01}</path>
        <path>{file-path-to-input-file02}</path>
    </payload>
</invoice-meta-info>

If you remove or change any field, then the XSD validation will be failed and the invoice meta information file will be moved to the user defined error file destination.

Now you can run the integration flow and check the file mediating working as expected. If you put a input file with .xml file name extension to the configured directory source location, your integration flow should fetch the file validate it over provided XSD and upload it to the target message queue.

In this topic
In this topic
Contact Us