Metrics Engine

UltraESB metrics engine has two main parts with one being the in-memory metrics engine and the other being ElasticSearch based metrics engine. More details will be in respective sections. Configuration of the metrics engine is in the ultra-metrics.xml file in conf/monitoring directory which is statically imported to ultra-root.xml. The import line of ultra-metrics is commented out by default. So to enable metrics engine first we have to uncomment that line. Let’s take a brief look at the main configuration beans in ultra-metrics.

<bean id="esMetricsCollector" class="org.adroitlogic.metrics.core.ESMetricsCollectorImpl">
    <property name="primaryMetricsTemplateValidTime" value="20000"/>
    <property name="metricsReportGenerationTimePeriod" value="30000"/>
    <property name="metricsTemplateSyncTimePeriod" value="20000"/>
    <property name="esStatisticsPublisher" ref="es-pub"/>
    <property name="metricsReportGenerationInitialDelay" value="20000"/>
    <property name="metricsTemplateSyncInitialDelay" value="10000"/>
</bean>

<bean id="in-memory-metrics-engine" class="org.adroitlogic.metrics.core.InMemoryMetricsCollectorImpl"/>

<bean id="metrics-engine" class="org.adroitlogic.metrics.core.MetricsEngineImpl">
    <constructor-arg ref="esMetricsCollector"/>
    <constructor-arg ref="in-memory-metrics-engine"/>
</bean>

Here we can see the bean esMetricsCollector bean which is for publishing statistics to elasticsearch and the bean in-memory-metrics-engine which is the in-memory metrics engine. You can use either one of these or both for statistics by the changing the constructor-args in the "metrics-engine" bean. More details of this configuration can be found in respective sections.

InMemory Metrics Engine

This guide describes the InMemory Metrics Engine used in UltraESB along with information related to the internal architecture of the the in-memory metrics engine.

Overall Architecture

UltraESB in-memory metrics engine is not a full-featured metrics implementation, rather it focuses more on delivering high performance, near real-time, light-weight solution for monitoring the system for a pre-configured window time. It uses a RRD-Tools like architecture to collect and store metrics records. However it DOES NOT persist the records, meaning that it is not capable of providing historical analysis of the metrics records or any sort of a long standing analytical features.

It is capable of delivering a summarized view of the metrics in a given window, where this window is configurable. It is NOT capable of giving out the absolute data points, as it keeps only a summarization within a pre-configured step time. The metrics engine uses a circular buffer like in-memory data structure to keep all the metrics records, where the circular buffer contains a set of objects, each of which represents a step time of the metrics engine. Step time is the least time window that the metrics engine is capable of reporting metrics summarizations. It is also considered as the general accuracy of the metrics summary that the engine is capable of producing. The amount of records that are there in this circular buffer is determined by the configured step time and the metrics window time.

metrics diagram

Any metrics data point reported to the in-memory metrics engine should have an metrics stream associated with it, which could possibly continue in all or a sub set of the step records in the metrics window. A given stream can have zero or more reports within a given step time. There are different metrics stream types which will be discussed in down below under the Metrics Stream Types section.

While the in-memory metrics engine keeps a reference to one of the steps in this buffer as its active record which is used to report any metrics data points by the engine. When a new step record is inserted (actually it will be reusing an existing step record in the underlying implementation to make it efficient) into the window, the most long standing record in the window will be evicted from the window, keeping absolutely a constant amount of step records in the window at any given time.

Both absolute data point recording and step record eviction can be intercepted by writing custom extensions to the in-memory metrics engine which we will be discussing in a later section of this guide.

Metrics Window and Step Time

The key parameters that controls the dimensions of the in-memory metrics engine and the resources consumed by the engine are the metrics window time and the step time. The metrics window can be defined as a moving window with the corresponding metrics records for a window time amount of past metrics records. So in effect the maximum history that the in-memory metrics engine is capable of providing metrics information is bounded by the metrics window time.

The step time is the lower bound of the metrics record history as the in-memory metrics engine keeps on accumulating the metrics data received to the engine for a step time before the record being retired to the moving window. The in-memory metrics engine cycles this process for any metrics streams reported to the in-memory metrics engine. Step time is also the least granularity that the in-memory metrics engine is capable of providing the metrics information, as it only keeps a summarized view of the step time, due to many reasons. Most important among these reasons are;

  • to provide metrics with the least possible memory foot print

  • avoid having to loop many records to present data into the presentation layer for monitoring

The limitations that are introduced by this summarization are the lose of actual data points and having a time window for any violation checks and alerting criteria. Since the in-memory metrics engine is optimized to run on this step time, any exceptional behaviors etc will only be notified in step time intervals, so you need to carefully analyze your requirements prior to selecting the step time and the window times to satisfy the non-functional requirements of the integration system.

For the configuration matters of the in-memory metrics engine it requires the step time and the number of steps, where as the number of steps (N) is given by the following formula, given that the metrics window time is (W) and the step time is (S).

Number of steps (N) = ceiling(W/S)

While a particular deployment may want more granular metrics information, which requires "S" to be minimized, certain other deployments might need more history, which requires "W" to be maximized. If you want both granular metrics information and a long history record as well, you could still configure the engine, but you need to be careful about the memory consumption, (Please refer to the section on Resource Consumption Formula) and the performance aspects too.

Note
You might have already understood from the architectural design of the in-memory metrics engine that these parameters are static for a single runtime instance, where it cannot be dynamically changed at runtime.

Ideal values for "S" and "W" for a production deployment are 3 seconds and 15 minutes meaning that the system will keep maximum 300 step records (N) in memory circular buffer to provide metrics history of 15 minutes, which is also the default configuration of the in-memory metrics engine.

Metrics Stream Types

There are different types of the metrics streams defined in the in-memory metrics engine, each of which is specialized in presenting data of specific nature. These metrics stream types are as follows;

Counter Metrics Streams

Counter streams just counts a particular event, or a occurrences of an action. Example counter stream would be the received messages count for a proxy service. Counters are heavily used in UltraESB to report metrics, it is taking the least amount of memory with compared to the other types of metrics streams and doesn’t have the histogram data, as it is naturally not meaning full. A counter record will expose the following metrics data about itself;

Data Name Description

count

The number of occurences

rate

Rate of the occurrences of an action as a per second value

Following operations can be invoked on a counter;

Operation Name Description

increment

Increases the action occurrence count

decrement

Decreases the action occurrence count

Gauge Metrics Streams

Gauge streams collects the information about the gauge of a particular object seen over the time. Example gauge stream would be the received message size via a given transport. Gauges are used in UltraESB mainly to report message sizes, system load, CPU utilizations and memory usage, it is taking a bit more memory with compared to the other types of metrics streams how ever it does have the histogram data, as it is very important to have a histogram of a gauge to understand the behavior of that particular gauge over the time. A gauge record will expose the following metrics data about itself;

Data Name Description

sum

Sum of the gauges reported in the complete step window

max

Maximum gauge value reported in this step window

min

Minimum none zero gauge reported in this step window

mean

Mathematical mean (average) of all the gauges reported in this step window

averageRate

Average data flow rate of the reported gauges in this step window

count

Number of gauges records in this step window

median

Statistical median of the recorded gauge distribution in this step window

75th percentile

75th percentile of the recorded gauge distribution in this step window

95th percentile

95th percentile of the recorded gauge distribution in this step window

98th percentile

98th percentile of the recorded gauge distribution in this step window

99th percentile

99th percentile of the recorded gauge distribution in this step window

99.9th percentile

99.9th percentile of the recorded gauge distribution in this step window

99.99th percentile

99.99th percentile of the recorded gauge distribution in this step window

Following operations can be invoked on a gauge;

Operation Name Description

add

Adds a gauge value point to the metrics stream

Timer Metrics Streams

Timer streams collects the information about the elapsed time of a particular action over the time. Example timer streams would be the incoming message processing time of a proxy service or the response time (round-trip time) of an endpoint. Timers are used in UltraESB mainly to measure the latencies, such as message processing time, response time, execution times etc.., it is very similar in memory consumption to that of a gauge records and have the histogram data. The histogram data including the percentiles plays a key role as it helps identify the outliers from the time series and find the 99% of the processing time etc.. such that the distribution gives the user an understanding of the general case leaving the exceptions out. A timer record will expose the following metrics data about itself;

Data Name Description

max

Maximum elapsed time reported in this step window

min

Minimum none zero elapsed time reported in this step window

mean

Mathematical mean (average) of all elapsed times reported in this step window

count

Number of timers recorded in this step window

median

Statistical median of the elapsed time distribution in this step window

75th percentile

75th percentile of the elapsed time distribution in this step window

95th percentile

95th percentile of the elapsed time distribution in this step window

98th percentile

98th percentile of the elapsed time distribution in this step window

99th percentile

99th percentile of the elapsed time distribution in this step window

99.9th percentile

99.9th percentile of the elapsed time distribution in this step window

99.99th percentile

99.99th percentile of the elapsed time distribution in this step window

Following operations can be invoked on a gauge;

Operation Name Description

reportTime

Reports an elapsed time data point to the metrics stream

Event Metrics Streams

Event streams collects the information about a set of events of a particular action over the time. Example event streams would be the message times event stream of a proxy service or the thread usage event stream of the work manager. Events streams are used in UltraESB mainly to measure relative average of an event w.r.t. a common base or a time difference between events in an event stream or  alert for exceptional events such as the first use of the secondary thread pool in the work manager etc.., the memory consumption of an event record is subjective to the number of possible events per stream. An event record will expose the following metrics data about itself;

Data Name Description

relativeAverage

The relative average of a particular event in the stream, with compared to the common base time

Metrics Engine Tuner

Tuner is designed to tune the in-memory metrics engine and its behavior. It does not directly affect the amount of recourses that the engine will utilize, rather it defines the operational behavior of the in-memory metrics engine. However, it may affect the resources, if you configure any of the metrics streams to be turned off. In-general the tuner doesn’t affect the major sizing parameters of the in-memory metrics engine.

The tuner can be configured to turn off the metrics window completely such the the metrics data points has no effect at all, or the metrics collection can be enabled while the metrics window is disabled, so that the recorded data points will be notified to any of the registered external data point listeners but the window will not be maintained in-memory.

Another feature of the tuner is the ability to map system default stream names to a user defined stream name. For example lets say the built-in stream for memory usage, which is "ue:memory:usedGauge" needs to be changed to something like, "app.memory-usage", that can be done by specifying a mapping from the default names to the custom name in the tuner. It is also possible to use wild cards for mapping, where you can map stream name that starts with "ue:" (default system streams) into "myApp." etc..

It can be configured to disable alerts completely from a single point. This will be of great use in cases where you might have different alerts configured to be triggered for critical events, and for an instance you do not want any of the alerts being triggered, for example for a performance test. For those circumstances you can directly disable all alerts from the in-memory metrics engine tuner.

Historical Analysis and Third-party Tools Integration

If you need historical analysis and heavy weight processing of metrics streams such as CEP on the metrics data, it is recommended that a third party solution to be integrated into the in-memory metrics engine. The core in-memory metrics engine facilitates this integration in 2 mechanisms, both with its own advantages as well as disadvantages. So it is the duty of the user to carefully select what you need in facilitating the integration of a third party system. These 2 integration points are;

Metrics Data Point Listener

Metrics engine reports the absolute metrics data points as and when it receives into the Data Point Listeners registered for it. This focuses more on the architecture of it.

There are few key aspects that you need to know architecturally about the data point listeners, first of that is the performance of a data point listener. Data point listener gets notified for absolute data points of the in-memory metrics engine, and is being invoked by the in-memory metrics engine itself. So to keep the lean behavior of the in-memory metrics engine, the data point listener should be designed in a manner that it performs the data point recording to the external party in the best possible fashion. If the third party system and the purpose of using the third party system is not time critical (no need to be real-time), one possible option would be to batch process the data points using a queue to accumulate data points and post in batches to the external system.

In an effort to provide a built-in mechanism to make this data point listeners asynchronous and not to affect the data flow of the main task, the UltraESB metrics implementation provides a Asynchronous version of the data point listener, where the asynchronous execution of the data point listener is guaranteed by the engine it self. If you need all data points to be recorded then and there to the external system you may use this option.

Metrics Record Eviction Handler

The eviction handler as the name depicts is called only when a given metrics step record is evicted out from the metrics window, so theoretically any subscriber listening for the metrics records with the eviction handler will receive records only after a window time, as the step record will be evicted after its life-cycle in the metrics window. One other limitation of the eviction handler is that it only receives the summarization of the step time and NOT absolute data points. So in a nut-shell, if you either need near real-time analysis or absolute metrics data points this is not the option for you. However this option has the advantage of the least overhead on the runtime, as it just reports the complete metrics activity set of a step time once and it is guaranteed that you will receive the summarized metrics records latests after a window time, and not before that.

This option is useful if you need some sort of a persistence layer for the metrics records averaged for a step time, where you can then analyze the historical data points about a given stream beyond your configured metrics window time, with an external tool to query the persisted metrics data points.

Alerting and Notifications

The near real-time, high performance nature of the in-memory metrics engine makes it very handy at alerting and notifications. Alerts are configured with a JSON formatted file which will be discussed in detail under the Alert Configuration section. Alerts also inherits the same summarization semantics of the core in-memory metrics engine, making it to be able to alert only in step time intervals. Once a step record completes its active life time, it gets stored in the metrics window for a window amount of time. In the mean time, each and every step record that is retired to the metrics window asynchronously evaluates the alert configurations associated with the metrics streams that it carries.

The asynchronous behavior of the alert processing makes sure that it adds the least possible overhead to the messages being processed (data flow) through the ESB. It also enables the ability to notify on alerts, as the alert generation and all alerts related activity is completely disconnected from the message processing. The disconnected nature makes sure it doesn’t affect the throughput of the ESB server, provided that the ESB is not running under 100% resource utilization.

To understand the alerts and the configuration of an alert it is very important to understand the alert functions.

Alert Functions

Alert functions are the element which defines the execution function of the metrics stream data over the defined criteria in the alert configuration. Different metrics record types are associated with different alert functions and the below table illustrates the association of the alert functions to the metrics record types.

Alert Function

Description

Applicability of Types

Counter

Gauge

Timer

Event

SUM

Sum of the values reported in the step

NO

YES

YES

NO

AVERAGE

Average of the values reported in the step

NO

YES

YES

NO

RATE

Rate of data flow in the step

YES

YES

NO

NO

COUNT

Number of data reports in step

YES

YES

YES

YES

MIN

Minimum (non zero) value reported in the step

NO

YES

YES

NO

MAX

Maximum (finite) value reported in the step

NO

YES

YES

NO

DIFF

Relative time difference between 2 events

NO

NO

NO

YES

Alert Severity

The alert severity can be configured per alert in the alert configuration, there are 5 alert severities defined for in-memory metrics engine alert configurations which are as follows; it is recommended that users keep the following criteria in mind when assigning a severity to an alert as that will help anybody understand the alerts importance etc.

Severity Name Description

FATAL

System has either encountered a state from which it cannot be recovered by itself

CRITICAL

Condition where the affect is critical into the live system

ERROR

Unintended behavior of the system

WARNING

Unexpected behavior that may be recovered by the system on its own

INFORMATION

No harm alert to notify the system has reached a certain milestone or an event

Resource Consumption Formula

This section presents a resource consumption formula for the in-memory metrics engine, from which it is very easy to derive a sizing metrics for the in-memory metrics engine of UltraESB.

Lets first of all list the general sizes of individual stream types;

Record Type General Memory Consumption (bytes) Total Consumption per Entry GMC + 16 (int key) + 32 (entry itself)

CounterRecord

48

48 + 48 = 96

TimerRecord

80

80 + 48 = 128

GaugeRecord

80

80 + 48 = 128

EventRecord

184

184 + 48 = 232

Now lets assume the following setup where there are artifacts of the following count;

Artifact Type Number of Items Counter Records Timer Records Gauge Records Event Records

Transport Senders + Listeners

T

4

 

2

 

Work Managers

W

4

 

 

1

File Caches

F

5

 

 

1

Proxy Services

P

6

2

 

 

Sequences

S

4

1

 

 

Endpoints

E

9

1

 

 

Interceptors

I

6

1

 

 

System

1

4

 

4

 

Based on the above table we can derive a formula for the memory consumption of records based on artifacts;

Memory Consumption per Step Time (M/N) = (4 x 96 + 2 x 128) T + (4 x 96 + 1 x 232) W + (5 x 96 + 1 x 232) F + (6 x 96 + 2 x 128) P + (4 x 96 + 1 x 128) S + (9 x 96 + 1 x 128) E + (6 x 96 + 1 x 128) I + 4 x 96 + 4 x 128

Memory consumption complete formula per step time

M/N = 640 T + 616 W + 712 F + 832 P + 512 S + 992 E + 704 I + 896

Now lets assume a general setup where there is only 1 file cache and 1 work manager (which is the 99% of the case) and assume the HTTP and HTTPS case which will result in 4 transports 2 listeners and senders, and no interceptors. Then the formula will be a function of deployment unit artifacts.

M = 640 x 4 + 616 x 1 + 712 x 1 + 832 P + 512 S + 992 E + 704 x 0 + 896

General memory consumption for HTTP/S per step time

M/N = 832 P + 512 S + 992 E + 4784

Total memory consumption of the in-memory metrics engine can now be derived as;

MTot = (832 P + 512 S + 992 E + 4784) x (N + 2)

where the N is the number of steps.

Deriving a sizing metrics based on this formula for services, sequences and endpoints etc.. for the default in-memory metrics engine configuration where the N is 300,

Proxy Services

Sequences

Endpoints

Metrics Records Theoretical Memory Consumption

Measured Memory Usage

Metrics Window

Complete Engine

2

5

4

3,918,752 = 3.7 MB

4,718,376 = 4.5 MB

9,730,296 = 9.3 MB

5

10

10

7,243,168 = 6.9 MB

8,374,632 = 8 MB

15,674,488 = 14.9 MB

5

10

50

19,226,528 = 18.3 MB

21,011,688 = 20 MB

33,755,768 = 32.2 MB

10

20

20

13,041,568 = 12.4 MB

14,230,696 = 13.6 MB

25,564,272 = 24.4 MB

100

200

200

117,412,768 = 112 MB

128,591,208 = 122.6 MB

212,584,272 = 202.7 MB

1000

2000

2000

1,161,124,768 = 1 GB

1000

10

2000

853,423,008 = 813.9 MB

900,374,152 = 858.7 MB

1,444,303,552 = 1.3 GB

Built-in Metrics Streams

UltraESB by default reports as many metrics streams as it can providing a good view of the system information to the users to monitor it. This is possible due to the lean architecture of the UltraESB metrics engine and the ability to configure the metrics window and the step time of your choice to match your requirements with a good memory profile and least performance overhead.

Value Metrics Streams

The reported value metrics streams are explained in the following table.

Repetetive Streams
Any stream name which contains ${name} will be repeated for each item of that category, for example in an UltraESB system which has 2 proxy services namely "foo" and "bar", the stream proxy:${name}:handledErrorInCount will be reported as 2 streams each with the name proxy:foo:handledErrorInCount and proxy:bar:handledErrorInCount
Category Stream Name Metrics Type Description

System Information

ue:memory:usedGauge

Gauge

Heap Memory usage of the UltraESB process reported in bytes

ue:system:loadAvgGauge

Gauge

CPU load average of the UltraESB process reported as X, where X/1000 gives the load average to 3 decimal points

ue:system:openFDGauge

Gauge

The file descriptor usage of the UltraESB process, in other words how many files being opened by UltraESB

ue:thread:activeGauge

Gauge

Number of active Java threads in the UltraESB, including system threads

ue:messages:receivedCount

Counter

Total number of messages received to the UltraESB irrespective of the transports or proxy services

ue:messages:sentCount

Counter

Total number of messages sent out by the UltraESB irrespective of the transports or proxy services

ue:fc:fileUsage

Counter

Global file cache usage of the UltraESB of all used file caches

ue:wm:threadUsage

Counter

Global all work manager thread usage (task executions) of the UltraESB

Transport Metrics

trp:${name}:receivedMessageCount

Counter

Number of received messages through this transport

trp:${name}:receivingFaultCount

Counter

Number of faults encountered while receiving messages through this transport

trp:${name}:receivedBytesGauge

Gauge

Amount of bytes received by this transport

trp:${name}:sentMessageCount

Counter

Number of messages sent by this transport

trp:${name}:sendingFaultCount

Counter

Number of faults encountered while sending messages via this transport

trp:${name}:sentBytesGauge

Gauge

Amount of bytes sent by this transport

Proxy Service Metrics

proxy:${name}:processedInCount

Counter

Number of incoming messages processed by this proxy service

proxy:${name}:processedOutCount

Counter

Number of outgoing messages processed by this proxy service

proxy:${name}:handledErrorInCount

Counter

Number of errors of incoming messages handled by this proxy service

proxy:${name}:handledErrorOutCount

Counter

Number of errors of outgoing messages handled by this proxy service

proxy:${name}:failedInCount

Counter

Number of incoming message un-handled failures in this proxy service

proxy:${name}:failedOutCount

Counter

Number of outgoing message un-handled failures in this proxy service

proxy:${name}:processingInTime

Timer

Amount of time it takes to process an incoming message in this proxy service reported in nano seconds

proxy:${name}:processingOutTime

Timer

Amount of time it takes to process an outgoing message in this proxy service reported in nano seconds

proxy:${name}:responseTime

Timer

Amount of time between message acceptance to response by the proxy service.

Sequence Metrics

seq:${name}:successCount

Counter

Number of messages that has been executed in this sequence successfully (without any errors)

seq:${name}:handledErrorCount

Counter

Number of messages that encounter an error which is handled with an error sequence, within this sequence

seq:${name}:unhandledErrorCount

Counter

Number of messages that encounter an error but has not been handled within this sequence

seq:${name}:totalProcessedCount

Counter

All messages that reached this sequence

seq:${name}:executionTime

Timer

The number of nano seconds it took for a message to execute within this sequence

Endpoint Metrics

ep:${name}:processedMessageCount

Counter

Number of messages that has reached this endpoint, including fail-over messages

ep:${name}:uniqueMessageCount

Counter

Number of unique messages that has reached this endpoint (excluding fail-over messages)

ep:${name}:failOverMessageCount

Counter

Number of fail-over messages that has reached this endpoint

ep:${name}:successfulMessageCount

Counter

Number of messages that got successfully delivered over this endpoint

ep:${name}:suspendErrorSendingMessageCount

Counter

Number of suspended errors that are encountered while sending the message to the remote EPR via this endpoint

ep:${name}:suspendErrorReceivingMessageCount

Counter

Number of suspended errors that are encountered while receiving the response from the remote EPR in this endpoint

ep:${name}:temporaryErrorSendingMessageCount

Counter

Number of temporary errors that are encountered while sending the message to the remote EPR via this endpoint

ep:${name}:temporaryErrorReceivingMessageCount

Counter

Number of temporary errors that are encountered while receiving the response from the remote EPR in this endpoint

ep:${name}:otherErrorsCount

Counter

Number of other uncategorized errors that are encountered within this endpoint

ep:${name}:responseTime

Timer

The response (round-trip) time of this endpoint

File Cache Metrics

fc:${name}:fileCacheUsage

Counter

Number of files used from this file cache

fc:${name}:fileCacheOverflow

Counter

Number of files that overflowed to disk from this file cache

fc:${name}:createdFiles

Counter

Number of created files within the file cache

fc:${name}:filesInUse

Counter

Number of files currently acquired for processing from this file cache

fc:${name}:overflowFilesInUse

Counter

Number of files currently acquired for processing as overflowed files from this file cache

Work Manager Metrics

wm:${name}:primaryThreadUsage

Counter

Number of tasks that got invoked with primary threads from this work manager

wm:${name}:secondaryThreadUsage

Counter

Number of tasks that got invoked with secondary threads from this work manager

wm:${name}:wipMessages

Counter

Number of messages that are currently being processed (work-in-progress) by a thread of this work manager

wm:${name}:messageQueueSize

Counter

Number of messages that are queued to be executed by this work manager

Interceptor Metrics

int:${name}:processedCount

Counter

Number of interceptor invocations

int:${name}:successCount

Counter

Successful interceptor invocations without and error

int:${name}:handledErrorCount

Counter

Number of errors that got handled by an error handler in an interceptor

int:${name}:unhandledErrorCount

Counter

Number of errors that are not handled in an interceptor

int:${name}:acceptedCount

Counter

Number of accepted invocations by the interceptor (# of return true, times)

int:${name}:rejectedCount

Counter

Number of rejected invocations by the interceptor (# of return false, times)

int:${name}:executionTime

Timer

Execution time of the interception

Event Metrics Streams
Category Stream Name Event Name Description

Proxy Service Metrics

proxy:${name}:messageTimes

proxy:in:start

Start time of the incoming message processing relative to the base time within this proxy service

proxy:in:end

End time of the incoming message processing relative to the base time within this proxy service

proxy:out:start

Start time of the outgoing message processing relative to the base time within this proxy service

proxy:out:end

End time of the outgoing message processing relative to the base time within this proxy service

in:seq:${name}:start

Start time of the in-sequence execution of this proxy service. Note that if you call another sequence within the proxy in sequence this event will be triggered twice with the ${name} changed for the 2 sequences

in:seq:${name}:end

End time of the in-sequence execution of this proxy service. Note that if you call another sequence within the proxy in sequence this event will be triggered twice with the ${name} changed for the 2 sequences

out:seq:${name}:start

Start time of the out-sequence execution of this proxy service. Note that if you call another sequence within the proxy out sequence this event will be triggered twice with the ${name} changed for the 2 sequences

out:seq:${name}:end

End time of the out-sequence execution of this proxy service. Note that if you call another sequence within the proxy out sequence this event will be triggered twice with the ${name} changed for the 2 sequences

in:ep:${name}:start

Start time of the in-destination processing. If there are more than one in destinations all respective destinations will trigger this event.

in:ep:${name}:end

End time of the in-destination processing. If there are more than one in destinations all respective destinations will trigger this event.

out:ep:${name}:start

Start time of the out-destination processing. If there are more than one out destinations all respective destinations will trigger this event.

out:ep:${name}:end

End time of the out-destination processing. If there are more than one out destinations all respective destinations will trigger this event.

Sequence Metrics

seq:${name}:messageTimes

seq:start

Sequence execution start time relative to the base time

seq:end

Sequence execution end time relative to the base time

Endpoint Metrics

ep:${name}:messageTimes

ep:start

Endpoint processing start time relative to the base time

ep:end

Endpoint processing end time relative to the base time

File Cache Metrics

fc:${name}:fileCacheState

Work Manager Metrics

wm:${name}:threadPoolEvents

ElasticSearch based Metrics Engine

Elasticsearch based metrics engine publishes stats to elasticsearch. Elasticsearch is a search engine based on Lucene which provides a distributed, multitenant-capable full-text search engine with an HTTP web interface and schema-free JSON documents. Stats published to elasticsearch can be monitored through AdroitLogic Integration Monitor (IMonitor), which is an independent Web Application.

AdroitLogic Integration Monitor - IMonitor
AdroitLogic Integration Monitor executes as an independent Web Application, and allows the easy management of a single UltraESB instance or a cluster of instances. Let it be a single instance or a cluster of ESB nodes, IMonitor delivers business level statistics and monitoring at the best. Apart from the operational statistics, IMonitor is capable of presenting friendly troubleshooting & diagnostics capabilities. It’s your step towards improved organisational efficiency saving hours of developer time. Note that IMonitor comes as a replacement for UConsole which was there in previous UltraESB releases and is covered separately in AdroitLogic - Integration Monitor User Guide.

Quite similar to in-memory metrics engine, any metrics data point reported to the ElasticSearch based metrics engine should have an metrics stream associated with it and all the metrics streams of type counter stream and gauge stream reported to the in-memory metrics engine are reported to the ElasticSearch based metrics engine.

Configuration

Following are the configuration beans related to ElasticSearch based metrics engine which reside in in ultra-metrics.xml. As you can see below there are three beans in the configuration.

  • es-server - ElasticSearch server configuration

  • esMetricsCollector - Metrics collection component in ElasticSearch based metrics engine

  • es-pub - Metrics publishing component in ElasticSearch based metrics engine

<bean id="es-server" class="org.adroitlogic.metrics.core.InternalESStatisticsServer"/>

<!-- If you want to use an standalone elasticsearch server for statistics, uncomment this bean and
make sure to comment the above bean "es-server" which is for internal (embedded) es server -->
<!--bean id="es-server" class="org.adroitlogic.metrics.core.ExternalESStatisticsServer">
    <constructor-arg>
        <map>
            <entry key="localhost" value="9300"/>
        </map>
    </constructor-arg>
    <constructor-arg value="elasticsearch" type="java.lang.String"/>
</bean-->

<bean id="es-pub" class="org.adroitlogic.metrics.core.ElasticSearchStatisticsPublisher">
    <property name="esStatisticsServer" ref="es-server"/>

    <property name="esIndexName" value="ultraesb"/>
    <property name="esRecordType" value="msg_statistics"/>
    <property name="esMetricsDefaultTTL" value="15m"/>
    <property name="waitForYellowTimeoutInMillis" value="5000"/>
    <property name="awaitCloseTimeoutInMillis" value="10000"/>
    <!--<property name="esMetricsDefaultTTL" value="15m"/>-->
    <property name="numberOfShards" value="1"/>
    <property name="numberOfReplicas" value="1"/>
    <property name="indexRefreshInterval" value="5s"/>

    <!-- Elasticsearch BulkProcessor configuration -->
    <property name="bulkActions" value="1000"/>
    <property name="bulkSizeInMB" value="5"/>
    <property name="flushIntervalInSeconds" value="60"/>
    <property name="concurrentRequests" value="1"/>
    <property name="backOffExponentialTimeinMillis" value="100"/>
    <property name="backOffExponentialRetryAttempts" value="3"/>
    <property name="optimized" value="true"/>
</bean>

<bean id="esMetricsCollector" class="org.adroitlogic.metrics.core.ESMetricsCollectorImpl">
    <property name="primaryMetricsTemplateValidTime" value="20000"/>
    <property name="metricsReportGenerationTimePeriod" value="30000"/>
    <property name="metricsTemplateSyncTimePeriod" value="20000"/>
    <property name="esStatisticsPublisher" ref="es-pub"/>
    <property name="metricsReportGenerationInitialDelay" value="20000"/>
    <property name="metricsTemplateSyncInitialDelay" value="10000"/>
</bean>
ElasticSearch Server Cofiguration

The es-server bean is the ElasticSearch server configuration for which the metrics will be published. As you can see above there are two configurations for "es-server" with one commented out. The first "es-server" bean (the uncommented one) is for the embedded ElasticSearch server shipped with UltraESB 2.6 release. The embedded es server is mainly for integration test purposes. ES yaml configuration for the embedded server can be found at conf/monitoring directory.

Use stand-alone ElasticSearch servers in production
It is recommended to use stand alone ElasticSearch server in production use.

The commented out "es-server" bean configuration is the configuration to publish statistics to an stand-alone ElasticSearch server. First and stand-alone ES server must be configured for the UltraESB to publish statistics to that. After setting up an external ES server the host and the port which the external server is listening to and the cluster name of the external server must be provided in the bean configuration.

Publishing Metrics to ElasticSearch

The bean es-pub is responsible for configuring the component which published statistics to ElasticSearch. Understanding this might require prior knowledge on ElasticSearch and hence only a few most important points will be covered here.

esIndexName

UltraESB statistics will be published under this index name and in relational database perspective it is something close to database name.

esRecordType

This is another level of categorization and this can be somewhat thought of as a table in a relational database. Please note this association with relational databases is just for general understanding purposes and how ElasticSearch and relational databases work are much more apart.

esMetricsDefaultTTL

We can give a TTL value for the statistics which are published to ElasticSearch. In the default configuration it is set to 15 minutes.

Do not delete ElasticSearch data directory manually
Deleting ElasticSearch data directory manually may cause issues in ElasticSearch and it is not the recommended way to deleting data. Refer this guide from ElasticSearch documentation on how to delete an index.

Bulk Processor Configuration

ElasticSearch Bulk Processor is in operation when publishing statistics in ES based metrics engine. This configuration allows to configure and tune the ES Bulk Processor along with some custom optimizations. For more details on Bulk Processor configuration parameters please refer this guide from ElasticSearch documentation.

In this topic
In this topic
Contact Us