metrics

Metrics and Statistics

Version: 17.07

Overview

Let it be an Enterprise Service Bus or any other integration product, more often than not it is required to collect metrics and statistics on different aspects of the product. Adroitlogic Project-X caters this requirement by providing built-in metrics collection and publishing capabilities with the flexibility to plugin a metrics engine only if it is required.

Project-X currently provides a metrics engine implementation based on the popular Elasticsearch analytics engine and the user has the freedom to either use an external Elasticsearch server or use the embedded Elasticsearch server of Project-X to collect statistics.

The metrics collector collects all the reported metrics values inside a configurable metrics window and at the end of the metrics window, it calculates an aggregated value for each metrics stream and reports to the metrics publisher. In configurable time intervals, the metrics publisher publishes these reports as a bulk to the underlying statistics server. The algorithm used by the metrics collector to calculate the aggregated value depends on the Record Type of the metrics stream.

Metrics Record Types

Project-X metrics engine currently supports the following metrics record types and based on these types.

Counter

Counter type should be used for metrics streams that are countable one by one. Metrics streams of this type start with the value 0 and can be either incremented by 1 or decremented by 1 at a time. The final count at the end of metrics window will be considered as the aggregated value and it can be either positive or negative based on the number of increment operations and decrement operations performed.

For example, this record type can be used for a metrics stream that counts the number of connections provided by a database connection pool. A metrics stream with Counter type can be initialized and it can be incremented at each time a connection is borrowed from the pool and can be decremented each time a connection is returned to the pool.

Counter Map

Counter Map type is very much similar to the Counter type, but it can be used to keep several counters under a single metrics stream. Unlike in Counter type where a single value is either incremented or decremented, Counter Map type has set of counters each with a unique key and any one of them can be incremented or decremented independently providing the corresponding key with the increment or decrement operation.

For example, this record type can be used for a metrics stream that counts the different types of HTTP response codes received by a HTTP egress connector. A metrics stream with Counter Map type can be initialized and at each time a response is received, the increment operation can be called providing the received response code as the key. Ultimately this would result in a metrics record that has the response codes and the number of times each response code is received as key value pairs.

Summation Gauge

Gauge type can be used for a metrics streams that reports a different value each time. This reporting value can be either a whole number of a decimal. But it is always assumed that this reported value to be non-negative and any negative values reported for a Gauge type metrics stream will be ignored.

In the Summation Gauge, the summation of all the reported values at the end of metrics window will be considered as the aggregated value.

For example, this record type can be used for a metrics stream that record the number of bytes received by an ingress connector. A metrics stream with Summation Gauge type can be initialized and at each time a message is received to the connector, its size in bytes can be reported to the metrics stream. Ultimately the aggregated value reported by this metrics stream would be the total number bytes received by the ingress connector within that metrics window.

Average Gauge

Average Gauge type is very much similar to the Summation Gauge type, except for the fact that the average of all the reported values at the end of metrics window will be considered as the aggregated value.

For example, this record type can be used for a metrics stream that record CPU utilization of an UltraESB-X instance. A metrics stream with Average Gauge type can be initialized and a timer task can be configured to report the current CPU utilization in regular time intervals. Ultimately the aggregated value reported by this metrics stream would be the average CPU usage of the instance within that metrics window.

Metrics Configuration

Enabling Metrics

To enable Elasticsearch based metrics, the easiest way is to use an UltraESB-X complete distribution which contains all the required dependencies and configuration. Extract the distribution to a location of your choice and navigate to the file conf/server.xml. In the file, make sure that the following spring import statement is present and it is not in a commented state.

<import resource="metrics/metrics.xml"/>

All the metrics related configurations of Project-X are enclosed in the conf/metrics/metrics.xml file. Open that file and configure the parameters related to Elasticsearch Server, Metrics Publisher, Metrics Collector and Metrics Engine as follows.

Configuring Elasticsearch Server

As mentioned before, Project-X provides the capability to either use its embedded Elasticsearch server or to use an external standalone Elasticsearch server to collect statistics. To enable the embedded Elasticsearch server, configure the es-server bean as follows.

<bean id="es-server" class="org.adroitlogic.x.metrics.EmbeddedElasticsearchServer"/>

This embedded Elasticsearch sever uses the configuration file conf/metrics/elasticsearch.yml and can be fine-tuned by altering the parameters in that file. Please refer Elasticsearch Documentation on these configurations.

If one or more external Elasticsearch servers are being used, es-server bean should be configured as below.

<bean id="es-server" class="org.adroitlogic.x.metrics.ExternalElasticsearchServer">
    <constructor-arg>
        <map>
            <entry key="server1" value="9300"/>
            <entry key="server2" value="9300"/>
        </map>
    </constructor-arg>
    <constructor-arg value="es-cluster" type="java.lang.String"/>
</bean>
Property Description Default Value

Constructor Arguments

Map with the hostname and connection port of each external Elasticsearch server as key-value pairs

localhost:9300

Name of the Elasticsearch cluster

elasticsearch
The embedded Elasticsearch server of Project-X is intended only to be used for the development/testing purposes and it is highly recommended to use one or more external Elasticsearch servers in production deployments.

Configuring Metrics Publisher

The Elasticsearch based Metrics Publisher of Project-X can be configured as below.

<bean id="es-publisher" class="org.adroitlogic.x.metrics.ESStatisticsPublisher">

    <constructor-arg ref="es-server"/>

    <property name="esHealthTimeoutMillis" value="5000"/>

    <!--Template & Alias Configuration-->
    <property name="templateFile" value="ultraesb-x-template.json"/>
    <property name="inactiveSettingFile" value="inactive-index-setting.json"/>
    <property name="indexTemplateName" value="ultraesb_x_template"/>
    <property name="initialIndexName" value="ultraesb_x-000001"/>
    <property name="activeIndexAliasName" value="active_ultraesb_x"/>
    <property name="inactiveIndexAliasName" value="inactive_ultraesb_x"/>
    <property name="indexMappingType" value="x_statistics"/>

    <!--BulkProcessor Configuration-->
    <property name="bulkActions" value="1000"/>
    <property name="bulkSizeInMB" value="5"/>
    <property name="flushIntervalInSeconds" value="60"/>
    <property name="concurrentRequests" value="1"/>
    <property name="backOffExponentialTimeInMillis" value="100"/>
    <property name="backOffExponentialRetryAttempts" value="3"/>
    <property name="optimized" value="true"/>

    <!--RollOver Configuration-->
    <property name="maxDocs" value="1000"/>
    <property name="maxAgeInMinutes" value="15"/>
    <property name="rollOverTimePeriodInMinutes" value="10"/>
    <property name="indexRollOverEnabled" value="true"/>
    <property name="deleteOldestIndexAfterRollOver" value="true"/>
    <property name="deleteInactiveIndicesAtInit" value="false"/>
    <property name="deleteInactiveIndicesAtDestroy" value="false"/>

</bean>
Property Description Default Value

Constructor Arguments

Reference to the Elasticsearch Server bean configured in the previous step

esHealthTimeoutMillis

Time in milliseconds to wait until Elasticsearch cluster status becomes yellow

5000

templateFile

Name of the JSON file containing the index template which includes the mapping, setting and aliases

It is recommended not to change the provided index template unless absolutely necessary

ultraesb-x-template.json

inactiveSettingFile

Name of the JSON file containing the setting which will be set to indices which are rolled over.

inactive-index-setting.json

indexTemplateName

Name for the index template in the JSON file pointed by templateFile property.

ultraesb_x_template

initialIndexName

If an ultraesb index does not exist at the startup, a new index will be created. This property represents that initial index name.

ultraesb_x-000001

activeIndexAliasName

Elasticsearch aliases are used to separate out active indices and inactive indices. At any point active alias will only point to one index and that index will be used for indexing new documents. Inactive alias points to indices which are rolled over. Inactive indices will only be used for querying. This property represents name for the active index alias.

active_ultraesb_x

inactiveIndexAliasName

This property represents name for the inactive index alias.

inactive_ultraesb_x

indexMappingType

Name of the mapping type to be used for statistics publishing

x_statistics

bulkActions

The number of currently added actions to flush a new bulk publishing request. Can be set to -1 to disable flushing based on number of actions.

1000

bulkSizeInMB

The size of currently added actions in Mega Bytes to flush a new bulk publishing request. Can be set to -1 to disable flushing based on the size of actions.

5

flushIntervalInSeconds

The time interval in seconds to flush bulk publishing requests

60

concurrentRequests

The number of concurrent bulk publishing requests allowed to be executed. A value of 0 means that only a single request will be allowed to be executed and a value of 1 means 1 concurrent request is allowed to be executed while accumulating new bulk requests.

1

backOffExponentialTimeInMillis

In case of bulk request failure, the Exponential Backoff Policy will be used to retry the failed request. This property specifies the initial waiting time in milliseconds before the first retry attempt.

100

backOffExponentialRetryAttempts

In case of bulk request failure, the Exponential Backoff Policy will be used to retry the failed request. This property specifies the maximum number of retry attempts.

3

optimized

Setting optimized to true will attempt to optimize indexing performance of bulk requests. Basically this will set index refresh internal to -1 and number of index replicas to 0 before each bulk import and revert back to normal values after the bulk operation. For more technical details related to this optimization read Indexing Performance Tips in elasticsearch documentation.

This optimization will only be useful in cases where the bulk request size is huge.

false

indexRollOverEnabled

This enabled elasticsearch index roll over which means once any of the given conditions becomes true elasticsearch indexed will be rolled ever and a new index will be created. For more technical details related to this read Roll Over Documentation

true

maxDocs

This is one of the roll over conditions which represents the number of documents in the index.

1000

maxAgeInMinutes

This is the other condition for elasticsearch rollover. It represents the age of the index in minutes (time elapsed since the index was created).

15

rollOverTimePeriodInMinutes

Index roll over is done using a timer task which periodically call the roll over API. This property can be used to set the time interval between two roll over calls.

10

deleteOldestIndexAfterRollOver

When this property is set to true, the oldest rolled over index will be deleted. Say we start with ultraesb_x-000001 and after one rollover cycle we have ultraesb_x-000001 and ultraesb_x-000002. Now when the next rollover cycle completes, which mean once ultraesb_x-000003 is created we deletes the oldest index which is ultraesb_x-000001.

true

deleteInactiveIndicesAtInit

If this is set to true, we will delete all the indices which are marked with inactive_ultraesb_x alias at es-publisher init.

false

deleteInactiveIndicesAtDestroy

If this is set to true, we will delete all the indices which are marked with inactive_ultraesb_x alias at es-publisher destroy.

false

Configuring Metrics Collector

The Elasticsearch based Metrics Collector of Project-X can be configured as below.

<bean id="es-collector" class="org.adroitlogic.x.metrics.ESMetricsCollector">

    <constructor-arg ref="es-publisher" />
    <property name="primaryMetricsTemplateValidTime" value="20000"/>
    <property name="metricsReportGenerationTimePeriod" value="30000"/>
    <property name="metricsTemplateSyncTimePeriod" value="20000"/>

    <property name="customFields">
        <map>
            <entry key="key1" value="constant_value"/>
            <entry key="key2" value="env:env_var_name"/>
            <entry key="key3" value="sys:sys_prop_key"/>
        </map>
    </property>
</bean>
Property Description Default Value

Constructor Arguments

Reference to the Elasticsearch Metrics Publisher bean configured in the previous step

primaryMetricsTemplateValidTime

The duration in milliseconds within which a primary metrics template is valid from the time of its creation or from the time of its last reset

60000

metricsReportGenerationTimePeriod

The time interval in milliseconds in which the metrics reports should be generated and send to the Metrics Publisher

60000

metricsTemplateSyncTimePeriod

The time interval in milliseconds in which the primary and secondary metrics template syncing is performed.

60000

customFields

This parameter can be specified as a map to include any additional fields in each metrics record. The key of the entry will be the key of the metrics record field. The value of the metrics record field can be either a constant value, an environment variable value or a system property value.

To use an environment variable as the value, the value of the map entry should be of the format env:env_var_key where env_var_key is the key of the environment variable. To use a system property as the value, the value of the map entry should be of the format sys:sys_prop_key where sys_prop_key is the key of the system property.

Configuring Metrics Engine

The Elasticsearch based Metrics Engine of Project-X can be configured as below.

<bean id="metrics-engine" class="org.adroitlogic.x.metrics.ESMetricsEngine">
    <constructor-arg ref="es-collector"/>
</bean>
Property Description Default Value

Constructor Arguments

Reference to the Elasticsearch Metrics Collector bean configured in the previous step

Built-in Metrics Streams

The following are the built-in metrics streams of Project-X and they will be reported automatically when metrics are enabled.

Category Stream Name Record Type Description

System Information

cpuUsageGauge

Average Gauge

Average CPU utilization of the instance as a value out of 100

usedMemoryGauge

Average Gauge

Heap memory utilization in bytes

usedMemoryPercentageGauge

Average Gauge

Heap memory utilization as a percentage of maximum available heap memory

activeThreadsGauge

Average Gauge

Average number of live threads including both daemon and non-daemon threads

openFDGauge

Average Gauge

Average number of open file descriptors

Scale-First Executor Service Metrics

activeThreadCount

Average Gauge

Average number of threads that are actively executing tasks

currentPoolSize

Average Gauge

Average number of threads in the pool

currentQueueLength

Average Gauge

Average length of the task queue

Integration Flow Metrics

executionSuccess

Counter

Number of successfully completed message flows

executionFailed

Counter

Number of failed messages flows

Ingress Connector Metrics

receivedMsgCount

Counter

Number of messages received by the ingress connector

receivedMsgSize

Summation Gauge

Total size of the messages received by the ingress connector

sentMsgCount

Counter

Number of messages sent by the ingress connector as responses

This metrics stream will be only reported by In-Out type ingress connectors such as HTTP.

sentMsgSize

Summation Gauge

Total size of the messages sent by the ingress connector as responses

This metrics stream will be only reported by In-Out type ingress connectors such as HTTP.

Egress Connector Metrics

sentMsgCount

Counter

Number of messages sent by the egress connector

sentMsgSize

Summation Gauge

Total size of the messages sent by the egress connector

receivedMsgCount

Counter

Number of messages received by the egress connector as responses

This metrics stream will be only reported by bi-directional type egress connectors such as HTTP.

receivedMsgSize

Summation Gauge

Total size of the messages received by the egress connector as responses

This metrics stream will be only reported by bi-directional type egress connectors such as HTTP.

sendingSuccess

Counter

Number of successfully sent out messages

sendingFailed

Counter

Number of sending failed messages
In this topic
In this topic
(c) Copyright 2016-2017 AdroitLogic. All rights reserved.
Terms of Use
Contact Us