Production Deployment Best Practices

This document describes few notes on doing a production deployment.

Operating System and Hardware

Operating system and hardware configuration is a major fact to be considered for a serious production deployment.

Linux Operating system using a Kernel 2.6 or later

The UltraESB will perform best on a Linux operating system using a Kernel of version 2.6 or later, where the epoll facility is available for Java NIO. On systems without support for epoll, the NIO subsystem will fall-back to using a traditional poll-based selector. A Linux Kernel 2.1 or later is required to enable the use of the sendfile system call for zero-copy proxying with DMA. AdroitLogic recommends Debian / Ubuntu Linux servers as the most suitable OS for the UltraESB, followed by RHEL/CentOS.

The following system configurations should be performed to enable best performance

/etc/sysctl.conf
net.ipv4.ip_local_port_range = 1024 65535
net.ipv4.tcp_fin_timeout = 30
fs.file-max = 2097152
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_syncookies = 0
net.core.somaxconn = 1024

/etc/security/limits.conf
* soft nofile 65535
* hard nofile 65535

/etc/pam.d/common-session
session required pam_limits.so

Log files should generally be located on a local disk drive.

Multiple Network cards maybe used. Ensure that Gigabit Networking is enabled and that the Network card offloading capabilities are enabled.

Selecting the UltraESB Distribution

It is recommended that the Minimal and Management Distributions be used to set-up a production deployment. You would receive the direct links to download each of the distributions when you register to download either distribution once. The minimal distribution is optimized for the most common production deployment scenario, and excludes documentation, samples and optional libraries or management utilities etc. The management utilities (i.e. IMonitor and UTerm) can be download separately and installed on any machine/s requiring access to manage the production nodes. If the production deployment requires any optional libraries (e.g. for JMS, JDBC, transformations etc), these JAR files needs to be made available at the ULTRA_HOME/lib/custom directory of each production node.

UltraESB configuration

The UltraESB configuration maybe optimized and tuned with the following options in general. However, it should be noted that any custom code developed by a user should be adequately tested for performance and memory utilization etc before being deployed onto a production instance, as a failing user component has the potential to put down an ESB node.

Thus the usual practice should be to test the user components for functionality, performance and memory usage, and stress testing, followed by monitoring the system and tuning aspects to suit the workload the the use cases handled by the system.

Tuning of the file cache

The RAMDiskFileCache should be used for production deployments, and should approximately provide adequate files for the concurrency level expected. (e.g. supporting 1024 concurrent HTTP requests and responses would require approximately 2048 files)

Creating the RAM disk at boot time

To create the RAM disk at boot time, define your RAM disk parameters in the /etc/fstab. An example configuration that creates a 512M disk mounted at /tmp/ram on Debian/Ubuntu is as follows. Note: It is recommended that the RAM disk be mounted at /tmp/ram so that in-case of a mount failure at machine restart, an error would be reported at ESB start-up time due to the non-availability of the /tmp/ram path as it is at the temporary directory which will be empty.

tmpfs  /tmp/ram  tmpfs   defaults,size=16M
Changing environment to "prod" in ultra-root configuration

Environment for UltraESB which controls the runtime behavior of the system. There are few pre-defined environments named as follows with the respective properties.

Environment Name Binary Class Reloading Configuration Auto Refresh

DEV

Enabled

Disabled

IDE

Enabled

Enabled

UNIT_TEST

Disabled

Disabled

SAMPLE

Disabled

Disabled

TEST

Enabled

Disabled

STAGE

Enabled

Disabled

PROD

Enabled

Disabled

When deploying UltraESB in production, environment should be marked as prod. This can be done by changing the constructor argument in the bean named "environment" in ultra-root.xml as shown below. Note that this value is case insensitive so "PROD", "prod", "Prod" all represent same.

    <bean id="environment" class="org.adroitlogic.ultraesb.api.Environment">
        <constructor-arg value="prod" type="java.lang.String"/>
        <property name="onTheFlyCompilationEnabled" value="true"/>
    </bean>

Advanced Configuration for Production Deployments

Its advisable to run the UltraESB process under a non-privileged user account. On Linux systems you could create a new account without the login privilege as follows

Ubuntu/Debian:     sudo adduser ultraesb --disabled-login
CentOS/RHEL:       sudo adduser ultraesb
                   sudo usermod -s /sbin/nologin ultraesb
Configuring Automatic Startup/Shutdown

You may create a script to start the UltraESB at boot time. On Debian/Ubuntu systems, this involves creating an LSB script at /etc/init.d as described below. First create a file /etc/init.d/ultraesb as root with the following contents

#!/bin/sh -e
### BEGIN INIT INFO
# Provides:          ultraesb
# Required-Start:    $local_fs $remote_fs $network $syslog
# Required-Stop:     $local_fs $remote_fs $network $syslog
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# X-Interactive:     true
# Short-Description: Start/stop AdroitLogic UltraESB
### END INIT INFO
#
# ultraesb        This init.d script is used to start the UltraESB
#
APP=ultraesb
#############################################################
#                      DEFINE THE FOLLOWING                 #
#############################################################
# Name of the user to run as
USER=ultraesb
# Location of application's bin directory
ULTRA_HOME=/opt/ultraesb
# Location of Java JDK with the ** unlimited strength ** policy files applied
export JAVA_HOME=/opt/jdk
case "$1" in
  # Start command
  start)
    echo "Starting $APP"
    /bin/su -m $USER -c "$ULTRA_HOME/bin/ultraesb-daemon.sh start &> /dev/null"
    ;;
  # Stop command
  stop)
    echo "Stopping $APP"
    /bin/su -m $USER -c "$ULTRA_HOME/bin/ultraesb-daemon.sh stop &> /dev/null"
    echo "$APP stopped successfully"
    ;;
  # Status command
  status)
    /bin/su -m $USER -c "$ULTRA_HOME/bin/ultraesb-daemon.sh status"
    ;;
   # Restart command
   restart | force-reload)
        $0 stop
        sleep 5
        $0 start
        ;;
  *)
    echo "Usage: /etc/init.d/$APP {start|restart|stop|status}"
    exit 1
    ;;
esac
exit 0

Next, make this file executable (e.g. sudo chmod +x /etc/init.d/ultraesb). This script invokes the bin/ultraesb-daemon.sh script to start/stop/restart the UltraESB or show its current status. Hence specify the Linux username under which the UltraESB would be executed, the UltraESB home directory ULTRA_HOME and the JAVA_HOME directory. Make sure that the JDK has been updated with the unlimited strength policy files for cryptography.

Next, register the LSB script to the host machine as follows for Ubuntu/Debian, or using "chkconfig –add ultraesb" for RHEL/CentOS

asankha@pub:/etc/init.d$ sudo update-rc.d ultraesb defaults
 Adding system startup for /etc/init.d/ultraesb ...
   /etc/rc0.d/K20ultraesb -> ../init.d/ultraesb
   /etc/rc1.d/K20ultraesb -> ../init.d/ultraesb
   /etc/rc6.d/K20ultraesb -> ../init.d/ultraesb
   /etc/rc2.d/S20ultraesb -> ../init.d/ultraesb
   /etc/rc3.d/S20ultraesb -> ../init.d/ultraesb
   /etc/rc4.d/S20ultraesb -> ../init.d/ultraesb
   /etc/rc5.d/S20ultraesb -> ../init.d/ultraesb

Note that the conf/wrapper.conf contains the application level Java property configuration aspects such as heap memory sizes, system properties, etc. Also specify the full path to the java executable in the conf/wrapper.conf as follows

..
# Java Application
wrapper.java.command=/opt/jdk/bin/java
..

You should now be able to start/stop/restart or check the status of the UltraESB as follows

asankha@pub:/etc/init.d$ sudo /etc/init.d/ultraesb start
Starting ultraesb
asankha@pub:/etc/init.d$ sudo /etc/init.d/ultraesb status
AdroitLogic - UltraESB is running (2124).
asankha@pub:/etc/init.d$ sudo /etc/init.d/ultraesb stop
Stopping ultraesb
ultraesb stopped successfully
asankha@pub:/etc/init.d$ sudo /etc/init.d/ultraesb status
AdroitLogic - UltraESB is not running.

Clustering

Refer to the Clustering Guide for information on setting up a cluster of nodes. This guide describes the underlying aspects of using clustering, and the steps to be followed for the set-up and configuration of a cluster.

It is recommended to obtain a production support subscription for critical deployments to ensure that the system that you design and deploy is insured from any unexpected issues. AdroitLogic guarantees to solve your production issues within the agreed SLA depending on the priority of the issues agreed against the selected support subscription.

You may go through the Production Support Reference for more information on the benefits of the Production Support.

In this topic
In this topic
Contact Us