troubleshoot ux term

UXTerm

Version: 17.07

Troubleshoot Reporting Guide

UltraESB-X facilitate the collection of diagnostic details of itself and its container (JVM), such as logs, configurations, heap dumps and thread dumps, via a set of troubleshoot reporting tasks.

Tasks

Each UltraESB-X instance may expose a set of troubleshoot reporting tasks. The Management API facilitates the invocation of a subset of these tasks sequentially, as a task queue, bundling the results of all tasks into an archive on the remote system, which can be fetched manually or downloaded via the API.

  • Some tasks, such as thread dumps and heap dumps, can be lengthy and consume high amounts of system resources (e.g. CPU and memory). Care should be taken when invoking them, as they may further de-stabilize the target system, e.g. in high-load situations. Such tasks are marked as intensive by the framework. During invocation of a task queue containing such intensive tasks, UXTerm will issue a warning and ask for confirmation before submitting the queue to the server.

  • Tasks that support the credential masking feature are marked as maskable.

Task Parameters

Some tasks allow customization via parameters. Such parameters can be specified during invocation by prepending the task name with a comma-separated list of key=value pairs, following the syntax

task1(param1=value1,param2=value2[,…​])[,…​]

where key corresponds to a parameter name and value is its value. If any of the parameters are not specified, a default value defined by the task will be utilized. For example,

thread-dump(period=5,count=3),conf-dir,detailed-logs(period=300,logger=org.adroitlogic.x.transport.http.nio)

will invoke the tasks

  • thread-dump, taking 3 (period) thread dumps with a delay of 5 seconds (period) between each pair

  • conf-dir

  • detailed-logs, increasing the log level of org.adroitlogic.x.transport.http.nio (logger) over 300 seconds (period), i.e. 5 minutes

Currently the following in-built tasks are provided in the UltraESB-X distribution:

Task Name Description Intensive? Maskable? Arguments (Name | Description | Default)

projects-dir

archives the UltraESB projects directory, optionally with credential masking

No

Yes

conf-dir

archives the UltraESB configuration directory, optionally with credential masking

No

Yes

lib-custom

archives the UltraESB custom libraries directory

No

No

thread-dump

produces thread dumps of the JVM running the UltraESB instance

Yes

No

period

period (in seconds) between thread dumps

5

count

number of thread dump samples

3

detailed-logs

takes a detailed log sample over a given time by escalating log levels of specified loggers

Yes

Yes

period

period (in seconds) over which to collect detailed logs

60

level

level to which logging should be escalated for sampling

DEBUG

logger

logger whose level should be escalated for sampling

org.adroitlogic

logs

archives the log files currently residing in the UltraESB instance, optionally with credential masking

No

Yes

heap-dump

produces a heap dump of the JVM running the UltraESB instance

Yes

No

lib-dir-info

Lists the files reside inside the library directory and its main sub directories

No

No

lib-patches

archives the UltraESB patch libraries directory

No

No

esb-info

produces a detailed version and status information report of the UltraESB instance

No

Yes

sys-info

produces a detailed report on the system and JVM running the UltraESB instance

No

Yes

Task Bundles

Additionally, the UXTerm configuration allows one to define several task bundles for quick invocation of groups of tasks. A task bundle is basically a predefined queue of tasks, denoted by a comma-separated list of task names, referred to by a single name (bundle name); for example,

ultra.terminal.troubleshoot.bundles.info=esb-info, sys-info

in the UXTerm configuration denotes a task bundle named info, including the two tasks esb-info and sys-info. Note that the bundle name should be added to the name list under ultra.terminal.troubleshoot.bundles property for it to be identified correctly by the commmand; for example,

ultra.terminal.troubleshoot.bundles=full, dump, info

Instead of a list of tasks, one can specify a task bundle name when invoking the troubleshoot report command, in which case the associated list of tasks will be submitted to the server.

Credential Masking

Some tasks, such as configuration archival, offer an option to run their output through a masking logic, encrypting portions of sensitive content (e.g. passwords or keys) using a user-specified password. This can be useful if you plan to submit the troubleshoot report to the original developers for further analysis, and in case the credentials need to be unmasked during a later stage, the password can be disclosed separately. The parameters -p, -m, -k and -c in the tbsrstr command govern the masking process. The same masking parameters are applied to all applicable tasks (having the maskable flag) in the submitted queue.

Picking Tasks for Execution

Tasks are picked using the -t and -b parameters of the tbsrstr command. Bundles specified in -b will be expanded into the respective task names, and merged with task names specified in -t. If -i (ignore) arguments are also specified, they would be removed from the generated task list, to derive the final list (queue) of tasks for execution. You can refer to a task or bundle name by a part of its name, as long as it is unambiguous. For example, when invoking tbsrstr with -b, instead of specifying the full name of a bundle named configuration-archival you can simply enter config, as long as no other task bundle name contains the word config.

Troubleshoot Reporting Commands

These commands facilitate the collection of diagnostic details from the connected UltraESB-X instance, via reporting tasks.

troubleshoot-report-option-list, tbsrol

Lists the troubleshoot reporting tasks available on the connected UltraESB-X instance.

Syntax: tbsrol [-v]

-b, --bundles

lists only the task bundles (defined in the UXTerm configuration)

-t, --tasks

lists only the available tasks (retrieved from the connected instance)

-v, --verbose

displays all available details of the listed tasks or bundles

troubleshoot-report-start, tbsrstr

Starts a troubleshoot reporting session on the connected instance.

Syntax: tbsrstr [-a] [-b <bundle-list>] [-c <cipher>] [-f <target-file>] [-i <tasks-to-ignore>] [-k <key-factoy>] [-m <mask>] [-p <password>] [-t <task-list>]

E.g. : tbsrstr -f /tmp/threads.zip -t heap,thread(period=5,count=3) -m password=[^\s]* -p ultraesb

-a, --async

asynchronous invocation (return to shell after submitting the reporting configuration to server)

<bundle-list>

comma-separated list of names of task bundles to be executed (tasks will be merged, and duplicates removed on a retain-first-found basis)

<cipher>

cipher name used for encryption under masking mode, applicable with -m (default if unspecified: ultra.terminal.troubleshoot.encryption.cipher in UXTerm configuration)

<target-file>

desired output path (must be absolute) where the report should be saved on the instance, optionally including file name

<tasks-to-ignore>

comma-separated list of tasks to be ignored when running the specified bundles (generally for use along with -b)

<key-factoy>

key factory name used for encryption, applicable with -m (default if unspecified: ultra.terminal.troubleshoot.encryption.key_factory in UXTerm configuration)

<mask>

regular expression for matching credentials to be masked or encrypted

<password>

password for encrypting matches of -m (if unspecified, matches will be irrecoverably masked with symbol characters)

<task-list>

comma-separated list of tasks to be executed, in the format task1(param1=value1,param2=value2[,…​])[,…​]

troubleshoot-report-status-check, tbsrstchk

Checks the status of a previous asynchronous troubleshoot reporting invocation.

Syntax: tbsrstchk

The output includes the execution state of each executed task, along with the completion status (success/failure) and the time taken by each, and the overall status and elapsed time for the whole invocation.

In this topic
In this topic
Contact Us