How to manage ActiveMQ subscriptions and settings within Sentinel RD and Log Manager.

  • 7007943
  • 17-Feb-2011
  • 26-Apr-2012

Environment

Novell Sentinel RD
Novell Log Manager
ActiveMQ

Situation

Memory dumps (Communication_Server.hprof) showing up regularly for Communications Server.

Multiple durable subscriptions incorrectly exist within ActiveMQ.

Resolution

Having multiple durable subscriptions for the Das_Binary service is a problem because the real Das_Binary process will only match one of those and pull events from it.  Because the duplicate subscription is also Durable events placed into it (all events destined for the permanent datastore) will never be removed and will fill ActiveMQ until it crashes.

To enable access to ActiveMQ to manage topics and subscriptions the running ActiveMQ process must have JMX enabled.  In order to interface with the process via JMX the system must have the Java Development Kit (JDK) installed.  This JDK should match the version of the running Java Runtime Environment (JRE) for the Sentinel or Log Manager service though different versions will sometimes work as well.  Download the appropriate 1.6.0_12 (or whichever is appropriate for your version of Sentinel or Log Manager) JDK from Sun and install it on the server where ActiveMQ is installed.  After making a modification to the Sentinel or Log Manager service JConsole will be used to interface with the ActiveMQ process which requires X libraries on the server if not a full X environment in which it can load.  If there is no GUI on the system JConsole can be tunneled (via SSH or equivalent) to a system with the requirements as long as the server has the required X libraries.

Within the Sentinel or Log Manager service's 'config' directory is an activemq.xml file which must be modified.  ActiveMQ only runs on the original Sentinel RD or Log Manager server.  Before making modifications create a backup of the existing file and then change the following line:

<broker xmlns="http://activemq.apache.org/schema/core" brokerName="localhost" dataDirectory="${activemq.base}/data" destroyApplicationContextOnStop="true">

to the following:

<broker useJmx="true" xmlns="http://activemq.apache.org/schema/core" brokerName="localhost" dataDirectory="${activemq.base}/data" destroyApplicationContextOnStop="true">

This step requires an ActiveMQ restart to apply the change, which requires restarting the Sentinel RD or Log Manager service. Lookup the Process ID (PID) of the process running ActiveMQ specifically using a command like the following:

ps aux | grep -i activemq | grep -v grep

or

pgrep -f activemq

Once completed load JConsole from the JDK located, for example, at /usr/java/jdk1.6.0_12-x86-64/bin/jconsole.  Be sure to load jconsole as the same user which is running the ActiveMQ process ('novell' is a likely system user being used) and an initial connection screen should be displayed.  From this connection screen choose the process with the PID matching the previously-found ActiveMQ process.  If the only process visible is JConsole it is likely that JConsole is being run as the wrong user.

Within the JConsole utility choose the MBeans tab, then drill down through org.apache.activemq: localhost: Subscription: Durable.  There are multiple options to choose among within the list of Durable subscriptions.  Specifically work through the various folders at this level to find items named 'Das_Binary' followed by a UUID.  There should only be one Das_Binary entry within each of the Durable Subscription structures.  For example, there should be one entry within ewizard_binary_event, another within database_tagged_event, and one within each of database_binary_event, correlation_binary_event_update, and correlation_binary_event.

If there are  multiple Das_Binary entries within each of these structures then it is likely that one of them is never being cleaned out by an external service which can lead to significant problems with the system as a whole as data are never flushed.  The correct Das_Binary entry should match the UUID of Das_Binary contained within the Sentinel or Log Manager install's configuration.xml file.  Expand the Das_Binary* duplicate (to-be-removed) entry and click on Operations beneath it.  In the right-hand pane click 'Destroy'.  Repeat this process for any Das_Binary instances which do NOT match the UUID in your service's configuration.xml file.  If there are two Das_Binary entries for each of the Durable Subscriptions, the same two UUID should be present for all of them in most cases and the same UUID should be removed from all of them.  Be sure that only Das_Binary* entries are being destroyed.  Attempting to remove a subscription in use (meaning the correctly-configured instance which should not be removed) will result in an exception such as:

Problem invoking destroy : java.rmi.UnmarshalException: Error unmarshaling return; nested exception is: java.lang.ClassNotFoundException: javax.jms.JMSException (no security manager: RMI class loader disabled)

Restart the Sentinel or Log Manager service to complete the process.