HP Diagnostics 8.0, 8.01, 8.02, 8.03, 8.04, 8.05

The first page of this readme document contains the following identifying information:

Version number, which indicates the software version.

Publish date, which changes each time the document is updated.

In addition to this readme, please see the Patch_Install_Instructions.pdf (in each patch
download package) for important upgrade and patch installation instructions.

The full Diagnostics documentation set was last updated and posted on the manuals download site for the
8.03 release. To check for recent updates or to verify that you are using the most recent edition, visit the
following URL:

This site requires that you register for an HP Passport and sign-in. To register for an HP Passport ID, go to:

You will also receive updated or new editions if you subscribe to the appropriate product support service.
Contact your HP sales representative for details.

To view files in PDF format (*.pdf), Adobe Acrobat Reader must be installed on your system. To download
Adobe Acrobat Reader, go to the following URL: www.adobe.com

Installation Overview

Diagnostics patch releases, such as 8.05, contain a full replacement of the Diagnostics components; therefore
you need to follow the same instructions as for an upgrade.

Upgrade and patch installation instructions are provided in the HP Diagnostics Installation and
Configuration Guide in Appendix G.

For patch releases, installation instructions are also provided in a document available with each package
download (Patch_Install_Instructions.pdf). The patch install instructions are a duplicate of
Appendix G, provided in each download package as a convenience.

The configuration guidelines found in the Diagnostics Installation guide for setting up SiteMinder module
are incorrect (defect 49236).

Set up Apache Server 2.2 with reverse proxy (reverse proxy redirection works fine) to Diagnostics

Enable the SiteMinder Agent in the Apache Server

Configured the SiteMinder Module in the jaas.configuration

When entering into the SiteMinder site and provide credentials, the Browser shows "Internal Server Error".

To correct this, add "ProxyPass /siteminderagent !" to the httpd.conf and before other ProxyPass directives.

An example of Apache reverse proxy setup on HP-UX - update procedure as follows:

ProxyPass / Error! Hyperlink reference not valid. of Diagnostics Server>:2006/
(2006 is the default Diagnostics Server port, use the port configured for your Diagnostics Server)

ProxyPassReverse / Error! Hyperlink reference not valid. of Diagnostics Server>:2006/
(2006 is the default Diagnostics Server port, use the port configured for your Diagnostics Server)

What's New in 8.05

Diagnostics 8.05 has a number of defect fixes and a number of new features. Note that the defect tracking
number shown (for example 35266) is generally prefixed with QCCR1I.

New Features

UI - The ability to print the thread dumps from the profiler view (47024)

Background: Prior to 8.05, it was not possible to print the thread dumps from the profiler view.

Description: New to this version is the ability to print the thread dumps from the profiler view.

Benefits: Usability

Java Agent - Felix Framework Support (34416)

Background: Customers who depended on the Felix framework for application development could not
measure the performance of their applications in Diagnostics.

Description: New instrumentation points have been added to measure the performance of applications
created using Felix.

Benefits: Functionality (Platforms)

Server - Extend the HP Diagnostics (MercuryStatusAlerts.mib) to transfer the information to Tivoli
(41715)

Background: In order to transfer information to the IBM Tivoli product (server name and probe of
the event), the HP Diagnostics (MercuryStatusAlerts.mib) should be enhanced.

Description: In 8.05, alertProbe, alertHost, and alertPath have been added to the MIB to provide
additional entity info when applicable.

Benefits: Functionality

Server - Keepalive from probes is impacting the performance on the server due to communication
overhead. (45642)

Background: Keep Alive from probes in some high volume customer environments will severely
impact the performance of the Commander server.

Description: The server was modified to improve the algorithm that sends the keep alive messages
so that performance is not as heavily impacted.

Benefits: Performance

Server - CAM Discovery: Make throttling of queries configurable (41529)

Background: Prior to this release, it was not possible to instruct the server on how often to run
discovery queries. In some cases, it is advantageous to throttle down or even throttle up the queries.

Description: The property file server.properties contains entries now to throttle queries.

# default max throughput for discovery ... making it faster will use more CPU and disk resource

#cam.discovery.throughput.max_hits=20

#cam.discovery.throughput.per_time=10s

Benefits: Performance

Java Agent - Support of AXIS2 Web Services (33847)

Background: Customers who depended on AXIS2 Web Services for application development could not
measure the performance of their applications in Diagnostics.

Description: New instrumentation points have been added to measure the performance of applications
created using AXIS2 Web Services.

Benefits: Functionality (Platforms)

UI - Allow the deletion of Server Requests (41717)

Background: Prior to this release, via the context menu, it was possible to delete a probe but not a
server request.

Description: It is now possible to delete a server request via the context menu.

Benefits: Usability, Functionality

UI - Throttle Enterprise UI Queries To Improve Performance (41821)

Background: In order to support 100 concurrent Enterprise UI sessions or more, the UI needs to throttle
the queries that the UI is executing so as to not inundate the server. This is like a freeway entrance
ramp light that only allows 2 cars every N seconds to keep the freeway from getting overly jammed with
cars.

Description: These values in the ui.properties can be used to throttle update frequency of UI queries to
the server. This is important when there are hundreds of Enterprise UI users. The Enterprise UI data
is updated based on the time control (last 5 minutes, 20 minutes, 1 hour, etc.) and data type. For
example trend data that populates charts are updated every 5 seconds when the time control is set to at
last 5 minutes. When the property
ui.query.update.frequency.multiplier.com.mercury.diagnostics.common.data.query.builder.handles.
TrendDataHandle=2 is enabled then the chart will update every 10 seconds ( 5 s * 2 multiplier).
Adjust the multiplier upward as the number of users increases.

#ui.query.update.frequency.multiplier.com.mercury.diagnostics.common.data.query.builder.handles.
SummarySetDataHandle=2.5

#ui.query.update.frequency.multiplier.com.mercury.diagnostics.common.data.query.builder.handles.
SummaryDataHandle=2

#ui.query.update.frequency.multiplier.com.mercury.diagnostics.common.data.query.builder.handles.
TrendDataHandle=2

#ui.query.update.frequency.multiplier.com.mercury.diagnostics.common.data.query.builder.handles.
EntityContentsDataHandle=2.5

#ui.query.update.frequency.multiplier.com.mercury.diagnostics.common.data.query.builder.handles.
GetAlertEventsHandle=2

#ui.query.update.frequency.multiplier.com.mercury.diagnostics.common.data.query.builder.handles.
GetMetaDataHandle=2

#ui.query.update.frequency.multiplier.com.mercury.diagnostics.common.data.query.builder.handles.
BizTxnTopologyDataHandle=3

#ui.query.update.frequency.multiplier.com.mercury.diagnostics.common.data.query.builder.handles.
ServerRequestTopologyDataHandle=3

Benefits: Performance

Defect Fixes

.NET Agent Fixes

42342 - A WCF method is not being detected as a WCF method

Problem - The problem is that when you define a method like IBooksPort.GetBook(), the method name
ends up with a fully qualified name like services.pernix.local.IBooksPort.GetBook. When this happens
Diagnostics cannot find this method in the metadata so Diag cannot figure out that it is a WCF method.
These methods should be detected as WCF methods.

Resolution - These methods will now be detected as WCF methods.

49315- .NET Probe crashes on VMWare environment. The same same version of Probes on physical non-virtualized servers which host the exact same .NET application pools, works fine without experiencing this problem. Problem is only reproducible on VM environment.

Problem - .NET probe crashes on VMWare environment. The same version of probes on physical non-virtualized servers which host the exact same .NET application pools, works fine without experiencing this problem. Problem is only reproducible on VM environment. The crash seems to be triggered by accessing perfmon counters for a .NET version 3.5.

Resolution - Added an option in probe_config.sml file which will prevent the probe from accessing perfmon counters. The committed memory metric for the probe will revert to using Total Memory.

To disable the perfcounter access set

The default is false

39569 - After upgrading the .NET agent, some features such as RUM/TV integration no longer work.

Problem - After upgrading the .NET agent, one or more .NET agent assemblies are missing that affect
RUM/TV integration.

Resolution - This is a known operating system issue (http://support.microsoft.com/kb/905238). The
"upgrade" mode of the 8.05 agent install package now applies a workaround automatically to avoid
this Microsoft defect.

42342 - When you disable the .NET Agent from the menu, IIS hosted applications continue to run the
probe even after an "iisreset".

Problem - The .net profiler checks for the environment variable COR_ENABLE_PROFILING=1 to
indicate that the probe is enabled. The environment variable is set in the "global" environment space
for all processes, and also specifically for the w3svc service. Performing a 'disable probe' action removes
the environment variable, indicating that the probe is disabled.

If a system is rebooted with the probe in the 'disabled' state (no environment variable), the svchost
serving up w3svc will never see the new environment variable (when an 'enable' occurs).

Resolution - Always keep COR_ENABLE_PROFILING around for w3svc, and just set it to '0' for disable
and '1' for enable.

44668 - Server Error: CLR detected an invalid program

Problem - This problem was caused by our instrumentation code not handling generic methods correctly.

Resolution - We made a change to the instrumentation logic to handle generic methods correctly

Collector Fixes

39409 - SAP Collector runs out of memory when under heavy load

Problem - By design, a single SAP Collector had to monitor the entire ABAP cluster (i.e. all the Dialog
Instances). This had scalability issues, especially in the presence of a lot of busy Dialog Instances. The
data we got back was simply too much for a single Collector to handle.

Resolution - A fix was Implemented which allows for splitting the cluster into multiple Collectors. For
example if the cluster has 7 Dialog Instances, the user can now monitor 4 instances with one Collector
and 3 with another. Previously all 7 had to be monitored by the same collector.

In order to do that, the user can use the new <dialogInstance> tag when editing the r3config.xml. Refer
to r3config.xsd for detailed documentation.

For example, previously the below configuration would cause the Collector to collect data from the entire
cluster (even if it has 10 instances)

<direct-connection>

<user>myuser</user>

<host>foo.mycompany.com</host>

</direct-connection>

</r3system>

The following configuration tells the Collector to only collect from 3 instances (00, 01, and 10). Similarly,
the rest of the instances can be monitored by a different Collector:

<direct-connection>

<user>myuser</user>

<host>foo.mycompany.com</host>

</direct-connection>

</r3system>

36676 - SAP Collector - Unable to start SAP Collector due to host name containing underscore

Problem - Customer can't start the collector because the host name contains underscores. The Collector's
log contains the following error:

2009-12-09 15:03:58,629 SEVERE r3collector [main] Unhandled exception for task InstanceDiscovery for
R07 due to:
java.lang.IllegalStateException:
Unexpected format for server name: a_b_c_d. Expected three tokens delimited by '_' at
com.mercury.diagnostics.sap.r3.function.servers.ServerRecord.getR3Identifier(ServerRecord.java:125) at
com.mercury.diagnostics.sap.r3.topology.R3Instance.createApplicationInstances(R3Instance.java:164) at
com.mercury.diagnostics.sap.r3.task.InstanceDiscovery.executeTask(InstanceDiscovery.java:62) at
com.mercury.diagnostics.sap.r3.task.AbstractTask.run(AbstractTask.java:150) at
com.mercury.diagnostics.sap.r3.topology.R3Instance.startTasks(R3Instance.java:119) at
com.mercury.diagnostics.sap.r3.R3Collector.start(R3Collector.java:64) at
com.mercury.diagnostics.probe.external.ExternalProbe.start(ExternalProbe.java:170) at
com.mercury.diagnostics.common.util.DiagnosticsProcess.postInitialize(DiagnosticsProcess.java:109) at
com.mercury.diagnostics.common.loader.ModuleLoader.initModules(ModuleLoader.java:690) at
com.mercury.diagnostics.common.loader.ModuleLoader.access$3(ModuleLoader.java) at
com.mercury.diagnostics.common.loader.ModuleLoader$1.run(ModuleLoader.java:173) at
java.security.AccessController.doPrivileged(Native Method) at
com.mercury.diagnostics.common.loader.ModuleLoader.initializeEverything(ModuleLoader.java:159) at
...

Cause - The Collector does something called "instance discovery", which discovers the individual Servers
within R/3. Basically it asks the main R/3 Server for all the Servers' information and gets back some
strings in the format:

<serverHost>_<R3systemID>_<systemNumber>.

The collector then needs to parse this string to extract the individual information. The presence of the
underscore in the <ServerHost> was causing this parsing to fail. As a result, the Collector could not
connect to the individual Servers so no collection was taking place. This will happen for any customer
whose R/3 host names contain underscores.

Resolution - The code has been modified to take into account underscores in the host name.

Java Agent Fixes

46154 - Updating dispatcher.properties files causes probe to 'forget' mediator assignment

Problem - Updating JAVA probes dispatcher.properties files causes probe to 'forget' mediator
assignment if set on the command line.

Resolution - If you do not want the mediator assignment from the file to override the command line
option, simply comment it out when you change the file. There will no longer be warnings about
NullPointerException.

39967 - Java Profiler crashes with NullPointerException for WebSphere on SPARC Solaris

Problem - When the user tries to log in to the Profiler with "change" capabilities, the Profiler crashes.
Using "view" only capabilities work.

Resolution - Internal coding issue, the crash no longer occurs.

48818 - BEA_JPD point has trended_methods setting OOB, resulting in thousands of trended methods

Problem - BEA_JPD point has trended_methods setting OOB, resulting in thousands of trended
methods. Diagnostics should not have any trended methods defined out of the box, especially one that
would result in so many entities.

Resolution - Removed trended_method qualification.

43212 - Instrumentation runtime error (NullPointerException) in
com.ibm.ws.webservices.engine.client.Connection.invokeEngine

Problem - In probe.log, the following can be seen:

2010-03-02 15:45:17,927 SEVERE com.mercury.opal.capture.proxy [WebContainer : 4] Instrumentation
runtime error

java.lang.NullPointerException

at com.ibm.ws.webservices.engine.client.Connection.invokeEngine(Connection.java)

at com.ibm.ws.webservices.engine.client.Connection.invoke(Connection.java:712)

at com.ibm.ws.webservices.engine.client.Connection.invoke(Connection.java:663)

at com.ibm.ws.webservices.engine.client.Connection.invoke(Connection.java:491)

at com.ibm.ws.webservices.engine.client.Call.invoke(Call.java:1515)

at $Proxy136.processMessage(Unknown Source)

Resolution - In a corner case, under unknown circumstances, the WebService may be null in
com.ibm.ws.webservices.engine.client.Connection.invokeEngine(). Necessary conditions to handle
this case gracefully was added.

40239 - WebSphere 6.1 crashes with 8.03 probe on Solaris 10

Problem - The application log contains

[2/16/10 9:34:25:361 CET] 00000049 EJBContainerI E WSVR0068E: Attempt to start
|EnterpriseBean MProMediationWSApp#MProMediationWSEJB.jar#MProMediationWS failed with exception:
java.lang.VerifyError: (class: com/ibm/wsspi/sibx/mediation/flow/ejb/MediationFlowBean, method:

getSessionContext signature: ()Ljavax/ejb/SessionContext;) Inconsistent stack height 1 != 2 at
java.lang.Class.getDeclaredConstructors0(Native Method)

and the application does not start.

Resolution - The root cause is the Diagnostics instrumentation, as the instrumented bytecodes did not
pass the verification step. Changes were made to produce correct instrumentation and the application
now starts.

39786 - Frequently logged messages "Metric group 'j2cModule' will be ignored (no stats found) " filling
log file.

Problem - Frequently logged messages "Metric group 'j2cModule' will be ignored (no stats found) " filling
log file.

Resolution - Changed the message level from 'info' to 'debug'.

39372 - Java Agent - Cross-vm RMI call profile is not showing for Servlet to EJB call in WebSphere 6.1.

Problem - 8 clusters of which 4 are web container and 4 are EJB container. These containers are never
on the same box. Load test is enabled with all the 8 probes.

The transaction view -> server requests does not contain methods from EJB container. The methods it
has are only from web container.

The same server request in 'server request ' view has the right information.

Cross-VM is not properly displayed.

Resolution - The solution to this problem is to enable Corba cross-VM on both probes. In 8.x cross-VM
support was added for pure IIOP which covers RMI over IIOP as well. After enabling Corba cross VM
on both probes, cross-VM instance will be shown.

Corba Cross-VM must be enabled by following these steps (on both sides):

1) Disable RMI in the points file

2) Enable the Corba points (there is a Corba section towards the end of the points file)

3) Take a look at the documentation in the Corba section of the points file and follow the steps listed
there. After doing so, the "jvmEntries" of your WAS server.xml should look something like below.
The important 2 new things to notice are:

-Dorg.omg.PortableInterceptor.ORBInitializerClass.com.mercury.opal.javaprobe.handler.corba.CorbaORBInitializer

and

<classpath>/opt/optibnch/JavaAgent/DiagnosticsAgent/lib/probeCorbaInterceptors.jar</classpath>

The "jvmEntries" section for the WAS server.xml looks something like this:

<jvmEntries xmi:id="JavaVirtualMachine_1182370866099" ....(ommitted for clarity).... genericJvmArguments="-
Dprobe.group=WAS_Group -Dprobe.id=was61-jvm15-ovribmt12 -
Xbootclasspath/p:/opt/optibnch/JavaAgent/DiagnosticsAgent/classes/IBM/1.5.0:/opt/optibnch/JavaAgent/DiagnosticsAgent/classes/b
oot -Xshareclasses:none -
Dorg.omg.PortableInterceptor.ORBInitializerClass.com.mercury.opal.javaprobe.handler.corba.CorbaORBInitializer"
disableJIT="false">
<classpath>/opt/optibnch/JavaAgent/DiagnosticsAgent/lib/probeCorbaInterceptors.jar</classpath>
</jvmEntriees>

35355 - Instrumentation problems with Apache CXF

Problem - The Diagnostics JAX-WS outbound web service instrumentation needed to exclude one of the
Apache CXF proxy classes, in order for the cross-VM correlation to work properly.

Resolution - Excluded the Apache CXF proxy class from the [JAX-WS-Outbound point]. Cross-VM for
Apache CXF works now.

35583 - Consider changing the default instrumentation point for [JAX-WS-Outbound] to work with
Apache CXF Framework

Problem - Using a simple Java client Apache CXF which invokes a web-service using the latest version
of Apache CXF (version 2.2.4), Diagnostics did not generate exceptions on the web services call.

Resolution - To fix this problem, the following changes were made in the instrumentation file
auto_detects.points. See ignore_cl:

[JAX-WS-Outbound]
class = javax.xml.ws.BindingProvider
ignore_cl = org.apache.cxf.jaxws.JaxWsClientProxy, ...

34636 - "SEVERE archive.... Locale issues can cause this." errors filling mediator log.

Problem - SEVERE messages in the server log:

2009-10-01 15:21:00,116: SEVERE archive : Cannot persist data due to missing latency for:
com.mercury.diagnostics.common.data.graph.node.WebServiceData:
weblogic.wsee.server.servlet.BaseWSServlet.void
service(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse)
/callchain2/CallChain2Service, null, Web
Service, null, CallChain2ImplService, m, _appsdir_CallChain2WSEar_ear, http://callchain2,
CallChain2SoapPort Locale issues can
cause this. Refer to persistence.enable.locale.independent.latency in server.properties.

Resolution - The message was misleading and incorrect, so we simply removed the message from the log.

42011 - When a proxy was enabled on the probe via dispatcher.proxy.host & dispatcher.proxy.port, user
permissions were not successfully being obtained for communication with the Mediator.

Problem - When a proxy was enabled on the probe via dispatcher.proxy.host & dispatcher.proxy.port,
user permissions were not successfully being obtained for communication with the Mediator. The probe
useradmin.log & probe.log contained http 504 gateway timeouts as the mediator was unreachable from
the probe except via the proxy.

Resolution - Updated the product to appropriately use a proxy url connection when doing server
communication.

42459 - Request that Diagnostics will capture the GET parameters, to allow server requests to be
differentiated.

Problem - Request that Diagnostics will capture the GET parameters, to allow server requests to be
differentiated.

Resolution - Diagnostics will capture the GET parameters to allow server requests to be differentiated.

41880 - Diagnostics JavaProbe caused severe problems when using duplicate/multiple -javaagent options
of the probeagent.jar to start probe

Problem - Problem - In certain rare circumstances, customers will mistakenly duplicate the command
line option "-javaagent:<diag-probe-dir>/lib/probeagent.jar" and the application server will not start.

Resolution - Even when duplicate/multiple -javaagent options of the probeagent.jar appear in the Java
commandline, the app_server will start and run with Diag javaprobe without any issue.

44826 - Upgrade to Probe version 8.03 causing issue

Problem - Customer migrated from HP Diagnostics probe 8.01 to 8.03 . They have a particular DB2
related reports failing after the migration.

Resolution - This is because of a known J9 JVM bug and Diagnostics has 2 workaround points to address
this bug. However, this was encountered in a class which our workaround did not include adding the
class to the list of "workaround points". See comment below for more details (copied from
<probe>/etc/auto_detect.points). Notice that the recommendation is to patch WAS and de-activate these
points.

; This is a series of 3 points introduced to avoid WebSphere 6.1 crashes due to a ; java.lang.VerifyError ...
invalid returnAddress for ret instruction ... at pc=65535 ; The root cause is a bug in IBM J9 2.3 JVM
(SR2), which is bundled with WebSphere 6.1.;
The point works by eliminating the specified method from instrumentation by [JDBC-Connection-
prepare].
; The SQL statements passed as arguments to the eliminated method(s) will not show up in SQL views.
; You may safely de-activate these points, if you are not a WebSphere 6.1 user, ; or if your version of
WebSphere 6.1 is patched.
IBM J9 2.3 JVM (SR5) is known to have the bug fixed.

41400 - Scenario Summary View saved as CSV file contains invalid data (like "12/31/1969 23:50").

Problem - Scenario Summary View saved as CSV file contains invalid data (like "12/31/1969 23:50").

Resolution - Data is now valid.

41718 - Enable uri.pattern.replace for "Static Content" rule in dynamic.properties.

Problem - Diagnostics needs to better handle situations where unique URLs (e.g. .../1234534535.pdf) are
created which causes symbol explosion on the server and probe and performance issues occur as a result.

Resolution - All pdf files will now be represented under a single server request, "Static Content".

48995 - Java Probe modifies response from the Oracle Application Server.

Problem - When the Java Probe is installed, the Application Server throws different responses as when
the Probe is not installed. So, it looks like the Java Probe modifies the response from the Application
Server.

This is in the following environment:

Solaris 5.10 - SPARC

Java J2EE 1.4

Java Probe 8.0x

Oracle Application Server 10.1.2

Resolution - For 8.04 and earlier, disabling [SOAP_Faults] point is a confirmed workaround. Versions
8.05 and 9.01 (and later) will work correctly out-of-box.

36067 - Apache Geronimo application server does not start with Java Agent.

Problem - An Apache Geronimo application server does not start with [HttpCorrelation] point enabled.
Environment:

Application Server: Geronimo 2.1.4 and using java 1.5.0_17 (SUN)

Diagnostics Probe Version: 8.03.39.85

OS: SUSE Linux Enterprise Server 10 (x86_64)

Resolution - The workaround is to disable the point for now.

42462 - Exception in production WAS server caused by HP probe.

Problem - Exception in production WAS server caused by HP probe.

Resolution - The issue is solved by using a new version of auto_detect.points

48462 - WAS with the probe does not start on z/OS.

Problem - Installation of Diagnostics 8.04 probe on WAS Portal Server v6.0 running on z/OS. The install
steps were:

- extract binaries from tar

- jreinstrumenter -i <path_to_was_java>

- jreinstrumenter -b <path_to_was_java>

... this produced Xbootclasspath params ...

- modify Xbootclasspath in WAS console

After restart WAS is unable to start and producing this exception:

Trace: 2010/07/05 15:35:55.952 01 t=AE3E88 c=UNK key=P8 (0000000A)

Description: Log Boss/390 Error

from filename: ./bbolrt.cpp

at line: 283

error message: BBOO0073E Shasta Runtime function logJavaException detected that the following
exception occurred in the Java VM ..

.JVM Exception Message = java.lang.NullPointerException JVM Stack Trace =
java.lang.NullPointerException

at com.mercury.opal.capture.JavaProbe.getLog(JavaProbe.java:85)

at com.mercury.opal.capture.proxy.InstrumenterProxy.instrument(InstrumenterProxy.java:196)

at com.mercury.opal.capture.proxy.InstrumenterProxy.instrument(InstrumenterProxy.java:120)

at java.lang.ClassLoader.merc_defineClass0(ClassLoader.java)

at java.lang.ClassLoader.defineClass(ClassLoader.java:808)

at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:147)

at java.net.URLClassLoader.defineClass(URLClassLoader.java:477)

at java.net.URLClassLoader.access$500(URLClassLoader.java:111)

at java.net.URLClassLoader$ClassFinder.run(URLClassLoader.java:849)

at java.security.AccessController.doPrivileged1(Native Method)

at java.security.AccessController.doPrivileged(AccessController.java:389)

at java.net.URLClassLoader.findClass(URLClassLoader.java:373)

at java.lang.ClassLoader.loadClass(ClassLoader.java:570)

at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:442)

at java.lang.ClassLoader.loadClass(ClassLoader.java:502)

Trace: 2010/07/05 15:35:55.953 01 t=AE3E88 c=UNK key=P8 (0000000A)

Description: Log Boss/390 Error

from filename: ./bbolrt.cpp

at line: 328

error message: BBOO0021E Assertion failed: '0', file ./bbolrt.cpp, line 328.

***

Resolution - Changed the logger to log to stderr if a log file cannot be created. And provided a
description for the workaround for the encoding issues.

47192 - Solaris systemmetrics can only handle 64 CPUs.

Problem - Systemmetrics core dumps on Solaris (Sparc) if there are more than 64 CPUs

Resolution - An internal array was expanded and the systemmetrics process no longer aborts.

36178 - Application memory corruption and potential aborts on x64 based VMware systems. (.NET and
Java Probes).

Problem - Problem - On a 64 bit VM Ware system, the .NET probe can cause the application that it is
monitoring to abort.

Cause - The code that was detecting a VM Ware situation and obtaining the correct timestamp was
flawed and was corrupting random memory locations.

Resolution - The code has been modified and simplified and no longer corrupts memory.

Server Fixes

47328 - Commander's MRU cache for server permissions is too small for big deployments

Problem - MRU list only keeps 30 mediators which causes a reload from file in case of a cache miss.

Resolution - Increased the MRU list to 80

41888 - Customer is seeing sever performance problems on the server caused by slow application
discovery.

Problem - The customer is seeing sever performance problems on the server caused by slow application
discovery.

Resolution - Improved performance by not following cross server sync lists if user selects skipping of
topological discovery.

49067 - Port 2006 still enabled after enabling HTTPS.

Problem - Even when SSL is enabled Diagnostics always keeps the non-SSL port open. Normally
firewalls would be used to secure this.

Resolution - In webserver.properties, there is now a commented out setting:

# jetty.ssl.loopback.host=127.0.0.1

If you can't use a firewall, to secure the non-ssl port (e.g., 0.0.0.0:2006), then you can use this setting to
make that port open for loopback only. This will allow the server to communicate with itself without
allow external access. If this is enabled on the commander, then the commander and mediators would
need to be configured to use SSL to communicate with each other.

Enterprise UI Fixes

34324 - CSV Export Footer Zoomed Time Range doesn't make sense.

Problem - The Zoomed Time Range for "a run" doesn't make sense. It says Zoomed Time Range:
6196:30:00 to 6207:44:59.

Resolution - The local time was being used instead of run time. (Run start is also shown when
appicable).

48450 - Data missing from views when saved as HTML page

Problem - The printable area is smaller than the table size so columns are dropped.

Resolution - Fixed by proportionally adjusting each column based on the printable area.

48882 - Call Profile export generates excessive white space when diagram is larger than the viewport.

Problem - Call Profile export generates excessive white space when diagram is larger than the viewport.

Resolution - The excessive whitespace has been eliminated.

What's New in 8.04

Diagnostics 8.04 has a number of defects fixed and a number of new features. Note that the defect tracking
number shown (for example 35266) is generally prefixed with QCCR1I.

New Features

Add default PMI metrics with the `EXPAND_PMI' parameter (35266)

Background: Prior to 8.04, there are no default EXPAND_PMI metrics configured for WebSphere JDBC
ConnectionPool module.

Description: The default PMI metrics configurations for JDBC Connection Pool of the WebSphere5 and
WebSphere6 collectors are changed by adding EXPAND_PMI[*]. Also, default JMX metric configuration
for the WebSphere JDBC DataSource MBeans was added.

Benefits: Usability

Allow expand and contract all in the inspector of the Enterprise UI (34441)

Background: Previously, each category in the metrics inspector had to be expanded and contracted
one at a time.

Description: A toolbar toggle button was added to the inspector to collapse / expand all categories.

Benefits: Usability

Display probe-id on the main probe page http://probename:35000/ (37873)

Background: Prior to 8.04 http://probename:35000/files shows HP Diagnostics J2EE Probe ".. Probe Name ...",
version, but the main screen http://probename:35000 does not show that same line, this would be very
useful.

Description: The main screen http://probename:35000/ now shows HP Diagnostics J2EE Probe ".. Probe
Name ...", version as well.

Benefits: Usability

.NET probe - Added the ability to exclude an assembly from being instrumented (35024).

Background: The excludeassembly parameter was added for the .NET probe for use in a customer
environment where they needed to exclude sensitive assemblies from instrumentation (example,
a product was used to obfuscate and encrypt code in sensitive assemblies and the code would
throw exceptions if it was instrumented). In .NET an assembly is a .exe or .dll file.

Description: To configure the exclusion of an assembly, you add <excludeassembly
name=<AssemblyNameToExclude> as a child to a process element in the probe_config.xml file.
Do not include the file extension, .exe or .dll as part of the name of the assembly. An example of
excluding an assembly called WebService.dll:

</appdomain>

</process>

Benefits: Functionality

Expose additional jetty parameters (38279)

Background: For performance reasons, it is important to expose additional properties for jetty in
webserver.properties to influence performance.

Description: The following properties are now included in webserver. properties:

jetty.acceptor_threads (1)

jetty.accept_queue_size (0)

jetty.linger_time (30)

Benefits: Performance

Display probe-id on the main probe page http://<probe>:35000/ (37873)

Background: Prior to 8.04, http:/probename:35000/files shows HP Diagnostics J2EE Probe " .. Probe
Name ...", version, but the main screen http://probename:35000 does not show that same line, this would
be very useful.

Description: The main screen http:/probename:35000/ now shows HP Diagnostics J2EE Probe " .. Probe
Name ...", version as well.

Benefits: Usability

Add TopN filter to UI (38538)

Background: If you have a large deployment with many probes, the probes views may populate slowly
due to the volume of data.

Description: The probe views (Probes, MA Managers, Oracle Probes, SQL Server Probes, and SAP
Probes) are configured to use the top ui property located on the Diagnostics (commander) server in the
etc/ui.properties file. By default the ui.topn property is not enabled. When enabled the Enterprise UI
receives this property from the commander on start up. The UI will use this value to limit the number of
probes in the probe views mentioned above.

For example with the value of ui.topn=20, the ui will ask the server to retrieve top 20 (times a small
factor that always gives more than requested) probes on each mediator. The top probes requested are
determined by the view's sorting metric (default is VM Heap Used). Additionally, the top n probes of
each mediator are returned with each change of the filters on the natural language title bar.

For example with:

- ui.top=20

-10 mediators with each mediator having 100 reporting probes, all having foo in their name.

- the Title bar reading "Java Probes in All Probe Groups filtered by 'Probe Name containing foo' with
Top 5 by Latency (Avg) charted for the last 5 minutes" .

The probe view will populate with at least 200 Java probes (10 mediators * 20) . From each mediator the
top 20 by Latency (Avg) with a name containing foo are returned to the commander. The commander
then sends all 200+ probe records to the Enterprise UI probe view.

Benefits: Usability, Performance

Make the creation of the symbol backup directory configurable (33771)

Background: Sometimes, the backup folder for jdb files under symboltable uses up a lot of disk
space when there are a large number of symbol files. This should be made configurable.

Description: Backing up the symbol table is now configurable. Backing up the symbol table can
now be enabled/disabled by setting the server property "symboltable.backup" to true/false in the
server.properties file.

Symbol table backup frequency (daily, weekly, monthly, and yearly) can now also be configurable
by setting the server property "symboltable.backup.majors". Set the property, using a common
separated list, to the desired backup frequency. The frequency names are the names as defined
by the persistency.major.x.name property. For example, to backup the symbol table weekly, use
the property "persistency.major.2.name", which is 'Weeks'.

The default configuration for symbol table backup, as defined in the server.properties files are:

# Should the server backup the symboltable?

symboltable.backup = true

# Which majors should be backed up?

symboltable.backup.majors = Days,Weeks,Months

Benefits: Performance

Remote backup script improvements (36487)

Background: The remote-backup scripts from 8.03 and before have some issues:

They backup the symboltable/backup directory which is not needed

They can accumulate files that are no longer relevant (not on the server but stay in the
backup location)

They try to backup the obsolete SymbolTable.db

The -v option doesn't work in the cmd file

The .sh file has DOS line `enders'

Description: A -c option (clean was added). When specified, files that exist in the ouput dir and do
not exist on the server will be removed (the others need to be kept for better performance with
timestamping feature on download). The jdb and PathedSymbolTable.db download stop at level=3
so that the backup files won't get picked up. The dirs are created do to the recursive wget, but
the files will not be downloaded. Also fixed -v and added it to usage message.

Removed the SymbolTable.db download. We're getting the .jdb files and the PathedSymbolTable.db.
The SymbolTable.db didn't make sense anymore.

Benefits: Performance, Usability

Java Probe - Document probe hookup configuration for Tomcat 5 running as a windows service (33531).

Background: The documentation should add information on how to configure Tomcat 5 running as
a Windows service to work with a Diagnostics Java Agent.

Description: Follow these steps to configure Tomcat 5.x/6.x running as a web service for the Java Agent:

Tomcat 5.x/6.x is installed in service mode and the Tomcat service is started.

From the Windows Task bar, right-click on the Apache Tomcat service icon and then select
Configure.

In the Apache Tomcat Properties dialog box, select the Java tab.

Add the -javaagent option and other Java parameters to the Java options section.

Restart the Tomcat service.

Benefits: Functionality

Defect Fixes

Java Agent Defect Fixes

35819 - On a Tru64 system using Classic VM 1.4.2, the probe can't communicate with the mediator
and a SEVERE error appears in the probe.log

Problem - Diagnostics 8.03, Tru64, Classic VM 1.4.2

The probe.log file contains:

2009-11-19 11:37:27,166 SEVERE com.mercury.opal.util.webserver [Thread-1] error configuring
webserver

java.lang.NoClassDefFoundError: com/ibm/jsse/IBMJSSEProvider

at com.mercury.diagnostics.common.modules.webserver.RangeSocketListener.createListener(RangeSocket
Listener.java)

at com.mercury.diagnostics.common.modules.webserver.JettyWebServer.createListener(JettyWebServer.ja
va)

at com.mercury.diagnostics.common.modules.webserver.JettyWebServer.configureServer(JettyWebServer.j
ava)

at com.mercury.diagnostics.common.modules.webserver.JettyWebServer.initServer(JettyWebServer.java)

at com.mercury.diagnostics.common.modules.webserver.JettyWebServer.start(JettyWebServer.java)

at com.mercury.diagnostics.common.modules.webserver.AutostartJettyWebServer$1.run(AutostartJettyWe
bServer.java)

at java.lang.Thread.run(Thread.java:534)

and there's no communication between the probe and the mediator.

Resolution - The problem is fixed, the log entry is gone.

30714-Adding JMX Apache Tomcat support caused WebSphere JMX to stop working in some situations

Problem - When running the 8.0 probe against any WebSphere system, an error occurs and
PMI metrics are not collected.

Errors in logfiles similar to the following are seen:

2009-03-19 19:39:36,023 WARN com.mercury.diagnostics.capture.metrics [Metrics Collection] Error
initializing
com.mercury.diagnostics.capture.metrics.jmx.JMXCollector@3a1164f4

java.lang.IllegalAccessError: com.mercury.diagnostics.capture.metrics.jmx.JMXCollector tried to access
method
com/mercury/diagnostics/capture/metrics/jmx/JMXCollector$AttributesAndDescriptors.add(Ljava/lang/S
tring;Lcom/mercury/
diagnostics/common/metrics/MetricDescriptor;)V20

Resolution - The fix is to set the depends.on.class of Apache Tomcat JMX collector to
javax.management.MBeanServer.

35172 - Heap walker is not working with WebSphere 6.1 (IBM JVM 1.5 on x86_Linux)

Problem - Heapwalker does not work in the following environment:

Websphere 6.1.0.21 on Redhat Linux AS 4 (JVM 1.5.0_11 IBM j9)

Red Hat Enterprise Linux AS release 4 (Nahant Update 5)

java version "1.5.0_11"

Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_11-b03)

Java HotSpot(TM) Server VM (build 1.5.0_11-b03, mixed mode)

Using:

-javaagent and -agentpath: No heapdump at all is seen

-Xbootclasspath and -agentpath: Heapdump works, but heapwalker does not work

Probe Version is 8.03.39.85

Cause - There is a defect in the Heapwalker code which prevented it from working, even if the
new approach is used which is described in the addendum for 8.03 at the end of these release notes.

Resolution - This has been fixed and is available as a patch on top of 8.03 and will be in the 8.04
release as well. Contact support and reference this defect number to obtain the patch.

36447- Corba cross-VM cannot be enabled on Java 1.4 VMs

Problem - Cannot enable Corba in the probe if the JVM is 1.4. Errors are thrown:

java.lang.UnsupportedClassVersionError:
com/mercury/opal/javaprobe/handler/corba/CorbaORBInitializer (Unsupported
major.minor version 49.0)

at java.lang.ClassLoader.defineClass0(Native Method)

at java.lang.ClassLoader.merc_defineClass0(ClassLoader.java(Compiled Code))

at java.lang.ClassLoader.defineClass(ClassLoader.java(Compiled Code))

at java.security.SecureClassLoader.defineClass(SecureClassLoader.java(Compiled Code))

at java.net.URLClassLoader.defineClass(URLClassLoader.java(Compiled Code))

at java.net.URLClassLoader.access$500(URLClassLoader.java(Inlined Compiled Code))

at java.net.URLClassLoader$ClassFinder.run(URLClassLoader.java(Compiled Code))

Cause - Corba interceptors were incorrectly compiled with Java 1.5

Resolution - Corba interceptors are now compiled with 1.4.

32360- Java Agent: NullPointerException in BasicMethodCaptureAgent.beforeInvocation

Problem - In probe.log, one can see massive messages like this:

2009-06-19 13:13:17,335 SEVERE com.mercury.opal.capture [[ACTIVE] ExecuteThread: '4' for queue:
'weblogic.kernel.Default (self-tuning)'] Error capturing invocation

java.lang.NullPointerException

at com.mercury.opal.capture.BasicMethodCaptureAgent.beforeInvocation(BasicMethodCaptureAgent.java:
191)

at com.mercury.opal.capture.proxy.MethodCaptureProxy.beforeInvocation(MethodCaptureProxy.java:78)

at com.mercury.opal.capture.proxy.MethodCaptureProxy.beforeInvocation(MethodCaptureProxy.java:131)

at weblogic.jdbc.wrapper.PoolConnection_com_pointbase_net_netJDBCConnection30.commit(Unknown
Source)

at weblogic.store.io.jdbc.ReservedConnection.commit(ReservedConnection.java:170)

at weblogic.store.io.jdbc.JDBCStoreIO.getTableOwnershipPhysical(JDBCStoreIO.java:2054)

at weblogic.store.io.jdbc.JDBCStoreIO.updateTableOwnership(JDBCStoreIO.java:2270)

at weblogic.store.io.jdbc.JDBCStoreIO.updateTableOwnershipFromTimer(JDBCStoreIO.java:2200)

at weblogic.store.io.jdbc.ReservedConnection.timerExpired(ReservedConnection.java:436)

at weblogic.timers.internal.TimerImpl.run(TimerImpl.java:273)

at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:
464)

at weblogic.work.ExecuteThread.execute(ExecuteThread.java:200)

at weblogic.work.ExecuteThread.run(ExecuteThread.java:172)

2009-06-19 13:13:17,335 SEVERE com.mercury.opal.capture [[ACTIVE] ExecuteThread: '4' for queue:
'weblogic.kernel.Default (self-tuning)'] Error capturing invocation

java.lang.NullPointerException

at com.mercury.opal.capture.BasicMethodCaptureAgent.beforeInvocation(BasicMethodCaptureAgent.java:
191)

at com.mercury.opal.capture.proxy.MethodCaptureProxy.beforeInvocation(MethodCaptureProxy.java:78)

at com.mercury.opal.capture.proxy.MethodCaptureProxy.beforeInvocation(MethodCaptureProxy.java:131)

at com.pointbase.net.netJDBCConnection.commit(DashoA13*..:669)