Server Automation can only run a single audit - needs restart for another one

  • KM719397
  • 23-Apr-2009
  • 23-Apr-2009

Archived Content: This information is no longer maintained and is provided "as is" for your convenience.

Summary

After starting Server Automation, only a single audit job can be run. A subesequent audit job never finishes. The log in the spin indicates lots of garbage collection. After stopping and restarting Server Automation, another audit job can again be run.

Error

STARTUP LOG CONTENTS:

"CheckBotUp: Mon Apr 20 15:02:18 2009: Succesfully connected to server on 1018
***NOTICE - "smb" not installed on this system
>> Verify twist dependencies...
Verify hostname "spin" is resolvable: SUCCESS
Verify hostname "twist" is resolvable: SUCCESS
Verify hostname "spin" is listening on port 1004: SUCCESS
Verify communication with "spin": SUCCESS
Verify hostname "localhost" is listening on port 1007: SUCCESS
Verify communication with local Spin service: SUCCESS
Verify hostname "truth" is listening on port 1642: SUCCESS
Verifying en_US.UTF-8 locale is available: SUCCESS
>> Starting twist ...
Starting twist:
couldn't figure out TIBCO_HOME, Web Services Data Access Engine is running in Standalone mode.
Starting Web Services Data Access Engine, synchronously.
..............VerifyingTwist started, but problems remain therein. Consult ye twist logs.
CompilerOracle: exclude weblogic/utils/classfile/expr/LocalVariableExpression codeAssign
CompilerOracle: exclude weblogic/utils/classfile/expr/AssignStatement <init>
Mon Apr 20 15:04:01 UTC 2009 W Unable to connect to HTTP port: 1026. Still trying...
Mon Apr 20 15:05:01 UTC 2009 E Failed to connect to HTTP port.
opsware-sas: "twist" failed to startsync
Failed to perform "startsync" operation on Opsware SAS components.
"
TWIST LOG CONTENTS

###<Apr 20, 2009 3:04:01 PM UTC> <Error> <HTTP> <cmxp5120> <twist> <ExecuteThread: '26' for queue: 'default'> <<WLS Kernel>> <> <BEA-101017> <[ServletContext(id=12738757,name=DefaultWebApp,context-path=)] Root cause of ServletException.
javax.naming.NameNotFoundException: Unable to resolve 'twist.APIVersionLabel' Resolved twist; remaining name 'APIVersionLabel'
at weblogic.jndi.internal.BasicNamingNode.newNameNotFoundException(BasicNamingNode.java:924)
at weblogic.jndi.internal.BasicNamingNode.lookupHere(BasicNamingNode.java:230)
at weblogic.jndi.internal.ServerNamingNode.lookupHere(ServerNamingNode.java:154)
at weblogic.jndi.internal.BasicNamingNode.lookup(BasicNamingNode.java:188)
at weblogic.jndi.internal.BasicNamingNode.lookup(BasicNamingNode.java:196)
at weblogic.jndi.internal.WLEventContextImpl.lookup(WLEventContextImpl.java:258)
at weblogic.jndi.internal.WLContextImpl.lookup(WLContextImpl.java:363)
at javax.naming.InitialContext.lookup(InitialContext.java:347)
at jsp_servlet.__index._jspService(__index.java:158)
at weblogic.servlet.jsp.JspBase.service(JspBase.java:33)
at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:1077)
at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:465)
at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:526)
at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:348)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:7047)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:121)
at weblogic.servlet.internal.WebAppServletContext.invokeServlet(WebAppServletContext.java:3902)
at weblogic.servlet.internal.ServletRequestImpl.execute(ServletRequestImpl.java:2773)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:224)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:183)

Cause

Garbage Collection (GC) problem as a result of Oracle Upgrade.

 

During the installation of Server Automation the script creates a public synonym for audit_actions that points to truth.audit_actions.

After an upgrade of the Oracle database from 10.2.0.2 -> 10.2.0.4.  the public sysnonym is overwritten. The Oracle upgrade recreates the public synonim audit_actions to point to sys.audit_actions.

 


Fix

Customer repaired the root cause.
HP Support broght in dba experts in HP development. Customer admini brought in the customer's dba.
HP dba experts (Aarti, Brian Kris, Robert) had provide a script for diagnois. It showed that there was no alert log for SA.   Step 4 indicates a problem (see below)

=================================

Run the following on the Oracle db machine
su - oracle
sqlplus "/ as sysdba"

1) select name, value from audit_params;

2)
Col schema_user format a10
Col what format a50
Set line 200
Select job, schema_user, last_date, this_date, next_date,broken, what from dba_jobs;

3) show parameter job

4) Get the Alert.log file from customers Oracle database.
=================================
4) Get the Alert.log file from customers Oracle database. ???????????????
From customers there is no alert.log file.
=================================

Opsware public synonym "audit_actions" was reverted back to pointing to "sys.audit_actions" as a consequence of on Oracle Upgrade 10.2.0.2 to 10.2.0.4