Environment
NetIQ Identity Manager Roles Based Provisioning Module 4.5
Clustered configuration
Situation
2017-01-19 00:00:19,727 [Tribes-Task-Receiver-6] ERROR org.apache.catalina.ha.session.DeltaManager- Manager [localhost#/IDMProv]: Unable to receive message through TCP channel
java.io.InvalidObjectException:
at com.sssw.fw.directory.core.EboUserCredentials.readResolve(EboUserCredentials.java:236)
at sun.reflect.GeneratedMethodAccessor779.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at java.io.ObjectStreamClass.invokeReadResolve(ObjectStreamClass.java:1148)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1810)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
at java.util.concurrent.ConcurrentHashMap.readObject(ConcurrentHashMap.java:1445)
at sun.reflect.GeneratedMethodAccessor117.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1058)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1900)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2000)
at java.io.ObjectInputStream.defaultReadObject(ObjectInputStream.java:501)
at com.sssw.fw.session.EboBaseSession.readObject(EboBaseSession.java:241)
at sun.reflect.GeneratedMethodAccessor118.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1058)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1900)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
at org.apache.catalina.ha.session.DeltaRequest$AttributeInfo.readExternal(DeltaRequest.java:378)
at org.apache.catalina.ha.session.DeltaRequest.readExternal(DeltaRequest.java:268)
at org.apache.catalina.ha.session.DeltaManager.deserializeDeltaRequest(DeltaManager.java:591)
at org.apache.catalina.ha.session.DeltaManager.handleSESSION_DELTA(DeltaManager.java:1279)
at org.apache.catalina.ha.session.DeltaManager.messageReceived(DeltaManager.java:1226)
at org.apache.catalina.ha.session.DeltaManager.messageDataReceived(DeltaManager.java:944)
at org.apache.catalina.ha.session.ClusterSessionListener.messageReceived(ClusterSessionListener.java:91)
at org.apache.catalina.ha.tcp.SimpleTcpCluster.messageReceived(SimpleTcpCluster.java:936)
at org.apache.catalina.ha.tcp.SimpleTcpCluster.messageReceived(SimpleTcpCluster.java:917)
at org.apache.catalina.tribes.group.GroupChannel.messageReceived(GroupChannel.java:278)
at org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:82)
at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.messageReceived(TcpFailureDetector.java:117)
at org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:82)
at org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:82)
at org.apache.catalina.tribes.group.ChannelCoordinator.messageReceived(ChannelCoordinator.java:252)
at org.apache.catalina.tribes.transport.ReceiverBase.messageDataReceived(ReceiverBase.java:287)
at org.apache.catalina.tribes.transport.nio.NioReplicationTask.drainChannel(NioReplicationTask.java:210)
at org.apache.catalina.tribes.transport.nio.NioReplicationTask.run(NioReplicationTask.java:99)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.sssw.fw.util.crypto.DecryptionException: Decryption failed.
at com.sssw.fw.util.crypto.KeyManager.decrypt(KeyManager.java:263)
at com.sssw.fw.directory.core.EboUserCredentials.decrypt(EboUserCredentials.java:220)
at com.sssw.fw.directory.core.EboUserCredentials.readResolve(EboUserCredentials.java:234)
... 47 more
Caused by: javax.crypto.BadPaddingException: Given final block not properly padded
at com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:966)
at com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:824)
at com.sun.crypto.provider.AESCipher.engineDoFinal(AESCipher.java:436)
at javax.crypto.Cipher.doFinal(Cipher.java:2165)
at com.sssw.fw.util.crypto.KeyManager.decrypt(KeyManager.java:256)
... 49 more
2017-01-19 00:00:20,696 [pool-6-thread-6] INFO com.novell.soa.af.impl.LogEvent- [RBPM] [Workflow_Forwarded] Initiated by System, Process ID: a73dfebbf2364cb2bdd0b67703a011e7, Process Name: cn=RemedyWorkflow,cn=RequestDefs,cn=AppConfig,cn=UserApplication,cn=DriverSet,o=system:56, Activity: Activity, Recipient: cn=USER1,ou=Users,o=novell
2017-01-19 00:00:20,705 [pool-6-thread-6] INFO com.novell.soa.af.impl.activity.LogActivity- [RBPM] [User_Message] Initiated by cn=uaadmin,ou=sa,o=novell, Process ID: a73dfebbf2364cb2bdd0b67703a011e7, Process Name: cn=RemedyWorkflow,cn=RequestDefs,cn=AppConfig,cn=UserApplication,cn=DriverSet,o=system:56, Activity: Activity4, Recipient: cn=USER1,ou=Users,o=novell, Author: {Enter author name here}, Message: GEN_V1_NOVIDVTREE_DEV_UserApplication_Remedy Workflow_Activity##Thu Jan 19 00:00:20 UTC 2017##-1##1##com.sssw.b2b.rt.GNVException: rt001801:Document I/O error:
Status: 502
HTTP Reason: Bad Gateway
Error Info: <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1"/>
<title>502 - Web server received an invalid response while acting as a gateway or proxy server.</title>
<style type="text/css">
<!--
body{margin:0;font-size:.7em;font-family:Verdana, Arial, Helvetica, sans-serif;background:#EEEEEE;}
fieldset{padding:0 15px 10px 15px;}
h1{font-size:2.4em;margin:0;color:#FFF;}
h2{font-size:1.7em;margin:0;color:#CC0000;}
h3{font-size:1.2em;margin:10px 0 0 0;color:#000000;}
#header{width:96%;margin:0 0 0 0;padding:6px 2% 6px 2%;font-family:"trebuchet MS", Verdana, sans-serif;color:#FFF;
background-color:#555555;}
#content{margin:0 0 0 2%;position:relative;}
.content-container{background:#FFF;width:96%;margin-top:8px;padding:10px;position:relative;}
-->
</style>
</head>
<body>
<div id="header"><h1>Server Error</h1></div>
<div id="content">
<div class="content-container"><fieldset>
<h2>502 - Web server received an invalid response while acting as a gateway or proxy server.</h2>
<h3>There is a problem with the page you are looking for, and it cannot be displayed. When the Web server (while acting as a gateway or proxy) contacted the upstream content server, it received an invalid response from the content server.</h3>
</fieldset></div>
</div>
</body>
</html>
;
---> nested com.sssw.b2b.rt.GNVException: rt008103:
Status: 502
HTTP Reason: Bad Gateway
Error Info: <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1"/>
<title>502 - Web server received an invalid response while acting as a gateway or proxy server.</title>
<style type="text/css">
<!--
body{margin:0;font-size:.7em;font-family:Verdana, Arial, Helvetica, sans-serif;background:#EEEEEE;}
fieldset{padding:0 15px 10px 15px;}
h1{font-size:2.4em;margin:0;color:#FFF;}
h2{font-size:1.7em;margin:0;color:#CC0000;}
h3{font-size:1.2em;margin:10px 0 0 0;color:#000000;}
#header{width:96%;margin:0 0 0 0;padding:6px 2% 6px 2%;font-family:"trebuchet MS", Verdana, sans-serif;color:#FFF;
background-color:#555555;}
#content{margin:0 0 0 2%;position:relative;}
.content-container{background:#FFF;width:96%;margin-top:8px;padding:10px;position:relative;}
-->
</style>
</head>
<body>
<div id="header"><h1>Server Error</h1></div>
<div id="content">
<div class="content-container"><fieldset>
<h2>502 - Web server received an invalid response while acting as a gateway or proxy server.</h2>
<h3>There is a problem with the page you are looking for, and it cannot be displayed. When the Web server (while acting as a gateway or proxy) contacted the upstream content server, it received an invalid response from the content server.</h3>
</fieldset></div>
</div>
</body>
</html>
java.io.InvalidObjectException:
at com.sssw.fw.directory.core.EboUserCredentials.readResolve(EboUserCredentials.java:236)
at sun.reflect.GeneratedMethodAccessor779.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at java.io.ObjectStreamClass.invokeReadResolve(ObjectStreamClass.java:1148)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1810)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
at java.util.concurrent.ConcurrentHashMap.readObject(ConcurrentHashMap.java:1445)
at sun.reflect.GeneratedMethodAccessor117.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1058)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1900)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2000)
at java.io.ObjectInputStream.defaultReadObject(ObjectInputStream.java:501)
at com.sssw.fw.session.EboBaseSession.readObject(EboBaseSession.java:241)
at sun.reflect.GeneratedMethodAccessor118.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1058)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1900)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
at org.apache.catalina.ha.session.DeltaRequest$AttributeInfo.readExternal(DeltaRequest.java:378)
at org.apache.catalina.ha.session.DeltaRequest.readExternal(DeltaRequest.java:268)
at org.apache.catalina.ha.session.DeltaManager.deserializeDeltaRequest(DeltaManager.java:591)
at org.apache.catalina.ha.session.DeltaManager.handleSESSION_DELTA(DeltaManager.java:1279)
at org.apache.catalina.ha.session.DeltaManager.messageReceived(DeltaManager.java:1226)
at org.apache.catalina.ha.session.DeltaManager.messageDataReceived(DeltaManager.java:944)
at org.apache.catalina.ha.session.ClusterSessionListener.messageReceived(ClusterSessionListener.java:91)
at org.apache.catalina.ha.tcp.SimpleTcpCluster.messageReceived(SimpleTcpCluster.java:936)
at org.apache.catalina.ha.tcp.SimpleTcpCluster.messageReceived(SimpleTcpCluster.java:917)
at org.apache.catalina.tribes.group.GroupChannel.messageReceived(GroupChannel.java:278)
at org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:82)
at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.messageReceived(TcpFailureDetector.java:117)
at org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:82)
at org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:82)
at org.apache.catalina.tribes.group.ChannelCoordinator.messageReceived(ChannelCoordinator.java:252)
at org.apache.catalina.tribes.transport.ReceiverBase.messageDataReceived(ReceiverBase.java:287)
at org.apache.catalina.tribes.transport.nio.NioReplicationTask.drainChannel(NioReplicationTask.java:210)
at org.apache.catalina.tribes.transport.nio.NioReplicationTask.run(NioReplicationTask.java:99)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.sssw.fw.util.crypto.DecryptionException: Decryption failed.
at com.sssw.fw.util.crypto.KeyManager.decrypt(KeyManager.java:263)
at com.sssw.fw.directory.core.EboUserCredentials.decrypt(EboUserCredentials.java:220)
at com.sssw.fw.directory.core.EboUserCredentials.readResolve(EboUserCredentials.java:234)
... 47 more
Caused by: javax.crypto.BadPaddingException: Given final block not properly padded
at com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:966)
at com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:824)
at com.sun.crypto.provider.AESCipher.engineDoFinal(AESCipher.java:436)
at javax.crypto.Cipher.doFinal(Cipher.java:2165)
at com.sssw.fw.util.crypto.KeyManager.decrypt(KeyManager.java:256)
... 49 more
2017-01-19 00:00:20,696 [pool-6-thread-6] INFO com.novell.soa.af.impl.LogEvent- [RBPM] [Workflow_Forwarded] Initiated by System, Process ID: a73dfebbf2364cb2bdd0b67703a011e7, Process Name: cn=RemedyWorkflow,cn=RequestDefs,cn=AppConfig,cn=UserApplication,cn=DriverSet,o=system:56, Activity: Activity, Recipient: cn=USER1,ou=Users,o=novell
2017-01-19 00:00:20,705 [pool-6-thread-6] INFO com.novell.soa.af.impl.activity.LogActivity- [RBPM] [User_Message] Initiated by cn=uaadmin,ou=sa,o=novell, Process ID: a73dfebbf2364cb2bdd0b67703a011e7, Process Name: cn=RemedyWorkflow,cn=RequestDefs,cn=AppConfig,cn=UserApplication,cn=DriverSet,o=system:56, Activity: Activity4, Recipient: cn=USER1,ou=Users,o=novell, Author: {Enter author name here}, Message: GEN_V1_NOVIDVTREE_DEV_UserApplication_Remedy Workflow_Activity##Thu Jan 19 00:00:20 UTC 2017##-1##1##com.sssw.b2b.rt.GNVException: rt001801:Document I/O error:
Status: 502
HTTP Reason: Bad Gateway
Error Info: <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1"/>
<title>502 - Web server received an invalid response while acting as a gateway or proxy server.</title>
<style type="text/css">
<!--
body{margin:0;font-size:.7em;font-family:Verdana, Arial, Helvetica, sans-serif;background:#EEEEEE;}
fieldset{padding:0 15px 10px 15px;}
h1{font-size:2.4em;margin:0;color:#FFF;}
h2{font-size:1.7em;margin:0;color:#CC0000;}
h3{font-size:1.2em;margin:10px 0 0 0;color:#000000;}
#header{width:96%;margin:0 0 0 0;padding:6px 2% 6px 2%;font-family:"trebuchet MS", Verdana, sans-serif;color:#FFF;
background-color:#555555;}
#content{margin:0 0 0 2%;position:relative;}
.content-container{background:#FFF;width:96%;margin-top:8px;padding:10px;position:relative;}
-->
</style>
</head>
<body>
<div id="header"><h1>Server Error</h1></div>
<div id="content">
<div class="content-container"><fieldset>
<h2>502 - Web server received an invalid response while acting as a gateway or proxy server.</h2>
<h3>There is a problem with the page you are looking for, and it cannot be displayed. When the Web server (while acting as a gateway or proxy) contacted the upstream content server, it received an invalid response from the content server.</h3>
</fieldset></div>
</div>
</body>
</html>
;
---> nested com.sssw.b2b.rt.GNVException: rt008103:
Status: 502
HTTP Reason: Bad Gateway
Error Info: <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1"/>
<title>502 - Web server received an invalid response while acting as a gateway or proxy server.</title>
<style type="text/css">
<!--
body{margin:0;font-size:.7em;font-family:Verdana, Arial, Helvetica, sans-serif;background:#EEEEEE;}
fieldset{padding:0 15px 10px 15px;}
h1{font-size:2.4em;margin:0;color:#FFF;}
h2{font-size:1.7em;margin:0;color:#CC0000;}
h3{font-size:1.2em;margin:10px 0 0 0;color:#000000;}
#header{width:96%;margin:0 0 0 0;padding:6px 2% 6px 2%;font-family:"trebuchet MS", Verdana, sans-serif;color:#FFF;
background-color:#555555;}
#content{margin:0 0 0 2%;position:relative;}
.content-container{background:#FFF;width:96%;margin-top:8px;padding:10px;position:relative;}
-->
</style>
</head>
<body>
<div id="header"><h1>Server Error</h1></div>
<div id="content">
<div class="content-container"><fieldset>
<h2>502 - Web server received an invalid response while acting as a gateway or proxy server.</h2>
<h3>There is a problem with the page you are looking for, and it cannot be displayed. When the Web server (while acting as a gateway or proxy) contacted the upstream content server, it received an invalid response from the content server.</h3>
</fieldset></div>
</div>
</body>
</html>
Resolution
This issue was isolated around the Oracle database RBPM was configured to use.
Restarting the Oracle database and database listener on the clustered node resolved the issue.
Restarting the Oracle database and database listener on the clustered node resolved the issue.