Environment
Access Manager 4.3
Access Manager 4.2
Access Gateway on all platforms
Identity Server on all platforms
Situation
Access Manager setup and working fine. Administrator makes a change to the IDP or ESP cluster and after upgrading the IDP or AG, the service fails to start. Looking at the IDP or Admin Console logs after the event, there's an indication that the keystore cannot be found. In some of the reported cases, the administrator was changing certificates or trusted roots, but on other occasions no certificate changes were reported before seeing the issue.
In each case, the catalina.out file and app_sc log file on the Admin Console can give a clue as to what is happening. Here's an entry from the catalina at the time the change was made:
// catalina snippet - note the delete operation after error
<amLogEntry> 2017-04-10T08:30:43Z SEVERE DeviceManager: AM#100901031: Error - In CertHandler.getClusterDisplayName for cluster (SCCtv7nez): The
cluster object was not found in the configuration store. </amLogEntry>
Deleting orphaned cluster keystores of nonexistent NIDP cluster (SCCtv7nez).
Looking at the app_sc logs for the same timestamp, we can see the following where we again appear to be deleting the keystore:
138098(D)2017-04-10T06:30:42Z(L)webui.sc(T)112(C)com.volera.roma.app.handler.CertHandler(M)createTrustedRoot(Msg)In createTrustedRoot - adding a new trusted root TEMP_DP_MaQS_Mavi
138099(D)2017-04-10T06:30:42Z(L)application.sc.core(T)112(C)com.volera.vcdn.application.sc.core.KeyManager(M)
138099(D)2017-04-10T06:30:42Z(L)application.sc.core(T)112(C)com.volera.vcdn.application.sc.core.KeyManager(M)
createCertEntry(Msg)creating cert context ou=TEMP_DP_MaQS_Mavi
138100(D)2017-04-10T06:30:42Z(L)webui.sc(T)112(C) com.volera.vcdn.webui.sc.core.UIManager(M)log(Msg)key cert.cert_select.aliasPerTruststore not found in resources.volsc.SystemControllerResources
138101(D)2017-04-10T06:30:43Z(L)application.sc.core(T)112(C)com.volera.vcdn.application.sc.core.KeyManager(M)deleteKeyStoreEntry(Msg)deleting cert context ou=SCCtv7nez-signing
138102(D)2017-04-10T06:30:43Z(L)application.sc.core(T)112(C)com.volera.vcdn.application.sc.core.KeyManager(M)
138100(D)2017-04-10T06:30:42Z(L)webui.sc(T)112(C) com.volera.vcdn.webui.sc.core.UIManager(M)log(Msg)key cert.cert_select.aliasPerTruststore not found in resources.volsc.SystemControllerResources
138101(D)2017-04-10T06:30:43Z(L)application.sc.core(T)112(C)com.volera.vcdn.application.sc.core.KeyManager(M)deleteKeyStoreEntry(Msg)deleting cert context ou=SCCtv7nez-signing
138102(D)2017-04-10T06:30:43Z(L)application.sc.core(T)112(C)com.volera.vcdn.application.sc.core.KeyManager(M)
deleteKeyStoreEntry(Msg)deleting cert context ou=SCCtv7nez-encryption
138103(D)2017-04-10T06:30:43Z(L)application.sc.core(T)112(C) com.volera.vcdn.application.sc.core.KeyManager(M)deleteKeyStoreEntry(Msg)deleting cert context ou=SCCtv7nez-connector
138104(D)2017-04-10T06:30:43Z(L)application.sc.core(T)112(C)com.volera.vcdn.application.sc.core.KeyManager(M)deleteKeyStoreEntry(Msg)deleting cert context ou=SCCtv7nez-truststore
138105(D)2017-04-10T06:30:43Z(L)application.sc.core(T)112(C)com.volera.vcdn.application.sc.core.KeyManager(M)
138103(D)2017-04-10T06:30:43Z(L)application.sc.core(T)112(C) com.volera.vcdn.application.sc.core.KeyManager(M)deleteKeyStoreEntry(Msg)deleting cert context ou=SCCtv7nez-connector
138104(D)2017-04-10T06:30:43Z(L)application.sc.core(T)112(C)com.volera.vcdn.application.sc.core.KeyManager(M)deleteKeyStoreEntry(Msg)deleting cert context ou=SCCtv7nez-truststore
138105(D)2017-04-10T06:30:43Z(L)application.sc.core(T)112(C)com.volera.vcdn.application.sc.core.KeyManager(M)
deleteKeyStoreEntry(Msg)deleting cert context ou=SCCtv7nez-ocsp-truststore
138106(D)2017-04-10T06:30:43Z(L)application.sc.core(T)112(C) com.volera.vcdn.application.sc.core.KeyManager(M)deleteKeyStoreEntry(Msg)deleting cert context ou=SCCtv7nez-provider
138107(D)2017-04-10T06:30:43Z(L)application.sc.core(T)112(C)com.volera.vcdn.application.sc.core.KeyManager(M)deleteKeyStoreEntry(Msg)deleting cert context ou=SCCtv7nez-consumer
138106(D)2017-04-10T06:30:43Z(L)application.sc.core(T)112(C) com.volera.vcdn.application.sc.core.KeyManager(M)deleteKeyStoreEntry(Msg)deleting cert context ou=SCCtv7nez-provider
138107(D)2017-04-10T06:30:43Z(L)application.sc.core(T)112(C)com.volera.vcdn.application.sc.core.KeyManager(M)deleteKeyStoreEntry(Msg)deleting cert context ou=SCCtv7nez-consumer
Cause
Fixed in NAM 4.2.5 and NAM 4.3.3 releases (and shipping NAM 4.4.0).
Additional Information
This is a random issue that is seen if keystores are not found. It assumed that the cluster is orphaned and deleted it, which we no longer do.