Environment
Situation
Customer reported that suddenly all of their policies that use Credential Profile attributes for Single Sign on via Formfill (FF) and Identity Injection (II) stopped working all of a sudden.
Symptoms:
See the following errors in the catalina.out on the Identity Server (IDP):
"Completed Request. Response: WSCResponse:
Status: All Failure "..."Status: DataNotAvailable" (See Additional Information below for full snippet of WSCResponse)
Also saw the following message in the catalina.out of IDP:
"Unable to correctly initialize a WSFCache.
Exception message: "/opt/novell/nam/idp/webapps/nidp/WEB-INF/swap/WSCCACHEALREADYREAD_hle12ktmw9nv.SWP (Permission denied)" "
Resolution
Changed owndership of swap directory to novlwww:novlwww as it should be.
Example: chown novlwww:novlwww -R /opt/novell/nids/lib/webapp/WEB-INF/swap
HINT: When running into odd issues like this, it's a good idea to ensure that you enable debug logging on the IDP server for Application, Liberty, Web Consumer and Web Provider. Restart tomcat on the IDP and then once fully loaded, duplicate the problem and then search the catalina.out for "permission denied" messages.
Cause
Somehow the permissions of the swap directory located at /opt/novell/nam/idp/webapps/nidp/WEB-INF/swap/ was changed from novlwww:novlwww to root:root
The IDP in general runs as novlwww user. As a result the IDP could not write to this directory.
Couldn't determine if this was done manually by user or if somehow the IDP code did it. A code review couldn't find any clues on this happening programatically.
Additional Information
From Catalina.out on Identity Server:
Completed Request. Response: WSCResponse:
Status: All Failure
WSCQResponseEntry:
WSCQSSToken:
Model Entry: Entry
Unique Id:
NEPXurn~3Anovell~3Acredentialprofile~3A2005-03~2Fcp~3ASecrets~2Fcp~3ASecret~2Fcp~3AEntry~40~40~40~40WSCQSSToken~40~40~40~40~2Fcp~3ASecrets~2Fcp~3ASecret~5Bcp~3AName~3D~22LDAPCredentials~22~5D~2Fcp~3AEntry~5Bcp~3AName~3D~22UserPassword~22~5D
Select String:
/cp:Secrets/cp:Secret[cp:Name="LDAPCredentials"]/cp:Entry[cp:Name="UserPassword"]
Overridden Display Name ResourceId:: SS.WKSELdapCredsUserPassword
Status: DataNotAvailable
WSCQResponseEntry:
WSCQSSToken:
Model Entry: Entry
Unique Id:
NEPXurn~3Anovell~3Acredentialprofile~3A2005-03~2Fcp~3ASecrets~2Fcp~3ASecret~2Fcp~3AEntry~40~40~40~40WSCQSSToken~40~40~40~40~2Fcp~3ASecrets~2Fcp~3ASecret~5Bcp~3AName~3D~22LDAPCredentials~22~5D~2Fcp~3AEntry~5Bcp~3AName~3D~22UserDN~22~5D
Select String:
/cp:Secrets/cp:Secret[cp:Name="LDAPCredentials"]/cp:Entry[cp:Name="UserDN"]
Overridden Display Name ResourceId:: SS.WKSELdapCredsUserDN
Status: DataNotAvailable
WSCQResponseEntry:
WSCQSSToken:
Model Entry: Entry
Unique Id:
NEPXurn~3Anovell~3Acredentialprofile~3A2005-03~2Fcp~3ASecrets~2Fcp~3ASecret~2Fcp~3AEntry~40~40~40~40WSCQSSToken~40~40~40~40~2Fcp~3ASecrets~2Fcp~3ASecret~5Bcp~3AName~3D~22LDAPCredentials~22~5D~2Fcp~3AEntry~5Bcp~3AName~3D~22UserName~22~5D
Select String:
/cp:Secrets/cp:Secret[cp:Name="LDAPCredentials"]/cp:Entry[cp:Name="UserName"]
Overridden Display Name ResourceId:: SS.WKSELdapCredsUserName
Status: DataNotAvailable
WSCQResponse: </amLogEntry>