NetIQ Access Manager NIDP server reports: the error: client_id is invalid on OAuth grant request

• NetIQ Access Manager IDP server configured with 3rd Party OAuth client.
  
  
• Grant request fails with error:
  
  { "error":"invalid_client",
       "error_description":"invalid_client: 396865ae-157d-4ec5-8a7b-f5f24276b52e
    client_id is invalid" }
  
  
• NIDP server catalina.out logs the following JAVA exception while starting up:
  
  java.lang.NullPointerException
      at com.novell.nam.nidp.oauth.config.authority.ldap.OAuthClientConfigAuthorityLdap.getClient(OAuthClientConfigAuthorityLdap.java:174)
      at com.novell.nam.nidp.oauth.config.authority.ldap.OAuthClientConfigAuthorityLdap.configureClient(OAuthClientConfigAuthorityLdap.java:30)
      at com.novell.nam.nidp.oauth.config.handler.OAuth2LdapConfigStore.readClientConfig(OAuth2LdapConfigStore.java:97)
      at com.novell.nam.nidp.oauth.config.OAuth2ConfigManager.loadOAuth2Client(OAuth2ConfigManager.java:204)
      at com.novell.nam.nidp.oauth.config.OAuth2ConfigManager.loadOAuth2Clients(OAuth2ConfigManager.java:188)
      at com.novell.nam.nidp.oauth.config.OAuth2ConfigManager.loadOAuth2Config(OAuth2ConfigManager.java:168)
      at com.novell.nam.nidp.oauth.config.OAuth2ConfigManager.loadTenant(OAuth2ConfigManager.java:124)
      at com.novell.nam.nidp.oauth.config.OAuth2ConfigManager.configure(OAuth2ConfigManager.java:101)
      at com.novell.nam.nidp.oauth.nidp.servlets.OAuth2ProtocolInitializer.initialize(OAuth2ProtocolInitializer.java:76)
      at com.novell.nidp.NIDPMeEntity.A(y:461)
      at com.novell.nidp.NIDPMeEntity.A(y:2180)
      at com.novell.nidp.NIDPMeEntity.hardInitialize(y:3277)
      at com.novell.nidp.NIDPContext.A(y:3527)
      at com.novell.nidp.NIDPContext.start(y:2229)
      at com.novell.nidp.servlets.NIDPServletContext.doCommand(y:3601)
      at com.novell.jcc.client.JCCClientImpl$JCCRMIClient.doCommand(y:1853)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:323)
      at sun.rmi.transport.Transport$1.run(Transport.java:200)
      at sun.rmi.transport.Transport$1.run(Transport.java:197)
      at java.security.AccessController.doPrivileged(Native Method)
      at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
      at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568)
      at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826)
      at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:683)
      at java.security.AccessController.doPrivileged(Native Method)
      at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:745)
  
  
• NIDP server catalina.out logs the following warning during process of loading the OAuth configuration:
  
  May 04, 2016 11:20:44 AM com.novell.nam.nidp.oauth.config.authority.ldap.OAuthClientConfigAuthorityLdap getClientConfigByDn
  WARNING: Could not find the client by dn.
  <amLogEntry> 2016-05-04T09:20:44Z DEBUG NIDS Application:
  Method: JNDILogEventListener.accept
  Thread: RMI TCP Connection(2)-127.0.0.1
  getNextConnection() attempting to get preferred replica from the IPreferredReplica interface </amLogEntry>
  
  
• The exact same client configuration works on another NetIQ Access Manager test setup

• Note: Please use the steps below after being sure you have a complete backup of your system
  on your own risk
  
  
  
• The key to solve this problem is the error message:
  
  May 04, 2016 11:20:44 AM com.novell.nam.nidp.oauth.config.authority.ldap.OAuthClientConfigAuthorityLdap getClientConfigByDn
  WARNING: Could not find the client by dn.
  <amLogEntry> 2016-05-04T09:20:44Z DEBUG NIDS Application:
  Method: JNDILogEventListener.accept
  Thread: RMI TCP Connection(2)-127.0.0.1
  getNextConnection() attempting to get preferred replica from the IPreferredReplica interface </amLogEntry>
  
  
• The object class "nidsOAuthClients" stores the OAuth client configuration
  
  
• The Client_ID will be stored at:
  
  
• the "nidsDisplayName" Attribute on the "nidsOAuthClients" object class store the OAuth Client ID
• the "nidsOAuthClientXML" Attribute (which is an XML document)  as "<ApplicationConfig id="
  
  
• use iManager to search for any object with the class "nidsAuthClients"
  iManager => "View Objects" => Search => Context: "cluster.nids.accessManagerContainer.novell" ; Type: "nidsOAuthClients"
  
  
• review each of the "nidsOAuthClients"objects and check if the "nidsDisplayName" is empty.
  
• review the "nidsOAuthClientXML" Attribute on this object and check if the "<ApplicationConfig id="  matches the failing OAuth client ID
• if the above condition is true remove the "nidsOAuthClients" object and restart your NIDP server

General useful OAuth troubleshooting options:

1. Please enable the following NIDP cluster logging options:
     Enable File Logging
     Echo To Console,
     Component File logger Levels:
        Application = debug"
   
   
2. Install the fiddler header trace tool at your browser client and enable SSL decryption
3. ssh into your NIDP server
4. edit the NDIP server logging config file: vi /opt/novell/nids/lib/webapp/WEB-INF/classes/log4j.properties
5. apply the following settings:
   =========================================================
   # Set root logger priority to FATAL and its only appender to CONSOLE.
   #log4j.rootCategory=FATAL, CONSOLE
   log4j.rootCategory=DEBUG, CONSOLE
   
   # CONSOLE is set to be a ConsoleAppender using a PatternLayout.
   log4j.appender.CONSOLE=org.apache.log4j.ConsoleAppender
   #log4j.appender.CONSOLE.Threshold=FATAL
   log4j.appender.CONSOLE.Threshold=DEBUG
   log4j.appender.CONSOLE.layout=org.apache.log4j.PatternLayout
   log4j.appender.CONSOLE.layout.ConversionPattern=- %m%n
   #
   # added for OAuth debugging
   log4j.logger.org.brickred.socialauth=DEBUG
   log4j.logger.org.brickred.socialauth.provider=DEBUG
   ==========================================================
6. stop your NIDP server: "/etc/init.d/novell-idp stop"
7. clean out the catalina.out: "echo > /var/opt/novell/nam/logs/idp/tomcat/catalina.out"
8. start your NIDP server: "/etc/init.d/novell-idp start"
   
   

Note: for the problem reported on this TID running "ndstrace" during the NIDP server start-up process would help to provide details which objects are getting accessed while running into the Java Exception.