Troubleshooting Application Authentication to DSfW

Basics on troubleshooting applications authenticating to DSfW.

To troubleshoot an application authenticating to a DSfW domain the following logs and traces will be needed.

1) Packet Trace (tcpdump or wireshark).  Take the packet traces from the DSfW server and server running the application.  Some times a workstation is involved or other servers are involved with the configuration and running of the application.  If that is the case take packet traces from those workstations and servers.  For Windows server(s) or workstation(s) where a packet trace is taken from disable the secure channel encryption.

2) ldap/nmas ndstrace  (TID 7009602)
3) The /var/opt/novell/xad/log/kdc.log

4) The /var/log/samba/log.smbd (samba log) with debug enabled

5) The /var/log/samba/log.nmbd

6) The /var/log/samba/log.winbindd

7) The /var/log/messages

8) /var/opt/novell/log/named/named.run (optional)

9) Event viewer logs (if application is running on windows server)

Generally the kdc.log, messages log, ndstrace (ldap), and packet trace have the most information, but sometimes a debug of samba is also needed.

The Packet Trace
Turn off secure channel encryption:
Before taking the packet trace turn of the secure channel encryption

A registry change is required to disable netlogon channel encryption. Change RequireSignOrSeal  from 1 to 0.

HKLM\SYSTEM\CurrentControlSet\Services\Netlogon\Parameters REG_DWORD  RequireSignOrSeal = 0 (Channel traffic need not be signed or sealed)

 

Taking the packet trace

Use tcpdump or wireshark to take a packet trace from the DSfW server.  A packet trace from the workstation might also be necessary.

tcpdump -n -v -i <interface> -s0 -w /<path>/<name_of_lan_trace>.cap

Press cntrl c to stop the trace.

To find the interface use ifconfig.  It will show the interfaces the the ip addresses.  Usually the interface is eth0 or eth1.

If there is only one IP address bound on the server another option is -i any.  Using any will listen on all interfaces. 

When using tcpdump between a server and workstation on the same network a filter can be helpful in filtering traffic.

In this example of using tcpdump the workstations IP is 192.168.100.200,

the servers interface is eth0, and the output is written to/tmp/wk_join_dsfw.cap :

Example with a filter between the host (dsfw server) and the windows computer 192.168.100.200:

tcpdump -n -v -s0 -i eth0 'host 192.168.100.200' -w /tmp/app_auth_dsfw.cap

The LDAP/NMAS Trace
To take the ldap trace first check the screen options, make sure the level is set to all
ldapconfig get |grep -i "ldap screen level"

Example of setting the screen level to all:
ldapconfig -s "ldap screen level=all" -a admin.novell

Then start the ndstrace

ndstrace	# brings up the ndstrace utility
set dstrace = nodebug	# Clear the filter
dstrace NMAS LDAP TIME TAGS AUTH	# Enable the LDAP, NMAS, TIME, TAGS, and AUTH
set ndstrace = *r	# Clear the log or rename the /var/opt/novell/eDirectory/log/ndstrace.log
ndstrace = on	# Start the logging and execute your command or task
set ndstrace = off	# This will stop logging
quit	# Exit ndstrace

The default location for the log is /var/opt/novell/eDirectory/log/ndstrace.log

 

To enable smb debug open  /etc/samba/smb.conf and at the end of the [global] section add  log level =10 or from the terminal type smbcntrl -d 10

The restart of smbd or the other DSfW services is not needed.

Examining the traces and logs
There are three main protocols to look in the traces:
   KERBEROS
   LDAP
   DNS

Start with these three protocols.  Depending on the issue other protocols might come into play like smb (Samba) or netbios.

For authentication issues look at kerberos packets in the packet trace and kerberos errors in the kdc.log and /var/log/messages.
Starting with the packet trace by filtering on kerberos.
To filter traffic in the packet trace simple type kerberos in the filter window and click apply.
Look for errors in the details of the AS-REQ, AS-REP and TGS-REQ, TGS-REP (requests and responses) packets.

If there is a kerberos error in the packet trace
1) Check that the principal (user or computer) in the request is valid.
2) Troubleshoot any errors found in the trace.
note: pre-authentication required errors (krb5kdc_error_preauth_required 25) are standard.  That error is to notify the sender that pre authentication (pa-enc-timestamp in the padata) is required for added security. 
3) Look in the kdc.log for error.

Some common errors seen in the kdc.log are:

Decrypt integrity check failed - bad password
    Look at the next line for the user or workstation trying to login with a bad password.  Workstations will have a $ at the end of the name and before the @domain name.
example:
AS_REQ 192.168.0.4 PREAUTH_FAILED: <workstation$@dsfw.lan> for krbtgt/dsfw.lan Preauthentication failed
Can cause slow logins, cause the domain controller to become unresponsive, or even crash the domain controller.  Implement an intruder lock out on the container where the user(s) or workstation(s) objects reside.

locked out - account has been locked out
    If this is for a workstation account this error message usually means their is a workstation with the same name trying to login.  The workstation with the duplicate name will attempt to login several time, triggering the intruder lockout and thus generating the Decrypt integrity check error.
If this is a user, the user account is locked usually do to intruder lockout.

client not found - account is not found in domain
    Similar to a 601 in eDirectory.  Look at the ndstrace to see if the search request is valid.  Check that the account exists in the domain and is samified.  This error can cause slow logins as well.
Sometimes taking a ndstrace with +time +tags +auth +ldap +vcln +dbg will help as well.

For password errors look at the ndstrace with ldap and nmas enabled in the filter.  Look for any NMAS errors.

Next filter on ldap in the packet trace.
For directory resolution of a user or computer object look at LDAP traffic in the packet trace and ndstrace.
See if the request is received and responding to the application properly.  There might be an error resolving an object or an attribute is requested in the search which doesn't exist or has the wrong value on the object.

For the ndstrace start usually the standard filters for a ldap/nmas trace are all that is needed.  These filters are TIME, TAGS, AUTH, LDAP and NMAS. 
Depending on the issue other filters might be enabled.   RECM, RSLV, and VCLN are often useful. 
Some times Agent Buffers (ABUF), Debug (DBG), and Miscellaneous (MISC) will also need to be enabled.  Beware ABUF, DBG, and MISC can greatly increase information that is returned and make it difficult find errors. 
NMAS should be enabled for authentication or password issues and always. 
Always enable TIME and TAGS.

If the ldap traffic is minimal or non existant it might be the the ldap screen level on the ldap server object was not set.  It should be set to ALL or "Operation| Connection| Config| Extensions| Error| Critical| DataConnection" at a minimum.
For more information on using ndstrace look at TID 7009602

If there are no kerberos packets or ldap traffic in the packet trace, there might be an issue with DNS resolving the domain or server.  Filter on DNS traffic, look for resolution errors. 
The /var/opt/novell/log/named/named.run and /var/log/messages generally provides more information on DNS resolution as well.

download the ndsPacketTrace script from CoolSolutions.