Environment
Open Enterprise Server 11 SP2 (OES11 SP2)
Open Enterprise Server 11 SP1 (OES11 SP1)
Open Enterprise Server 11 (OES11)
Open Enterprise Server 11 (OES11)
Domain Services for Windows
DSfW
Situation
Troubleshoot Kerberos
Kerberos fails to start
xad-krb5kdc fails to start
xadsd fails to start
Resolution
Missing uniquedomainid attribute on key objects
Do a ldapsearch for object with out uniquedomainid replace <domain name> with the name of the domain. Export the LDAPCONF before doing the ldap search (7003070).
export LDAPCONF=/etc/opt/novell/xad/openldap/ldap.conf
ldapsearch -Y EXTERNAL -LLL -Q -b <domain name> -s sub '(!(uniquedomainid=*))' dn
example:
ldapsearch -Y EXTERNAL -LLL -Q -b dc=novell,dc=com -s sub '(!(uniquedomainid=*))' dn
Missing uniquedomainid attribute on key objects
If an error "ldap_sasl_interactive_bind_s: Can't contact LDAP server (-1)" is returned proceed to Missing ldap interfaces on the ldap server object.
Objects that are missing the uniquedomainid should have it added. Use a ldif to add the attribute.
Some key objects to check for the uniquedomainid attribute are: krbtgt, domain controller object, domain controller container and the container mapped to the domain. If these 4 objects have a uniquedomainid then usually kerberos will start. Once all DSfW services are running the domain can be re-samified which should add the uniquedomainid attribute to any object in the domain missing this key attribute. See TID 7009851.
Missing ldap interfaces on the ldap server object
The ldap interfaces are missing from the LDAP Server object for the DSfW server.
This usually is a result of the ldap Group and ldap server objects were deleted and recreated.
The interfaces on the server objects and the mappings on the group object are unique on DSfW servers and need to be recreated with the correct settings.
See TID 7010319 on how to correctly recreate these objects.
Corrupt or missing libraries
Check that the libraries are in place and the symbolic links to the libraries.
Here is a list of key xad, ldap, and kerberos libraries with symbolic links as of the November 2012 Maintenance patch
/opt/novell/xad/lib64/krb5/plugins/kdb
-rwxr-xr-x 1 root root 74896 Nov 27 14:31 kdb_xad.so*
/opt/novell/xad/lib64/sasl2
-rwxr-xr-x 1 root root 31888 Aug 3 06:31 libgssapiv2.so*
:/opt/novell/xad/lib64/gss
lrwxrwxrwx 1 root root 25 Dec 14 12:40 libmech_netlogon.so.1 -> libmech_netlogon.so.1.0.0*
-rwxr-xr-x 1 root root 759960 Nov 27 14:31 libmech_netlogon.so.1.0.0*
-rwxr-xr-x 1 root root 123144 Aug 3 06:54 mech_krb5.so*
-rwxr-xr-x 1 root root 797528 Nov 27 14:31 mech_ntlm.so*
-rwxr-xr-x 1 root root 48272 Aug 3 06:54 mech_spnego.so*
/opt/novell/xad/lib64/nds-modules
-rwxr-xr-x 1 root root 14848 Nov 27 14:31 libanr-plugin.so*
-rwxr-xr-x 1 root root 35808 Nov 27 14:31 libcrossref-plugin.so*
-rwxr-xr-x 1 root root 23232 Nov 27 14:31 libdsearch-plugin.so*
-rwxr-xr-x 1 root root 263560 Nov 27 14:31 libnad-plugin.so*
-rwxr-xr-x 1 root root 31664 Nov 27 14:31 libnetlogon-plugin.so*
-rwxr-xr-x 1 root root 36256 Nov 27 14:31 libntacl-plugin.so*
-rwxr-xr-x 1 root root 32176 Nov 27 14:31 libsubschema-plugin.so*
-rwxr-xr-x 1 root root 15040 Nov 27 14:31 libtokengroups-plugin.so*
-rwxr-xr-x 1 root root 6448 Nov 27 14:31 libwhoami-plugin.so*
/opt/novell/xad/lib64
lrwxrwxrwx 1 root root 21 Dec 14 12:40 libadmpasswd.so.2 -> libadmpasswd.so.2.0.0*
-rwxr-xr-x 1 root root 6312 Nov 27 14:31 libadmpasswd.so.2.0.0*
-rwxr-xr-x 1 root root 32368 Nov 27 14:31 libauth_gss.so*
-rwxr-xr-x 1 root root 23248 Nov 27 14:31 libauth_netlogon.so*
lrwxrwxrwx 1 root root 17 Oct 8 10:49 libcom_err.so.3 -> libcom_err.so.3.0*
-rwxr-xr-x 1 root root 23136 Aug 3 06:54 libcom_err.so.3.0*
lrwxrwxrwx 1 root root 23 Dec 14 12:39 libdcepthreads.so.1 -> libdcepthreads.so.1.0.0*
-rwxr-xr-x 1 root root 35784 Nov 26 09:15 libdcepthreads.so.1.0.0*
lrwxrwxrwx 1 root root 18 Dec 14 12:39 libdcerpc.so.1 -> libdcerpc.so.1.0.2*
-rwxr-xr-x 1 root root 450496 Nov 26 09:15 libdcerpc.so.1.0.2*
lrwxrwxrwx 1 root root 16 Oct 8 10:49 libdes425.so.3 -> libdes425.so.3.0*
-rwxr-xr-x 1 root root 14688 Aug 3 06:54 libdes425.so.3.0*
lrwxrwxrwx 1 root root 19 Dec 14 12:40 libdssetup.so.1 -> libdssetup.so.1.0.0*
-rwxr-xr-x 1 root root 11200 Nov 27 14:31 libdssetup.so.1.0.0*
lrwxrwxrwx 1 root root 16 Oct 8 10:49 libgssapi.so.1 -> libgssapi.so.1.0*
-rwxr-xr-x 1 root root 80936 Aug 3 06:54 libgssapi.so.1.0*
lrwxrwxrwx 1 root root 21 Oct 8 10:49 libgssapi_krb5.so.2 -> libgssapi_krb5.so.2.2*
-rwxr-xr-x 1 root root 10512 Aug 3 06:54 libgssapi_krb5.so.2.2*
lrwxrwxrwx 1 root root 16 Oct 8 10:49 libgssrpc.so.4 -> libgssrpc.so.4.0*
-rwxr-xr-x 1 root root 115240 Aug 3 06:54 libgssrpc.so.4.0*
lrwxrwxrwx 1 root root 18 Oct 8 10:49 libk5crypto.so.3 -> libk5crypto.so.3.1*
-rwxr-xr-x 1 root root 854704 Aug 3 06:54 libk5crypto.so.3.1*
lrwxrwxrwx 1 root root 19 Oct 8 10:49 libkadm5clnt.so.5 -> libkadm5clnt.so.5.1*
-rwxr-xr-x 1 root root 81296 Aug 3 06:54 libkadm5clnt.so.5.1*
lrwxrwxrwx 1 root root 18 Oct 8 10:49 libkadm5srv.so.5 -> libkadm5srv.so.5.1*
-rwxr-xr-x 1 root root 106352 Aug 3 06:54 libkadm5srv.so.5.1*
lrwxrwxrwx 1 root root 14 Oct 8 10:49 libkdb5.so.4 -> libkdb5.so.4.0*
-rwxr-xr-x 1 root root 44104 Aug 3 06:54 libkdb5.so.4.0*
lrwxrwxrwx 1 root root 14 Oct 8 10:49 libkrb5.so.3 -> libkrb5.so.3.3*
-rwxr-xr-x 1 root root 606736 Aug 3 06:54 libkrb5.so.3.3*
lrwxrwxrwx 1 root root 21 Oct 8 10:49 libkrb5support.so.0 -> libkrb5support.so.0.1*
-rwxr-xr-x 1 root root 31720 Aug 3 06:54 libkrb5support.so.0.1*
lrwxrwxrwx 1 root root 18 Dec 14 12:40 liblsarpc.so.0 -> liblsarpc.so.0.0.0*
-rwxr-xr-x 1 root root 159184 Nov 27 14:31 liblsarpc.so.0.0.0*
-rwxr-xr-x 1 root root 19336 Nov 26 09:15 libnaf_ip.so*
-rwxr-xr-x 1 root root 19304 Nov 26 09:15 libnaf_np.so*
lrwxrwxrwx 1 root root 18 Dec 14 12:40 libnetapi.so.1 -> libnetapi.so.1.0.0*
-rwxr-xr-x 1 root root 105528 Nov 27 14:31 libnetapi.so.1.0.0*
lrwxrwxrwx 1 root root 17 Dec 14 12:40 libntacl.so.2 -> libntacl.so.2.0.0*
-rwxr-xr-x 1 root root 48904 Nov 27 14:31 libntacl.so.2.0.0*
lrwxrwxrwx 1 root root 19 Dec 14 12:40 libntdsapi.so.1 -> libntdsapi.so.1.0.0*
-rwxr-xr-x 1 root root 125808 Nov 27 14:31 libntdsapi.so.1.0.0*
-rwxr-xr-x 1 root root 169936 Nov 26 09:15 libprot_ncacn.so*
-rwxr-xr-x 1 root root 161760 Nov 26 09:15 libprot_ncadg.so*
-rwxr-xr-x 1 root root 6232 Nov 26 09:15 libpthread_ext.so*
-rwxr-xr-x 1 root root 6232 Nov 26 09:15 libpthread_ext.so.0*
-rwxr-xr-x 1 root root 10456 Nov 27 14:31 libw32time.so*
lrwxrwxrwx 1 root root 19 Dec 14 12:40 libwinbase.so.2 -> libwinbase.so.2.0.0*
-rwxr-xr-x 1 root root 82304 Nov 27 14:31 libwinbase.so.2.0.0*
lrwxrwxrwx 1 root root 18 Dec 14 12:40 libxaddrs.so.4 -> libxaddrs.so.4.0.0*
-rwxr-xr-x 1 root root 160672 Nov 27 14:31 libxaddrs.so.4.0.0*
lrwxrwxrwx 1 root root 18 Dec 14 12:40 libxaddsa.so.2 -> libxaddsa.so.2.0.0*
-rwxr-xr-x 1 root root 74512 Nov 27 14:31 libxaddsa.so.2.0.0*
lrwxrwxrwx 1 root root 18 Dec 14 12:40 libxaddsr.so.0 -> libxaddsr.so.0.0.0*
-rwxr-xr-x 1 root root 15488 Nov 27 14:31 libxaddsr.so.0.0.0*
lrwxrwxrwx 1 root root 18 Dec 14 12:40 libxadkcc.so.2 -> libxadkcc.so.2.0.0*
-rwxr-xr-x 1 root root 14928 Nov 27 14:31 libxadkcc.so.2.0.0*
lrwxrwxrwx 1 root root 18 Dec 14 12:40 libxadlsa.so.0 -> libxadlsa.so.0.0.0*
-rwxr-xr-x 1 root root 235584 Nov 27 14:31 libxadlsa.so.0.0.0*
lrwxrwxrwx 1 root root 18 Dec 14 12:40 libxadnds.so.2 -> libxadnds.so.2.0.0*
-rwxr-xr-x 1 root root 45032 Nov 27 14:31 libxadnds.so.2.0.0*
lrwxrwxrwx 1 root root 17 Dec 14 12:40 libxadnl.so.1 -> libxadnl.so.1.0.0*
-rwxr-xr-x 1 root root 206368 Nov 27 14:31 libxadnl.so.1.0.0*
lrwxrwxrwx 1 root root 19 Dec 14 12:40 libxadprov.so.2 -> libxadprov.so.2.0.0*
-rwxr-xr-x 1 root root 62144 Nov 27 14:31 libxadprov.so.2.0.0*
lrwxrwxrwx 1 root root 18 Dec 14 12:40 libxadsam.so.1 -> libxadsam.so.1.0.0*
-rwxr-xr-x 1 root root 207392 Nov 27 14:31 libxadsam.so.1.0.0*
lrwxrwxrwx 1 root root 18 Dec 14 12:40 libxadsdk.so.2 -> libxadsdk.so.2.0.0*
-rwxr-xr-x 1 root root 1316552 Nov 27 14:31 libxadsdk.so.2.0.0*
Mis-configured or missing kdc.conf
The kdc.conf is located at: /etc/opt/novell/xad/krb5.conf
Verify that a symbolic link exists at /etc/krb5.conf to /etc/opt/novell/xad/krb5.conf
sample kdc.conf
In this example the DSfW server's name is servername and the domain name is dsfw.lan
#ident $Id: krb5.conf,v 1.50 2006/03/05 02:58:10 lukeh Exp $
#
# Kerberos configuration for Domain Services for Windows
#
[libdefaults]
default_keytab_name = /var/opt/novell/xad/ds/krb5kdc/krb5.keytab
default_realm = DSFW.LAN
permitted_enctypes = arcfour-hmac-md5 des-cbc-md5 des-cbc-crc
default_tkt_enctypes = arcfour-hmac-md5
default_tgs_enctypes = arcfour-hmac-md5
noaddresses = true
case_sensitive = false
verify_ap_req_try_host_spn = true
dns_lookup_kdc = true
rdns = true
[appdefaults]
pam = {
krb4_convert = false
validate = true
ignore_unknown_upn = true
keytab = /var/opt/novell/xad/ds/krb5kdc/krb5.keytab
}
[logging]
kdc = FILE:/var/opt/novell/xad/log/kdc.log
admin_server = FILE:/var/opt/novell/xad/log/adm.log
kpasswdd = FILE:/var/opt/novell/xad/log/kpasswdd.log
kcm = FILE:/var/opt/novell/xad/log/kcm.log
[kcm]
system_ccache = {
principal = SERVERNAME$
spn_aliases = host/servername.dsfw.lan
spn_aliases = cifs/servername.dsfw.lan
spn_aliases = HTTP/servername.dsfw.lan
spn_aliases = ldap/servername.dsfw.lan
spn_aliases = DNS/servername.dsfw.lan
spn_aliases = HOST/SERVERNAME
}
[dbmodules]
XAD = {
db_module_dir = /opt/novell/xad/lib/krb5/plugins/kdb
db_library = kdb_xad
}
[realms]
DSFW.LAN = {
kdc = server.dsfw.lan
# kpasswd_server = server.dsfw.lan
database_module = XAD
}
[domain_realm]
.servername.dsfw = DSFW.LAN
servername.dsfw = DSFW.LAN
0 Byte stream file for sasLoginServerMethogdLinux64
Taking a LDAP/NMAS trace reports a -1693 error. See TID 7009590 to resolve this issue.
See TID 7009602 for taking eDirectory LDAP/NMAS traces.
Poorly Configured Network
Poorly Configured Network
When kerberos start it must contact the security container to access the NMAS methods in order to properly load the methods. The IPCEXTERNAL must be active and loaded for kerberos to start.
To verify the IPCEXTERNAL method is running do:
lsof -p `pgrep ndsd` |grep -i ipc
This command should return the following:
ndsd 3452 root mem REG 8,2 149416 571515 /var/opt/novell/eDirectory/data/nmas-methods/IPCLCMLIN_X64.SO
ndsd 3452 root mem REG 8,2 153544 571514 /var/opt/novell/eDirectory/data/nmas-methods/IPCLSMLIN_X64.SO
If IPCLSMLIN is not returned, the IPCEXTERNAL method is not loaded. It could be that the method and sequence has been set to inactive or that the DSfW server is unable to access the security container to update the reference and streamfiles to the needed NMAS methods.
Use iManager to verify the method is active.
Add a replica of root to the DSfW server or if the security container is partitioned, add a replica of the security partition.
Usually adding a replica that containers the security container is not needed unless there is a slow wan link, packet loss, traffic loop, packet congestion, packet collisions, router/switch configuration issues, misconfigured nic, or bad hardward (router/switch/nic)
If kerberos fails to start on reboot and 5 minutes later kerberos starts, the cause is most likely a poorly configured network.
This is usually seen on ADC servers where Spanning Tree is enabled on the router or switch.
When kerberos on the ADC starts, it attempts to resolve the short name (wins) for the domain. Configuring WINS on the PDC and adding wins server = <ip address of wins server> to the ADC can help. See TID 7012934 for more information on configuring WINS.
Taking a packet trace will most likely show a TCP Retransmission or TCP Dup ACK for every other packet. Ping for NDS, Start Update Replica, Resolve Name will all have TCP Retransmission and TCP Dup ACK.
The router or switch will have Spanning Tree enabled. Most vendors have a proprietory method to handle spanning tree issues. Before disabling Spanning Tree, enable the vendors method like Shortest Patch Bridging or Fabric Path. Then disable Spanning Tree. Restarting the server should allow the DSfW domain's short name to be found quickly and kerberos to start.
This is usually seen on ADC servers where Spanning Tree is enabled on the router or switch.
When kerberos on the ADC starts, it attempts to resolve the short name (wins) for the domain. Configuring WINS on the PDC and adding wins server = <ip address of wins server> to the ADC can help. See TID 7012934 for more information on configuring WINS.
Taking a packet trace will most likely show a TCP Retransmission or TCP Dup ACK for every other packet. Ping for NDS, Start Update Replica, Resolve Name will all have TCP Retransmission and TCP Dup ACK.
The router or switch will have Spanning Tree enabled. Most vendors have a proprietory method to handle spanning tree issues. Before disabling Spanning Tree, enable the vendors method like Shortest Patch Bridging or Fabric Path. Then disable Spanning Tree. Restarting the server should allow the DSfW domain's short name to be found quickly and kerberos to start.
Cause
If kerberos fails to start it is usually caused by one of the following reasons:
Missing uniquedomainid attribute on key objects
Missing ldap interfaces on the ldap server object
Corrupt or missing libraries
Mis-configured or missing kdc.conf
Missing uniquedomainid attribute on key objects
Missing ldap interfaces on the ldap server object
Corrupt or missing libraries
Mis-configured or missing kdc.conf