Dibcloning an eDirectory server results in multiple tree keys that are inconsistent

  • 7018175
  • 20-Oct-2016
  • 08-Aug-2018

Environment

NetIQ eDirectory 9.0
NetIQ eDirectory 8.8 SP8

Situation

After a dibclone operation NMAS UP passwords on the target server are not working: Error -669.

Sdidiag reports that the new server has created a new tree key.  Multiple tree keys are not a problem.  However, the new server has neither received the existing tree key nor sent its new key to the master as seen below.
sles11sp2:~ # /opt/novell/sdidiag/bin64/sdidiag64
SDIDIAG> LK -O /root/hv/list.txt

Server : .sles11sp1svr-1.emg.HV9_TREE.
   SDKey : 1
      Object Class : Secret Key
      Key Size : 168 bits
      Key Usage : 0x4400C0
      Key Format : DES-EDE3-CBC-IV8
      Key Id : F9 50 BB B9 CC 27 A0 7B 88 27 AC 9E 8D E1 C5 F7
      Validity : Wed Sep 14 23:45:30 2016  -  Sun Feb  3 23:59:00 2036

      Key Domain: CN=W0.KAP.Security
Server : .sles11sp2.emg.HV9_TREE.
   SDKey : 1
      Object Class : Secret Key
      Key Size : 168 bits
      Key Usage : 0x4400C0
      Key Format : DES-EDE3-CBC-IV8
      Key Id : 77 63 A2 C5 87 26 4E E3 5D CF 04 5B 41 C1 A1 BD
      Validity : Wed Sep 14 21:39:11 2016  -  Sun Feb  3 23:59:00 2036

      Key Domain: CN=W0.KAP.Security

Other times one may see that the master has both the old and new key but the dibcloned server only has the new one it created.

Server : .sles11sp1svr-1.emg.HV9_TREE.
   SDKey : 1
      Object Class : Secret Key
      Key Size : 168 bits
      Key Usage : 0x4400C0
      Key Format : DES-EDE3-CBC-IV8
      Key Id : 77 2C EA CA FC 82 0D E7 D3 21 6F 70 AD A9 11 D3
      Validity : Sun Oct 20 00:08:13 2016 - Sun Feb  3 23:59:00 2036
Server : .sles11sp2.emg.HV9_TREE.
   SDKey : 1
      Object Class : Secret Key
      Key Size : 168 bits
      Key Usage : 0x4400C0
      Key Format : DES-EDE3-CBC-IV8
      Key Id : 77 63 A2 C5 87 26 4E E3 5D CF 04 5B 41 C1 A1 BD
      Validity : Sun Sep 14 21:39:11 2016 - Sun Feb  3 23:59:00 2036
   SDKey : 2
      Object Class : Secret Key
      Key Size : 168 bits
      Key Usage : 0x4400C0
      Key Format : DES-EDE3-CBC-IV8
      Key Id : 77 2C EA CA FC 82 0D E7 D3 21 6F 70 AD A9 11 D3
      Validity : Sun Oct 20 00:08:13 2016 - Sun Feb  3 23:59:00 2036

Resolution

Note that this issue is resolved in eDirectory 8.8 SP8 Patch 11 and eDirectory 9.0.4.  The below information is for older versions.  The key takeaway for current versions of eDirectory is to never copy the nicisdi.key from one server to another.  This file is wrapped with the server's key and cannot be read by any other server.


This affects both eDirectory 8.8SP8 and 9.x though the effects are not seen in a pure 888 tree.  The problem is two fold:

1. The correct ACLs on the W0 object are not getting set for the server that is cloned.  The new server has a copy of the partition that holds the KAP object (normally the root partition).  Therefore, it will be made an additional key server.  It should be getting assigned browse rights to the Entry Rights and read\write to All Attributes.  The dibcloned server is only getting write rights (no read) to All Attributes.  Due to the lack of rights NICIext, when it runs, does not ask the master for its key.  The acls must be manually corrected.

2. The documentation incorrectly lists a step that copies the nicisdi.key file from the master to the dibclone.  This key is wrapped in the master server's unique SKS key.  Therefore, this file can only be read on that particular server.  The only way to give the new server its own tree key file is for NICIext to request the key from the master so it can be wrapped with the cloned server's SKS key and written to a newly created nicisdi.key file.

Cause

Performing a dibclone on eDirectory 8.8 SP8 also results in multiple active tree keys but the keys are consistent.  That is because in 9.0 an IRF is placed on the KAP object.  Server ACLs must be explicitly set on the W0 object itself.  Should the IRF be removed a hourly process will fire and place it back.





Incorrect rights as shown in iMonitor

Additional Information

Instructions
The following steps can be followed after the dibclone has been performed.  This should result in both the master and dibcloned server sharing the master's keys.

1. Using iManager modify the ACLs for the new server so that they are correct.
a. Login to iManager.  Navigate to Directory Administration - Modify Object and select the W0.KAP.Security object.
b. Click on the ACL attribute in the left pane then click Edit.




Above: incorrect rights as seen in iManager.





c. Correct the dibcloned server's ACLs to the following based on the server's role:
SDI Domain Server(s): browse\inherit for the Entry rights and read\write\inherit for All Attributes.
non-SDI Domain Servers: at a minimum: browse\inherit for the Entry rights and read\inherit for All Attributes.
Click Ok - Ok.

In the above example the dibcloned server is given the write rights to All Attributes.  Therefore, this server will be a peer SDI domain server for fault tolerance in tree key creation\distribution.  The write rights must be given, otherwise, a copy of the root partition must be placed on the server for it to perform this role.






2. Ensure the change synchronized to the dibcloned server if this server holds a replica containing the W0 object.  This is best done with iMonitor since one can control from which server it reads.

3. Stop eDirectory on the dibcloned server.
ndsmanage stopall

4. Move or rename the following files from the dibcloned server's filesystem
mv /var/opt/novell/nici/0/nicisdi.key /tmp
mv /var/opt/novell/nici/0/backup/nicisdi.key /tmp

5. Restart NDSD on the dibcloned server.
ndsmanage startall

6. To verify that the keys are synch'd install and run SDIDIAG.  The following command will output the keys for all servers.
SDIDIAG> LK -O /tmp/list16.txt

The output should show both server now only have those keys that were held by the SDI Domain Master.

Server : .sles11sp1svr-1.emg.HV9_TREE.
   SDKey : 1
      Object Class : Secret Key
      Key Size : 168 bits
      Key Usage : 0x4400C0
      Key Format : DES-EDE3-CBC-IV8
      Key Id : 77 63 A2 C5 87 26 4E E3 5D CF 04 5B 41 C1 A1 BD
      Validity : Wed Sep 14 21:39:11 2016  -  Sun Feb  3 23:59:00 2036
      Key Domain: CN=W0.KAP.Security
Server : .sles11sp2.emg.HV9_TREE.
   SDKey : 1
      Object Class : Secret Key
      Key Size : 168 bits
      Key Usage : 0x4400C0
      Key Format : DES-EDE3-CBC-IV8
      Key Id : 77 63 A2 C5 87 26 4E E3 5D CF 04 5B 41 C1 A1 BD
      Validity : Wed Sep 14 21:39:11 2016  -  Sun Feb  3 23:59:00 2036
      Key Domain: CN=W0.KAP.Security




NOTE1: in some cases the original tree key is received by the dibclone but the new key is not synch'd to the master.

Server : .sles11sp1svr-1.emg.HV9_TREE.
   SDKey : 1
      Object Class : Secret Key
      Key Size : 168 bits
      Key Usage : 0x4400C0
      Key Format : DES-EDE3-CBC-IV8
      Key Id : F9 50 BB B9 CC 27 A0 7B 88 27 AC 9E 8D E1 C5 F7
      Validity : Wed Sep 14 23:45:30 2016  -  Sun Feb  3 23:59:00 2036
      Key Domain: CN=W0.KAP.Security
   SDKey : 2
      Object Class : Secret Key
      Key Size : 168 bits
      Key Usage : 0x4400C0
      Key Format : DES-EDE3-CBC-IV8
      Key Id : 77 63 A2 C5 87 26 4E E3 5D CF 04 5B 41 C1 A1 BD
      Validity : Wed Sep 14 21:39:11 2016  -  Sun Feb  3 23:59:00 2036
      Key Domain: CN=W0.KAP.Security
Server : .sles11sp2.emg.HV9_TREE.
   SDKey : 1
      Object Class : Secret Key
      Key Size : 168 bits
      Key Usage : 0x4400C0
      Key Format : DES-EDE3-CBC-IV8
      Key Id : 77 63 A2 C5 87 26 4E E3 5D CF 04 5B 41 C1 A1 BD
      Validity : Wed Sep 14 21:39:11 2016  -  Sun Feb  3 23:59:00 2036
      Key Domain: CN=W0.KAP.Security


  In this case NICIext must be manually started on the dibcloned server.

1. Stop NDSD on the clone.

2. Modify the /etc/opt/novell/eDirectory/conf/ndsmodules.conf file and rem out the NICIext module.
# ndsmodules.conf: NDS Module Description File
# This file describes the modules to be loaded at bootup.  Note that modules
# that need to be loaded would have auto flags set.  Other modules can also
# be present here if a default command line need to be specified. Modules
# will be loaded in the order that's listed here.
#
# Syntax:
# modulename    flags   cmdline
# Each line in this file represents a modulename.  It should not
# contain prefix(lib) or suffix(.so, .la etc.).  We'll look at a
# corresponding .la file to pickup the correct modulefile.
# flags:        should be a comma seperated (no whitespace) list of valid options.
#               auto -> autoloaded when dhost comes up
#               system -> Will not be unloaded.
#               fail   -> Treat as an error and exit if loading fails.
#               noop   -> No flags. MUST for specifying command line without any flags
#

dhlog                   auto,fail               #DHost logger
ncpengine               auto,system,fail        #Core NCP Services
dsloader                auto,system,fail        #Loader
masv                    auto,system,fail        #Modular Authentication Services
nds                     auto,system,fail        #Core DS Services
#niciext                 auto
gams                    auto
snmp                    auto                    #snmp
httpstk                 auto                    #DHost HTTP Stack
hconserv                auto                    #HConServ
nldap                   auto                    #LDAP Server
imon                    auto                    #iMon
embox                   auto                    #eMBox
pkiserver               auto                    #PKI server
ssncp                   auto                    #SecretStore
#xdasauditds            auto                    #xdasauditds

3. Move or rename the following files from the dibcloned server's filesystem
mv /var/opt/novell/nici/0/nicisdi.key /tmp
mv /var/opt/novell/nici/0/backup/nicisdi.key /tmp

4. Start NDSD on the clone

5. Optional: to monitor the key synchronization process a ndstrace screen with the +NICI flag can be started on the clone server.

6. Type the following to load the NICIext module.
ndstrace -c "load NICIext"

A successful synch can be seen in the following ndstrace log:

Found key for the domain CN=W0.KAP.Security, no need to create
StartSync, ccode=0
   Delay time 20 minute(s)
   Delay time is 1200 seconds
 ********************  SYNC LOOP  ****************
HourIndex=2, timeOut=0, doSync=1

StartSync:Starting...
StartSync: Calling GetSDKeyList -- initial sync
Servicing NICISDI conn 11, inSize=13, verb= NSDI_VERB_REQUEST_SD_KEY_INFO_LIST_2
Entering RequestSDKeyInfoList version 2
Found 1 SD keys
Checking rights to W0
 - Entering NSDIGetRightsForConnection
     sles11sp1svr-1.emg Has READ rights on [All Attributes Rights] for CN=W0.KAP.Se
curity -- 2F
Checking rights to W1
 - Entering NSDIGetRightsForConnection
     sles11sp1svr-1.emg Has READ rights on [All Attributes Rights] for CN=W1.KAP.Se
curity -- 2F
KeyDomain:  CN=W0.KAP.Security   key ID: 7763A2C587264EE35DCF045B41C1A1BD   key Typ
e: 3-DES Key
Adding key to the list
Finished determing rights to available keys.  Now assembling list to return
Exiting RequestSDKeyInfoList with 0
Returning from conn 11,outLen=133,ccode=0
Starting NICISDI SyncServers sles11sp1svr-1.emg <- sles11sp2.emg
SyncServers: Calling GetSDKeyList for sles11sp2.emg
Key already exists on this server.  Key ID 7763A2C587264EE35DCF045B41C1A1BD  Type:
3-DES Key
Exiting NICISDI SyncServers    Error: 0
Checking to see if we need to create a new SD key.

Found key for the domain CN=W0.KAP.Security, no need to create
StartSync, ccode=0
   Delay time 20 minute(s)
   Delay time is 1200 seconds
Server: sles11sp1svr-1


Running SDIDIAG now shows only the original tree key and it is consistent between the two servers.

Server : .sles11sp1svr-1.emg.HV9_TREE.
   SDKey : 1
      Object Class : Secret Key
      Key Size : 168 bits
      Key Usage : 0x4400C0
      Key Format : DES-EDE3-CBC-IV8
      Key Id : 77 63 A2 C5 87 26 4E E3 5D CF 04 5B 41 C1 A1 BD
      Validity : Wed Sep 14 21:39:11 2016  -  Sun Feb  3 23:59:00 2036
      Key Domain: CN=W0.KAP.Security
Server : .sles11sp2.emg.HV9_TREE.
   SDKey : 1
      Object Class : Secret Key
      Key Size : 168 bits
      Key Usage : 0x4400C0
      Key Format : DES-EDE3-CBC-IV8
      Key Id : 77 63 A2 C5 87 26 4E E3 5D CF 04 5B 41 C1 A1 BD
      Validity : Wed Sep 14 21:39:11 2016