Quick Start: Advanced tuning parameters for eDirectory 8.8 SP8 and eDirectory 9 explained

  • 7015361
  • 14-Jul-2014
  • 02-May-2018

Environment

NetIQ eDirectory 8.8 SP8 for All Platforms
NetIQ eDirectory 9.0 for All Platforms

Situation

The challenging environments in which eDirectory can be found today have changed over time.  In the past, eDirectory was designed with a bias toward the client.  More priority was given to clients and applications than to its own background processes.
 
Today, however, environments have changed:
  - eDirectory databases are holding an ever greater number of objects and attaining larger sizes.
  - More servers are seen in the replica rings and they tend to be distributed geographically.
  - More dynamic environments: an ever increasing numbers of object modifications, deletions, renames and moves.  Many of these changes are programatic in nature (IE., IDM).
  - Authentications are now measured in logins per second rather than per day.
In a dynamic environment these factors could result in a delay in the processing of object changes in change cache before they are to be sent to other servers.  This results in ever greater overhead on a server while the processes responsible for processing these changes compete for time.  eDirectory 8.8 SP8 includes new background process optimizations that can drastically reduce the time to skulk and process object changes by multiple orders.  Obviously, the degree to which performance increases are actually seen in production will vary between environments. 
 
The good news is that many of these new features are enabled out of the box and require nothing additional from the Administrator.  Many customers have seen an overall 20% improvement doing nothing other than upgrading their servers to 888.  The other optimization features in 888 must be enabled and tuned by the Administrator but it is these optimizations that hold the greatest promise for significant performance improvements.  This TID will describe each tuning parameter as well as provide an example with some starting points.

Every site is different in terms of tuning eDirectory.  One's own internal testing is the only way to find the tuning sweet spots for any given organization.  For more detail on the features discussed here please refer to the eDirectory Admin Guide found on the NetIQ Documentation site: https://www.netiq.com/documentation/edir88/.

Resolution


Features enabled out of the box

 

Obituary Process Optimization

  • Prior to 8.8 SP8 eDirectory had two redundant methods of processing obituaries: backlinks and DRLs (Distributed Reference Links).  The DRL method has been removed eliminating these unnecessary cycles.
  • Additionally, if all servers in a ring are on 888, flag 2 is no longer used in processing delete, rename and move obituaries.  This results in the cycles required to process an obituary being reduced by 50-75%.  Some tests have shown a reduction of up to 230% in the total time to process obituaries. 
  • Another side benefit is there are far fewer "UsedBy" and "Obit_UsedBy" obituary attributes that need to be processed.

Obiturary Process Scheduling Optimization

  • Prior to 888, the obituary process would not run on a parition if the server was currently outbounding changes for that partition to other servers.  This would cause obituaries for that partition to be delayed.  In a busy environment with hundreds of changes per second this could lead to change cache buildup.  Now the obituary process can run in parallel with outbound synchronization thereby reducing obituary processing delays.

Priority Queue

  • Synchronization, especially by server mode, could result in only a few servers in a given replica ring actively skulking to each other at any given time.  The previous way eDirectory stored the ordering of the servers it needs to schedule for outbound synchronization was not a priority queue.  Now the scheduling of synchronization is a priority queue that implements a FIFO ordering so that servers just contacted go to the end of the queue.  Servers with the oldest scheduled time go to the front of the list.  Additionally, servers who returned a "-698 - Server Busy", when contacted, are now rescheduled to the front of the queue.
               


Features not enabled by default

 

Asynchronous Outbound Synchronization

  • In the past there was one thread responsible for interating through all entries in the change cache to see if there was a new value that needed to be sent to other servers.  This same thread was also responsible for putting the values in a packet, sending the packet over the wire and waiting on an achnowledgement from the receiving server before proceeding.  Performing these tasks sequentially were costly in terms of time required to process the changes out to other servers.  In eDirectory 8.8 SP8 the work has been split between two threads.  One examines the change cache, prepares the outgoing packets then fills a queue with the packets.  The second thread picks up the packets from the queue and sends them the remote server one at a time.  This has reduced the time to get changes out to other servers by up to 50% in some cases.
  • The default setting for Asynchronous Outbound Synchronization is disabled.  In eDirectory 8.8 SP8 iMonitor provides an interface to enable this feature.  If this feature is enabled the server will be much more aggressive in sending changes out to the remote servers.  This can put pressure on the receiving servers resulting in higher utilization.  Therefore, another setting has been included with this feature, the Async Dispatcher Thread Delay.  This setting allows the administrator to control, in milliseconds, the frequency in which changes are sent to the remote servers.  The Async Dispatcher Thread Delay setting, by default, has no delay (0 milliseconds).  The allowed range for this setting is 0 - 999 milliseconds.  If this value is very small, meaning the thread fires more often, higher CPU and I/O utilization may be seen on the receiving servers due to the higher amount of inbound traffic.  Therefore, this setting should be monitored and fine tuned for the specific enviroment.

 NOTE: In eDirectory 9 Asynchronous Outbound Synchronization is turned on by default.  It is using the Background Process Delay with a Maximum CPU Utilization  of 80%.


Example of use
Environment: all servers are upgraded to 8.8 SP8.
Goal: Synchronize changes out to other servers more quickly.  It is recommended for most customers that this setting be turned on for all servers.  By enabling this feature eDirectory will more aggressively process entries in change cache as well as sending changes to other servers in the replica ring.  In most environments a setting of 0 for the Async Dispatcher Thread Delay is recommended.  However, this could put stress on the receiving servers.  They should be monitored for CPU utilization and this delay can be adjusted if needed.  The next section, Background Process Delay Settings, is also used to inject delays if unacceptably high utilization is seen on the outbounding server.
 
Steps:
- Open a browser and connect to the server's iMonitor interface: https://x.x.x.x.:8030/nds
- Navigate to Agent Configuration - Background Process Settings - Asynchronous Outbound Synchronization Settings
- Click on the radio button to enable the feature, accept the default of 0 milliseconds for the delay then click on Submit.
 
 
 
 
 
 
 

Background Process Delay Settings
  • Background process scalability improvements were of primary importance during the development of eDirectory 8.8 SP8.  As mentioned earlier, the emphasis in the past was to make background processing a lower priority than that of applications and clients.  During 8.8 SP8's development an analysis of code and extensive testing revealed that many background processes were spending a significant percentage of time sleeping.  Dramatic gains in performance can be had in forcing these processes to fire more frequently.  These changes need to be carefully considered, especially on older servers, as an increased frequency can result in an increase of the utilization of the hardware.
  • Currently the following three processes have a hard coded delay of 100ms delay time.
    • Change Cache Processing Delay
    • ObitProc Delay
    • Purger Delay
  • Better performance can be had by lowering the delay times so the processes spend less time sleeping.  This can be done via a Hard Limit or CPU Based Policy.  Lowering these values can improve the following:
    • obituary processing speed
    • purging of objects in change cache
    • faster analysis of objects in change cache resulting in increased synchronization speed.
  • Policies
    • Hard Limit Policy - default
      • This allows the administrator to define a fixed value, in milliseconds, for how long each process will be delayed.  However, this approach means the administrator must manually find the right settings. 
    • CPU Based Dynamic Policy
      • This allows the system to either step up or down the delay based on current overall CPU usage.  The default, when enabled, is an 80% limit of CPU with a 100ms sleep time.  The server will check this utilization every 100 milliseconds.  Should the load on the server become higher than the configured maximum the system will begin to the increase the sleep time by 5 millisecond increments until utilization falls below the maximum. 
      • A test was performed using the CPU policy with a max CPU of 40% and a max Delay of 0.  This resulted in an 80% improvement when processing a bulk import of one million objects.
 
Example of use
 
Environment: all servers are upgraded to 8.8 SP8.
Goal: Speed up replication, the processing of obituaries and reduce the number of objects in change cache waiting to be processed and purged.
Steps: For most environments it is recommended to use the CPU-based Background Process Delay Setting rather than the Hard Limit.  This allows the server itself to find the sweet spot rather than the administrator having to manually try multiple values in order to find the right balance. 
 
- Open a browser and connect to the server's iMonitor interface: https://x.x.x.x.:8030/nds
- Navigate to Agent Configuration - Background Process Settings - Asynchronous Outbound Synchronization Settings
- In the Background Process Delay Settings section click on the radio button next to select the CPU-based method, accept the defaults of 80% for the Maximum CPU Utilization and 80ms for the Maximum Delay Limit. 
- Click on Submit.
 
 
 
 
 
 
Configuring the Mode and Threads used in Outbound Synchronization
  • eDirectory, by default, uses a dynamic (system computed) mode to determine both the mode of synchronization as well as the number of threads to be used in that mode.  Performance can be enhanced by both manually selecting the mode and setting the maximum number of threads to be used.  This can increase the number of other servers or partitions that a server can synchronize to simultaneously beyond what would have been auto calculated.
  • Setting the Mode of Synchronization
    • Modes
      • By Partition.  This is the most conservative as it uses one thread per partition to update multiple servers.  By default, if this mode is selected the number of threads will equal the number of partitions on the server up to a maximum of 8 threads.  If the System Computed  Synchronization Threads is disabled and the Maximum Manual Synchronization Threads is set to 12 the maximum number of threads used can go up to 12.  Changes are then synchronized simultaneously with other replica holding servers. 
        • If there are more unique partitions than there are unique servers in that server's replica rings this mode is the most efficient if there are dynamic changes to objects in multiple partitions. 
        • For example, if there are 12 partitions but only three servers holding replicas and all 12 partitions contain object changes, then 12 threads can be scheduled to send these changes out for all 12 partitions at once.
      • By Server.  This mode is chosen if the number of unique servers in the server's replica rings is greater than the number of partitions on that server.
        • Maximum number of threads used = [(number of unique servers +1) /2]
        • This mode might be chosen in the following example:  There are two partitions in the tree and 9 servers holding replicas.  Therefore the calculation used would be (9 unique servers +1) /2 = 5 threads.    If partition mode was choosen the server would only schedule two threads to outbound to the other servers. 
      • Dynamic mode.  This is the default.  The system calculates the mode and threads available for outbound synchronization.
        • Dynamic is enabled by default.  This allows the system to choose which mode and the number of threads to use on startup.
  • Setting the number of Synchronization Threads
    • Below are the new thread options:
      • Option A - System Computed Synchronization Threads  This is enabled by default.  Disabling this setting prevents the system from computing its own calculations of how many threads to use.  Once disabled, the administrator can set the amount of outbound threads using Option C below.
      • Option B - Maximum System Computed Synchronization Threads This number represent the maximum number of threads that can be used when the server is in dynamic mode.  By default the maximum threads dynamic mode can use is 8 threads.  This new option is only used with Option A enabled as well and allows the administrator to manually configure up to a maximum of 12 threads. 
      • Option C - Maximum Manual Setting Synchronization threads  This number is the maximum number of threads that can be used in either mode.  This option overrides Option A and allows the administrator to set the number of outbound threads up to the maximum value.  By manually selecting the mode and configuring the number of threads for outbound sync, administrators gain control over the number of outbound threads potentially used in outbound sync.  Therefore, using the example of 9 unique servers and 2 partitions in dynamic mode the server would choose server mode and calculate the number of threads to use in outbound synchronization to only 5.  However, by setting the following in iMonitor we can increase the number of threads so that all 9 servers can potentially receive synchronization changes at once.
        • Set the Synchronization Method to Server
        • Disable the System Computed Synchronization Threads
        • Set the Maximum Manual Setting Synchronization Threads to 9
          • NOTE: The Max. System Computed Synchronization Threads field and the Max. Manual Setting Synchronization Threads field are mutually exclusive.

 

 
 
Example 1
Environment: 6 servers - 12 partitions.  All servers hold all partitions.  This is a dynmaic environment where frequent object changes are occuring in all partitions.
Goal: Configure the most efficient synchronization mode and determine the optimal number of threads used for outbounding object changes.  Ideally, this will result in object changes being sent to as many servers as possible simultaneously.
  • Method: Partition Mode has been selected.  We have only 6 servers but 12 busy partitions.   The maximum number of threads that can be used for outbound synchronization is 16.  If partition mode is selected and the number of threads  is set to 12 this server can simlutanously outbound changes in all 12 partitions to remote servers.
  • System Computed Synchronization Threads: leave at enabled
  • Maximum System Computed Synchronization Threads: 12
Steps:
- Open a browser and connect to the server's iMonitor interface: https://x.x.x.x.:8030/nds
- Navigate to the Agent Configuration - Agent Synchronization page
- For the Synchronization Method click on By Partition.
- Continue to use System Computed Sychronization Threads
- Set the Maximum System Computed Synchronization Threads to 12
- Click on Submit.
 
 
  
 
 
Example 2
Environment: 6 servers and  2 partitions.  All servers hold both partititions and are upgraded to 8.8 SP8.  Modifications occur across all 6 servers.
Goal: In this case there are more servers than partitions.  Therefore the threads will be set to server mode with 6 threads selected.  In this configuration the server will assign each of the threads to a particular server so that all six threads are outbounding changes from the 2 partitions.
  • Method: Server Mode has been selected. 
  • System Computed Synchronization Threads: disabled.  In server mode, if this option was not disabled, the server would have calculated 4 threads to be used [(number of unique servers +1) /2]  Set the Maximum Manual
  • Maximum Manual Synchronization Threads: 6
Steps:
- Open a browser and connect to the server's iMonitor interface: https://x.x.x.x.:8030/nds
- Navigate to the Agent Configuration - Agent Synchronization page
- For the Synchronization Method click on By Server.
- Disable the System Computed Synchronization Threads
- Set the Maximum Manual Setting Synchronization Threads to 6.
- Click on Submit.
 
 
 
 
 

Policy Based Replication
 
eDirectory, by default, uses a mesh topology wherein each server must be able to contact all others in the tree.  In some environments, for example where modifications only occur on one server, the Administrator can enable this feature and create a policy that defines which servers a single server can outbound to.  This is a very advanced setting and should be used with caution.  The Administrator must have deep knowledge about the environment and connected systems.  This feature will not be covered in this TID but more information can be found in the eDirectory 8.8 SP8 Admin Guide, section 4.1.4.
 
 
 

Configuring the Purger Interval
  • The purger process is responsible for removing entries from the change cache once all servers have seen that object's changes.  The default is to run for 15 minutes followed by a 30 minute delay.  In earlier versions of eDirectory the purger would wait on the skulker to complete.  In sites that have continuous amounts of changes it could be hours before it fired.
  • In eDirectory 8.8 SP8 the purger is allowed to run alongside the skulker.  Further, the delay setting is now configurable.  This can be set lower so that the purger process runs more frequently and objects are removed from change cache more quickly once they are seen by all replica bearing servers. 10 minutes is the recommended minimum so that it does not run excessively.
Environment: This customer's environment is very dynamic.  Many programmatic modifications are occuring in the tree due to IDM drivers and a large number of user logins.  These changes are quickly getting out to other servers due to the previous tuning changes described above.  However, change cache is still growing excessively due to the number of changes between purger cycles. 
Goal:  To better maintain the backlog of entries in change cache whose changes have been seen by all servers.  In order to give the purger process more opportunity to remove these "have seen" entries from change cache the purger will be set to run more often, every 10 minutes.  If this proves too low (IE., it is running all the time) this will be adjusted up to 20 minutes.  Running more frequently should result in fewer objects in change cache when it does run. 
Steps:
- Open a browser and connect to the server's iMonitor interface: https://x.x.x.x.:8030/nds
- Navigate to the Agent Configuration - Background Process Settings page - Background Process Interval section - Purger Interval
- Enter the number of minutes the Purger Process will wait before running.  The 30 minute default works for most environments but for this use case it will be set to 10.  (Again, as this process kicks off other processes as well, it is not recommended to go below 10 minutes for this setting.)
- Click on Submit.
 
 
 
 
 

 Login Update Disable Interval
  • By default several attributes are updated on a user object each time it is used to login.  Among them are:
    • Login Time
    • Last Login Time
    • Network Address
    • Revision
    • modifiersName
  • These attribute changes must be synchronized to all other servers holding a replica for that user object.  In a busy environment this can add dramatically to the amount of data required to be synchronized between servers.  Previously these changes could only be disabled or enabled.  Some customers use these values to track logins so disabling them altogether was not an option for them. 
  • In eDirectory 8.8 SP8 a new setting has been implemented that allows an administrator to set a time interval between logins wherein these changes are not recorded on the object.  The default is 0, meaning this option has not been enabled.  By entering a value other than 0 it is enabled.  When enabling this setting the typical interval set is 3600 seconds (1 hour).  For example, when a user logs in for the first time at 8:00 AM, eDirectory updates these login attributes and the interval starts. If the same user logs in again before 9:00 AM, eDirectory will not update these attributes' values.
Environment: This environment is a heavy user of LDAP based applications.  The LDAP connections made by these applications are numerous but short-lived.  Login Update attributes are enabled both for tracking IP addresses from which users logged in as well as their last login time.  As these users are getting updated multiple times per minute or hour, change cache is growing exponentially and synchronization is near non-stop.
Goal:  Reduce the number of entries in change cache and minimize the number of changes that need to be skulked out to other servers.  Since the workstation IP addresses and last login times are tracked it has been decided to enable the Login Update Disable Interval functionality.  As users are not logging in multiple times per hour from different workstations it has been decided to only update the login attributes once per hour.  This will still allow the relevant information to be tracked but also reduce unwarranted synchronization traffic and change cache bloat. 
Steps:
- Open a browser and connect to the server's iMonitor interface: https://x.x.x.x.:8030/nds
- Navigate to the Agent Configuration - Login Settings page - Login Update Disable Interval field
- Enter the number of seconds between logins during which additional logins will not update a user's login attributes.
- Click on Submit.