ndsd is unresponsive or very slow to respond when running Novell Audit

  • 3363936
  • 30-Aug-2006
  • 26-Apr-2012

Environment

Novell eDirectory 8.7.3 for Solaris
Novell eDirectory 8.7.3 for Linux
Novell Audit 2.0.2 Platform Agent

Situation

eDirectory no longer responds to requests
LDAP fails to respond to requests
ndstrace is unresponsive or slow
imonitor is slow
ndsrepair -E or -T is slow to respond
If ndsd is restarted, eDirectory functions normally again for a while.
Problem occurs more frequent under higher load

gcore of ndsd process shows many threads in the following state:
sem_wait@GLIBC_2.0 ()

A look at where for the threads will show:
#0 0xb746a15b in sem_wait@GLIBC_2.0 () from /lib/tls/libpthread.so.0
#1 0xb5a76d00 in NMLogIDs () from /usr/lib//nds-modules/libauditds.so
#2 0xb597fee6 in LogEventExt () from /usr/sbin/../lib/liblogevent.so
#3 0xb5a60f79 in EventHandler () from /usr/lib//nds-modules/libauditds.so

"ndstrace -c threads" typically shows threads are available in the ndsd pool for processing

top does not indicate ndsd is consuming a large amount memory or CPU time.

If Nsure Audit is not loaded, the ndsd does not become unresponsive under heavy load.

Resolution

In this case, the Global option in Novell Audit "Register For Events Inline" was marked. To fix the slowness, you need to unmark (uncheck) the box next to "Register For Events Inline". Then you need to unload and reload the Novell Audit eDirectory instrumentation so the new settings are read.

The documentation indicates this option can cause a slowness in eDirectory.

Please do the following steps to see if you have the eDirectory instrumentation configured in inline mode or not.
1.) Open iManager. Typically the URL is http:///nps/iManager.html
2.) From the list of "Roles and Tasks", select"Directory Administration" and "Modify Object". Browse out to the NCP server object of the server and click "OK".
3.) Once you are in the NCP server object, click on the"Novell Audit" tab. Then click on the "eDirectory" link.
4.) In the "Global" section near the top, there is a setting"Register For Events Inline". By default this is unchecked. If there is a check next to it, uncheck it and save your settings.
5.) Now you will need to unload and reload the instrumentation for the changes to take effect. On Linux and Solaris, you can do the following:
ndstrace -c "unload auditds"
ndstrace -c "load auditds"
Optionally, you can stop ndsd (/etc/init.d/ndsd stop) and start ndsd (/etc/init.d/ndsd start). The eDirectory conf file should contain a section that will stop and start auditds so that's why stopping and starting eDirectory also stops and starts auditds. NOTE: If you stop and start eDirectory on the same server that is running the Secure Logging Server (SLS), then please stop and start the SLS after stopping and starting eDirectory.

Additional Information

When the eDirectory instrumentation is set to inline mode, events are processed serially. eDirectory will not be able to process the next eDirectory event/request until Novell Audit has committed the data to the SLS. If you have a server that is running under heavy eDirectory load, it is not recommended that you register for events inline.