Quick Start: checking eDirectory 9 health using the new cn=monitor functionality

In past versions of eDirectory various statistics were collected from multiple sources making it difficult to automate the gathering of health related data.  This included utilities and sources such as ndscheck, iMonitor, LDAP rootDSE searches, ndsrepair, etc.  Often modules that are loaded in the eDirectory address space, such as IDM and OES, have their own way of monitoring internal health statistics.  This is problematic as eDirectory interfaces can not be used for monitoring these modules.  

---

Overview

The eDirectory Monitoring Framework has been added as a feature to eDirectory 9.0.    This is to facilitate the monitoring of modules in the eDirectory address space.  Initially, only eDirectory has been included.  It is hoped this new feature will ultimately help to consolidate all the monitoring data from all modules into one common place and format.  The other advantage is this method of gathering health statistics is very quick, low overhead and lends itself very well to scripting.

Data Producer: the modules to be monitored are called data producers.  The data producer registers with the monitoring framework.  The framework registers and de-registers the data producers for monitoring.  
Consumer: consumers request the monitoring data from the framework.  In turn, the framework obtains all data from the producers real time and provides this data to the consumer.

eDirectory's LDAP server has been enhanced to allow for cn=monitor searches.  The monitoring framework generates and returns dynamic objects to requests.  Some objects are containers for other objects such as the top level object, cn=Monitor.  A subset of the virtual monitor objects is presented below.  Monitor data is available in LDAP object format with the LDAP server being a consumer. 

---

USE

The cn=monitor is a virtual object and is standardized on the OpenLDAP implementation.  Though LDAP clients will be used to return the information the search cannot be filtered.  In order to limit the information returned the output can either be grepped or the base of the seach can be changed.

NOTE: non-admin users running the cn=monitor search must have write rights to the NDSRightsToMonitor attribute on the server's NCP server object. 

Inputs
- cn=monitor will be the base
- scope will be sub tree
- Again,  the bind non-admin user should have write rights to the NDSRightsToMonitor attribute on the NCP server object. m This attribute is not populated by default, therefore, only admin or a supervisor of the NCP server is allowed initially.

---

Some Examples

Base DNs
List the attributes of the virtual cn=monitor object.  This example lists the base DNs of each monitor entry.
 LDAPTLS_CACERT=/var/opt/novell/eDirectory/data/SSCert.pem ldapsearch -H ldaps://192.168.211.51:636 -x -D 'cn=admin,o=emg' -w novell -s sub -b "cn=monitor" "(objectClass=*)" "1.1"

All Statistics available
Dump absolutely everything
 LDAPTLS_CACERT=/var/opt/novell/eDirectory/data/SSCert.pem ldapsearch -H ldaps://192.168.211.51:636 -x -D 'cn=admin,o=emg' -w novell -s sub -b cn=Monitor

LDAP
From the output above we see there are a number of LDAP statistics available.  By selecting the appropriate parent sub DN we can limit the information to just that of the LDAP server.
LDAPTLS_CACERT=/var/opt/novell/eDirectory/data/SSCert.pem ldapsearch -H ldaps://192.168.211.51:636 -x -D 'cn=admin,o=emg' -w novell -s sub -b "cn=LDAPStatistics,cn=LDAP,cn=Monitor"

THREAD POOL
We can also limit the return to only include thread information.  (Grepping would return RECMAN stats as well.)
LDAPTLS_CACERT=/var/opt/novell/eDirectory/data/SSCert.pem ldapsearch -H ldaps://192.168.211.51:636 -x -D 'cn=admin,o=emg' -w novell -s sub -b "cn=cn=CacheStatistics,cn=ThreadPool,cn=DHOST,cn=Monitor"

OBITS, Change Cache, Max Ring Delta
The following will return a wealth of partition health statistics such as the number of obits, their states, the change cache size as well as the max ring deltas.
 LDAPTLS_CACERT=/var/opt/novell/eDirectory/data/SSCert.pem ldapsearch -H ldaps://192.168.211.51:636 -x -D 'cn=admin,o=emg' -w novell -s sub -b "cn=Partition,cn=Agent,cn=Monitor"


For a complete list of Data Producers and Monitoring Data Attributes please refer to Using LDAP for Monitoring  in the eDirectory 9 Admin Guide.  There is also a table at the end of this TID.

The following shows the Producers, Operation Type and the attributes that can be obtained from it.

LDAP - Processing
    - Bindings   
            unAuthBinds – Number of unauthenticated binds
            simpleAuthBinds – Number of simple authenticated binds
            strongAuthBinds – Number of SASL credential binds
            bindSecurityErrors – Number of bind errors for confidential LDAP requests
    - Incoming Operations   
            inOps – Number of requests recieved
            readOps – Number of base searches
            compareOps – Number of compare operations
            addEntryOps – Number of add entry operations
            removeEntryOps – Number of remove entry operations
            modifyEntryOps – Number of modify entry operations
            modifyRDNOps – Number of modify RDN operations
            listOps – Number of list operations
            searchOps – Number of search operations
            oneLevelSearchOps – Number of One level searches
            wholeSubtreeSearchOps – Number of subtree searches
            abandonOps – Number of abandon operations
            extendedOps – Number of extended operations requests
    - Outgoing Operations   
            referralsReturned - Total number of referrals returned for requests
            chainings - Total number of chainings returned for requests
    - Traffic Volume   
            outBytes – Number of bytes written for LDAP operations
            inBytes – Number of bytes read for LDAP operations           

RECMAN - Processing (values are in KB.)
    MaximumSize    - The maximum size that the specified cache is allowed to grow.
        EntryCacheSize
        BlockCacheSize
        TotalSize
    CurrentSize - The current size of the specified cache.
        EntryCacheSize
        BlockCacheSize
        TotalSize
    ItemsCached - The number of items in the specified cache.
        EntryCacheSize
        BlockCacheSize
        TotalSize
    Old Version Cached Count - The number of old versions in the specified cache. Old versions of cache items are kept to maintain the consistency of read transactions in the database.
        EntryCacheSize
        BlockCacheSize
        TotalSize
    Old Version Cached Size - The size (in KB) of the old version items cached
        EntryCacheSize
        BlockCacheSize
        TotalSize
    Hits    - The number of times an item was successfully accessed from the specified cache
        EntryCacheSize
        BlockCacheSize
        TotalSize
    Hit Looks - The number of items looked at in the cache before an item was successfully accessed from the specified cache. The hit-look-to-hit ratio is a measure of cache lookup efficiency. Normally, the ratio should be close to 1:1.   
        EntryCacheSize
        BlockCacheSize
        TotalSize
    Cache Faults - The number of times an item was not found in the specified cache and had to be obtained in a lower level cache or from the disk.   
        EntryCacheSize
        BlockCacheSize
        TotalSize
    Cache Fault Looks - The number of items that could not be found already populated in cache.  The fault-look-tofault ratio is a measure of cache lookup efficiency. Normally, the ratio should be close to 1:1.   
        EntryCacheSize
        BlockCacheSize
        TotalSize
    DIBSize – Total size of record manager on file system

AGENT - monitoring partition processing and attributes
    Partition data producer
        ChangeCacheCount
        LastSucessfulSync
        MaxRingDelta
        ReplicaPerishableDelta
        ObituaryCount
    Background Process producer
        Interval
        State
        Scheduled
        StartTime
        Duration
    Agent status data producer
        TimeSyncStatus
        UpTime
        Version   

SYSTEM data producer - for operating system specific data
        CPUUtilization
        MemoryUtilization   

DHOST data producer - data for inbound connections will be obtained
        Inbound connection data producer - all connections requested to eDirectory server from outside servers.
            MaxInBoundConnection
            InBoundConnectionCount
        Outbound connection data producer - all connections requested from this eDirectory server to other eDirectory servers.
            TotalOutBoundContextCount
            MaxOutBoundConnection
            TotalOpenOutBoundConnection
            RefusedOutBoundConnection
        Threadpool data producer - provides thread pool data statistics
            ThreadsSpawned
            ThreadsDied
            ThreadsIdle
            ThreadsWorkers
            ThreadPeakWorkers
            ThreadPoolReadyQueueItems
            ThreadPoolReadyQueueMaxWaitTime
            ThreadMinWaitTime
            ThreadMaxWaitTime