How eDirectory 8.7.3 memory management works in Windows

  • 7012019
  • 25-Mar-2013
  • 25-Mar-2013

Environment

NetIQ eDirectory
Novell eDirectory 8.7 SP3 for Windows 2000
Microsoft Windows 2000 Server

Situation

How eDirectory 8.7.3 memory management works in Windows

Resolution

High level view:

The details of how memory is utilitized by an application such as eDir is dependant on the version of eDir, the type of
process that eDir spawns, the version of eDir, patch level,  OS being used, OS version and its patch level.  The following
applies to any platform with an emphasis on the memory manager being used on thje Windows platform.  Each OS platform
determines the "middle-man" or memory manager used.  Here we assume Win32 and eDirectory 873 IR4.
On Windows eDirectory 873 IR4 uses an open source memory manager called HOARD which allows dhost to be more scalable
relative to SMP machines.  In non-SMP environments the perfromance is about the same as the Windows native manager. 
Since it is open source the source code and all details are available to anyone interested by simply going to their website,
http://www.hoard.org/.  As far as internal MicroSoft memory functions, these can be researched via MicroSoft's public web
site.

From a high level view,  memory is released by eDirectory calling free within C code when a routine within a process no
longer needs it.  Sometimes memory is pooled. The heuristics also depend on the complete dhost environment such as
whether DirXML is involved.   If so, then the JVM is brought into eDirectory's process space as well. While its memory is
also managed via HOARD (albeit unbeknownst to it),   it has modified the pattern of memory usage within the dhost process
significantly. 
 
Theoretically, eDirectory cannot run out of memory because the OS will swap memory in one process for memory in another
if needed. However, as a practical matter, a program can use up lots of RAM - fill its address space - and then never let it go.
When this happens, the system performance degrades due to swap-thrashing.

Questions:

1. I have set a hard limit cache setting for eDirectory of 1GB.  Frequently eDirectory uses more than this amount though it
eventually go back to this setting.  Why is eDirectory using more than the amount I set?

A.   The eDirectory cache is simply a way to hold dib information in memory so the likelyhood of having to go to disk for it
is minimized.  Information can be retrieved from memory faster than from disk.  Think of it as a "virtual disk".   It cannot
do work on its own - it simply holds the information.  The ini file cache settings only affect the eDir cache itself.
When work needs to be done a thread is spawned.   eDirectory, for instance, makes a standard C call to ask the OS for a chunk
of memory for that thread to run in.  As the dib information is needed by a process\thread, we check to see if the
information needed is already in cache.  We either pull it from cache or if it is not there go to disk for it.  Either way in order
to process the returned information we must copy that information  into the separate chuck of memory the thread now
resides in.  This is why eDirectory can be seen to be  taking more memory than the cache setting.  eDirectory has to run its
processes somewhere and that is what is seen when memory goes over this limit. 

2. How does eDirectory handle the de-allocation of OS memory?

A.  As to deallocation, eDirectory or some process running under its name space (ldap for instance) will make a standard C
calls to dealloc memory once the memory is not longer required.  For instance, an ex-ref server needs a list of referrals.  The
backlinker will ask for a 4k chunk initially.  The referral list may be too large to fit into the 4k chunk allocated.  The sending
server of the real object will determine if it is big enough.  If not it returns a error: Insufficient Memory.  eDirectory will
return the 4k chunk then ask for a 16k chunk and so on until there is a buffer large enough  to handle the list to be returned. 
Once the backlinking operation has completed a C call free is called.  This call is caught by the library in Hoard.  Hoard then,
using it's algorythms will return this memory to the OS.  The Windows OS's garbage collection, always running in the
background as a low priority process will then perform the final cleanup moving this memory back into continguous available
memory. 

.

Additional Information

Formerly known as TID# 10097211
Formerly known as TID# NOVL101632