IDM engine removing attributes from input document after synthetic add

Document ID:7016816
Creation Date:02-Sep-2015
Modified Date:08-May-2018
- Micro Focus Products:
  Identity Manager

Environment

NetIQ Identity Manager (IDM) Engine Functionality

Situation

Some attributes are not listed in the input document after the operation turns into a synthetic add.

[08/31/15 04:41:35.364]:AD Driver ST:
<nds dtdversion="4.0" ndsversion="8.x">
<source>
  <product edition="Advanced" version="4.5.0.0">DirXML</product>
  <contact>NetIQ Corporation</contact>
</source>
<output>
  <instance class-name="User" qualified-src-dn="O=data\OU=users\CN=atest" src-dn="\IDVAULT-TREE\data\users\atest" src-entry-id="33594">
    <attr attr-name="Full Name">
      <value timestamp="1441017695#85" type="string">Firstname Test</value>
    </attr>
    <attr attr-name="Given Name">
      <value timestamp="1441017695#7" type="string">Firstname</value>
    </attr>
    <attr attr-name="Internet EMail Address">
      <value timestamp="1441017695#17" type="string">junk@junk.test</value>
    </attr>
    <attr attr-name="L">
      <value timestamp="1441017695#5" type="string">mycity</value>
    </attr>
    <attr attr-name="nspmDistributionPassword"><!-- content suppressed -->
    </attr>
    <attr attr-name="Surname">
      <value timestamp="1441017695#10" type="string">Test</value>
    </attr>
    <attr attr-name="Title">
      <value timestamp="1441017695#11" type="string">Account Manager</value>
    </attr>
  </instance>
  <status level="success"></status>
</output>
</nds>
[08/31/15 04:41:35.365]:AD Driver ST:Synthetic add:
[08/31/15 04:41:35.365]:AD Driver ST:
<nds dtdversion="4.0" ndsversion="8.x">
<source>
  <product edition="Advanced" version="4.5.0.0">DirXML</product>
  <contact>NetIQ Corporation</contact>
</source>
<input>
  <add cached-time="20150831104135.133Z" class-name="User" event-id="HR_PERSID=271,table=HR_PERS:0fe857e0-a025-4e9c-b2d5-0c6cc0f96973" qualified-src-dn="O=data\OU=users\CN=ajunk" src-dn="\IDVAULT-TREE\data\users\ajunk" src-entry-id="33594" timestamp="1441017695#69">
    <add-attr attr-name="Given Name">
      <value timestamp="1441017695#7" type="string">Firstname</value>
    </add-attr>
    <add-attr attr-name="Internet EMail Address">
      <value timestamp="1441017695#17" type="string">junkmail@junk.test</value>
    </add-attr>
    <add-attr attr-name="L">
      <value timestamp="1441017695#5" type="string">mycity</value>
    </add-attr>
    <add-attr attr-name="nspmDistributionPassword"><!-- content suppressed -->
    </add-attr>
    <add-attr attr-name="Surname">
      <value timestamp="1441017695#10" type="string">Test</value>
    </add-attr>
    <add-attr attr-name="Title">
      <value timestamp="1441017695#11" type="string">Account Manager</value>
    </add-attr>
  </add>
</input>
</nds>

Resolution

This will happen if the time stamp on the attribute is equal to or later than the timestamp of the event.

Earlier in the trace you will see the event timestamp:

[08/31/15 04:41:35.358]:AD Driver ST:
<nds dtdversion="4.0" ndsversion="8.x">
<source>
  <product edition="Advanced" version="4.5.0.0">DirXML</product>
  <contact>NetIQ Corporation</contact>
</source>
<input>
  <modify cached-time="20150831104135.133Z" class-name="User" event-id="HR_PERSID=271,table=HR_PERS:0fe857e0-a025-4e9c-b2d5-0c6cc0f96973" qualified-src-dn="O=data\OU=users\CN=atest" src-dn="\IDVAULT-TREE\data\users\atest" src-entry-id="33594" timestamp="1441017695#69">
    <modify-attr attr-name="nspmDistributionPassword"><!-- content suppressed -->
    </modify-attr>
  </modify>
</input>
</nds>

Then notice the time on the full name 1441017695#85 which is later.

Cause

This can be caused because of misconfiguration such as time drifting problems or future timestamp issues in eDirectory, or it can be caused by a normal delay in event processing.

If events are built up in the TAO (IDM cache) causing older (even a second or two) events coming through after subsequent objects changes have been made to the object, then this is cosmetic only and waiting for the cache to clear will correct the situation. This is the case because subsequent events will already be added to the IDM cache (the TAO file) and will process in order. As those changes are processed as a modify they will also, if the object is not already associated with the application, convert to an add (synthetic add) which will pull in all relevant attributes at that time, now with timestamps in the past relative to the current event. This is normal and expected functionality of an event-driven system.

If the problem is due to a time synchronization issue, correct time on eDirectory servers so that they are consistent.

If not, you can use the following policy at the end of the event transformation policies, though it may mask other problems. If implemented be sure to test all scenarios thoroughly as this will fundamentally impact most operations within the IDM engine.

<?xml version="1.0" encoding="UTF-8"?>
<policy>
  <rule>
    <description>Disable Optimization</description>
    <comment xml:space="preserve">When the event is a modify without a valid association, remove the timestamp XML attribute from the transaction to disable the IDM engine optimization routine.</comment>
    <conditions>
      <and>
        <if-class-name mode="nocase" op="equal">User</if-class-name>
        <if-operation mode="case" op="equal">modify</if-operation>
        <if-association op="not-associated"/>
      </and>
    </conditions>
    <actions>
      <do-strip-xpath expression="@timestamp"/>
    </actions>
  </rule>
</policy>

Additional Information

This has also been discussed in the public forum: https://forums.novell.com/showthread.php?t=507823

To convert form the Unixtime timetsamp used within eDirectory and shown on attribute nodes in XDS, to a format friendly for humans, use the following command from a Linux or Unix command line interface (CLI), usually available as the IDM host; notice the '@' before the Unixtime value which is then provided to the date command:

> date -d @1441017695
Mon Aug 31 04:41:35 MDT 2015