Resolution
In code the assignment rule is a IdentitySelector object and the filter is a CompoundFilter object. What you enter in the UI if you try to make a filter is the XML for a CompoundFilter. The reason this is more complex than profile filters is because there is more than one application in play here, the assignment rule can use combinations of attributes from all application links as well as identity attributes. The following is a collection of information gathered regarding these types of filters.
Here's an example of a filter, it's like XML format for the filters:
<CompoundFilter>
<Applications>
<Reference class="sailpoint.object.Application" id="4028758130badb630130badc231a0211" name="Active_Directory"/>
<Reference class="sailpoint.object.Application" id="4028758130badb630130badc24a40222" name="ERP_Global"/>
<Reference class="sailpoint.object.Application" id="4028758130badb630130badc2435021f" name="Composite_ERP_Global_Platform"/>
</Applications>
<CompositeFilter operation="AND">
<CompositeFilter operation="OR">
<Filter matchMode="ANYWHERE" operation="CONTAINS_ALL" property="1:groupmbr">
<Value>
<List>
<String>PayrollAnalysis</String>
</List>
</Value>
</Filter>
<Filter matchMode="ANYWHERE" operation="CONTAINS_ALL" property="2:groupmbr">
<Value>
<List>
<String>PayrollAnalysis</String>
</List>
</Value>
</Filter>
</CompositeFilter>
<Filter matchMode="ANYWHERE" operation="CONTAINS_ALL" property="0:groupmbr">
<Value>
<List>
<String>InvntryAnalysis</String>
</List>
</Value>
</Filter>
</CompositeFilter>
</CompoundFilter>
In the example above, the index in the property "index:groupmbr" refers to the index of the application listed at the start.
If you're writing these by hand, you may find it marginally easier to use the application name as the property prefix rather than a number. For example instead of this:
<CompoundFilter>
<Applications>
<Reference class="Application" name="Active_Directory"/>
</Applications>
<Filter matchMode="ANYWHERE" operation="CONTAINS_ALL" property="1:groupmbr">
<Value>
<List>
<String>PayrollAnalysis</String>
</List>
</Value>
</Filter>
</CompoundFilter>
You could also write this:
<CompoundFilter>
<Filter matchMode="ANYWHERE" operation="CONTAINS_ALL" property="Active_Directory:groupmbr">
<Value>
<List>
<String>PayrollAnalysis</String>
</List>
</Value>
</Filter>
</CompoundFilter>
It saves a few lines but you will lose tracking of Application renames.
Another thing you can do to make this simpler in some cases is use the "value" property of the Filter rather than the <Value> element.
These two forms are the same:
<CompoundFilter>
<Filter matchMode="ANYWHERE" operation="CONTAINS_ALL" property="Active_Directory:groupmbr">
<Value>
<List>
<String>PayrollAnalysis</String>
</List>
</Value>
</Filter>
</CompoundFilter>
<CompoundFilter>
<Filter matchMode="ANYWHERE" operation="CONTAINS_ALL" property="Active_Directory:groupmbr" value='PayrollAnalysis'/>
</CompoundFilter>
The Value/List/String stuff is only required if you need more than one value, but code that generates filters usually uses this form consistently.