How Retention Services and Item Store Flags Work

  • 7018888
  • 08-May-2017
  • 23-May-2017

Environment

Retain 3.x and Retain 4.x

Situation

I have a GroupWise system and some users are unable to empty their Trash.  Why is this?  Also, if I do not use GroupWise but use another email system instead (Exchange/O365/Gmail), how does Retain ensure that all items get archived?

Resolution

Exchange, O365, and Gmail systems all rely on Retain's "item store flag" to ensure that no item gets left behind.   Exchange and O365 rely on a journaling mailbox or the Recoverable Items folder for retention compliance.

GroupWise, on the other hand, has its own built-in feature called "Retention Services" that prevents items from being emptied from the mailbox until they have been successfully archived.

This article will first discuss Retain's support of the GroupWise Retention Services feature followed by a discussion of how Retain ensures that all items get archived in all other email systems.


GroupWise Retention Services

GroupWise has a feature that can be enabled in its ConsoleOne GroupWise Administration snap-in module called Retention Services.  It is accessed in GroupWise 8 through 2014R2 by highlighting the GroupWise system, domain, post office, or user object and selecting Tools | GroupWise System Operations | Retention Services

When enabled, GroupWise will prevent a user from emptying an item from Trash that has not yet been confirmed to have been archived.  The way it does this is through a date/time field in each user database called the "digest retention time".  It relies on third party archiving solutions like Retain to set that date/time, but GroupWise is the one that enforces it when set.  What this does is it prevents any item newer than the date/time set in the "digest retention time" field from being emptied from Trash.  This "digest retention time" is known in Retain as the "retention flag".

When Retain runs an archive job on a mailbox, it sets the digest retention time to the date/time of the newest/latest message it archived.  However, if an error occurs on any item during that job which prevents Retain from archiving it or its attachment, Retain will set the digest retention time in the GroupWise user database for that mailbox to the date/time of the item that could not be archived due to an error.

And, even though Retain encounters an error on an item and cannot archive it, it moves beyond that item and continues to archive all other mailbox items; however, again, it will not advance the retention flag past the date/time of the FIRST error it encountered.

Thus, when the next archive job gets run on that mailbox, Retain checks the digest retention time set in the user database and uses that date/time as its starting time for the new job.

Example

If today is September 17, 2014 but an item in the previous job produced an error, could not be archived because of that error, and had a delivered date/time of September 15, 2014 09:15, then when today's job runs, it will ask GroupWise for all items it has beginning with September 15, 2014 09:15 and on. 

Now let's say that a month has passed and the problematic mail message has not been properly dealt with and we run a job.  Even though Retain may have archived all items in the user's mailbox up to - let's say October 15th - it will still start the query with the digest retention time of September 15, 2014 09:15 because it could not advance the retention flag.  If it were to do so, then they problem message would never get archived because Retain starts the query for items beginning with the digest retention time.  Thus, if Retain were to advance the flag to the date/time of the newest/latest item it archived, then the problematic message would fail to fit within the query range and GroupWise would never send it to Retain.


So, when users complain they cannot empty trash, this typically has a few possible causes:

  • They have not exited the GroupWise client from the day before (or since a Retain job ran).  The client checks the digest retention time upon loading but never checks it after that.  Solution?  Have the user close GroupWise and make sure that Notify also closes; otherwise, Notify keeps the current session open and relaunching the client is the same as not having exited the client in the first place; thus, it does not check the digest retention time because it is entering an already existing session.  When the client launches and starts a new session, it checks the digest retention time and stores that for the session.
  • An archive job has not been run for a few days.  Login to the RetainServer web UI and check the job status to see if it has run lately.  If not, determine the cause (job disabled? license expired? Worker lost connection with Server?  Out of disk space on Retain server? etc) and run the job.
  • The user that cannot empty trash had errors on an item or items in his/her mailbox during the archive job.  Login to the RetainServer web UI.  That should bring up the Status & Updates screen.  Click on Archiving Stats and run the Summary report for the last 24 hours.  Find the user in the report and you'll likely see a number in the "Errors" column.  Click on that hyperlinked number to see what errors occurred.  Search the GWAVA Retain knowledgebase for solutions on resolving that error or contact technical support.
  • Corruption/issues in the GroupWise user database for that mailbox (least likely of all scenarios).  If this is the case, run GWCheck and rebuild the user database or contact Novell GroupWise support for help.

Exchange/O365/Gmail

T
hese email systems do not have a built-in retention service similar to GroupWise, there is no "digest retention time" field in any of their mail system databases that Retain can use; thus, Retain uses its own field in the "retain" database to keep track of its job starting point.

This date/time field is in the t_abook table of the "retain" database.  The field name is ts_item and it stored for every single Retain mailbox in the system.  The date/time is stored in Unix time and that number can be translated using any online Unix timestamp conversion utility.

This "item store flag" works just like the "retention flag" with GroupWise jobs.  That date/time gets set to the date/time of the newest/latest item archived for a given mailbox; or, if an error(s) occurred during a job, the item store flag gets set to the date/time of the first item that had an error.  That way, when the next archive job runs, it starts with the date/time of the item store flag, ensuring that Retain will see that item until it is properly archived.

However, it is important to note that not advancing the item store flag does not prevent the user from emptying the item from their Trash in these email systems because they do not have a retention feature similar to GroupWise.  Instead, Exchange/O365 give the customer the choice between setting up a journaling mailbox or using the Recoverable Items folder.

When a journaling mailbox is set up in Exchange, it can be configured in a way that redirects a copy of each message that is either sent or received throughout the entire mail system into they journaling mailbox.  Retain can be configured to include the journaling mailbox in its archive job.  Thus, even if a user empties an item from Trash, a copy of that item already exists in the journaling mailbox and remains in that mailbox until it is archived by Retain.  If configured properly, Retain will remove that item from the journaling mailbox upon successfully archiving it.

Items emptied from a user's Exchange mailbox but archived from the journaling mailbox do not appear in the user's Retain mailbox; however, they are searchable using the Retain search feature.

Another option is to use the Recoverable Items folder.  This is a hidden system folder in Exchange that exists for each mailbox; however, the users do not see in in their mailboxes in Outlook.  When a user empties an item from the Trash, it gets placed in the hidden Recoverable Items folder and remains there by default for two weeks (this is configurable).  Retain supports archiving the Recoverable Items folder - it is a checkbox in the profile that gets associated with the archive job.

Unlike the journaling mailbox, the Recoverable Items folder does appear and is accessible to the user in his/her Retain mailbox.

Because of the fact that duplicates of all email messages system wide get placed in the journaling mailbox, it can fill up fast.  For this reason, we recommend that you not use the journaling mailbox feature and go with the Recoverable Items feature instead.  If the journaling mailbox gets too big, Exchange will no longer be able to use it.  Thus, when Retain tries to run an archive job against it, it will fail because Exchange never responds back.

In the case of Gmail, it does not have retention services.  It requires the purchase of a premium service.