Retain 4 Index Migration Explained

Retain 4 has a new and improved high performance indexer.  Now, using that term causes some understandable confusion.  In Retain 4, there are two indexers: Standard and High Performance Indexer.  

Standard Indexer

Both use the same engine, so technically both are high performance indexers.  The standard indexer is embedded into the Retain Server software on the Retain server.  All of the indexing and searching occurs on the Retain server itself.  For most customer needs, the standard indexer should be sufficient.  This indexer is included with the standard Retain server license.

High Performance Indexer

This indexer has the same technology as the standard indexer; however, rather than being embedded on the Retain server itself, it gets installed on one or more external servers for faster searching performance and high availability.  It allows you to install multiple indexing systems on multiple servers and put them in a "cluster".  Customers that would be good candidates for the HPI have the following business needs:

You have a large legal team, staff, or other sources that would be simultaneously searching for messages in the Retain system.  These users would be pounding the Retain system hard with searches.  If search performance has been less than desireable in the past and if your server hardware is powerful, then you might want to consider the HPI.

eDiscovery is a high priority and you need high availability for the search engine so that searching is always available to users.

The HPI is an additionally licensed component.  It also requires the purchase of professional services to install it as the current installation is complicated and improper configuration can break indexing.  See your territory/sales manager for more details.

Now that we have the indexers explained, let's discuss what migrating to the "high performance index" means...

When you click on that button, Retain immediately activates the new indexing engine.  Since the format of the indexes themselves has changed, Retain must re-index all of your existing messages.  The word "migrate" is really misleading.  It is not technically migrating the old indexes to the new format; however, it is "migrating" your Retain system from using the old indexes to using the new ones, so thus the use of the term "migrate".  This process is simply re-indexing all existing messages and creating new indexes in a separate directory structure beneath the index directory.  The path of the old indexes in ..\index.  The path of the new indexes is ...\index\solrhome\retaincore\data\index.

While all of the messages are being re-indexed, the old Retain 3 indexer will continue to run in tandem with the new Retain 4 indexer; thus, if an archive job runs, each new message will be indexed twice:  once by the old indexer and once by the new indexer.  The indexing status of each message is tracked in the f_indexed field of the t_message table (t_message.f_indexed).  When a message and its attachments are successfully indexed with the old indexer, the f_indexed field gets a value of "1".  If there were problems indexing any of the attachments, there will be negative values in that field (e.g., -64).  On the other hand, the new indexer puts in a value of "2" for any message successfully indexed.  It also has other values it can put in that field, which are positive integers.

Retain 4 knows to have the the old and the new indexers index a message if the f_indexed field shows "0".  If it shows "1" or a negative integer, then it knows that the old indexer has processed the message and its attachments and that the new indexer has not, so it notifies the new indexer of that message so it can be processed.

While the "migration" of indexes is ocurring, the Search tab in the user's mailbox will use the Retain 3 search interface and the old indexes.  The users will see two other search tabs labeled New Search and Advanced Search, but they will not be able to use them.  In fact, when they attempt to go to those tabs, Retain will display a message that reads: 

"Retain 4 has an entirely new search, rewritten from the base up to improve results and experience.  The new search includes a fresh new intuitive interface, suggested results, instant searching, intuitive filters, auto complete fields, and more.  To use the new search Lucene must be upgraded. (After the upgrade the current search is till available as Legacy Search.)"

Once the new indexer has caught up and indexed all existing and new messages (see, "How to Determine When the Retain 4.0 Index Migration is Complete"), Retain will shut off the old indexer, rename the Search tab to Legacy Search and rename New Search to Search.  At that point, the user's searches will be using the new Retain 4 screen.

Now that the old indexer is not being used, the old indexes can be safely deleted.  See, "Removing old indexes after Retain 4.0 has completed re-indexing" for important instructions.  Do not attempt to do this without consulting that KB or contacting Support.