What To Do When Retain Runs Out of Disk Space

  • 7020647
  • 03-Feb-2016
  • 07-Aug-2017

Environment


Retain

Situation

I ran out of disk space on my Retain server!

Resolution

Assuming that your server is up enough to delete files so you can open up space on the disk.
Stop tomcat, if it is running.
Go to the log directory and delete the logs. This should open up a bit of space to affect repairs.

Depending on what exactly happened you may have to reload the bootstrap for each worker,  reindex all the messages, repair the database and file system and maybe reinstall Retain. Calling support for help is recommended. As long as you have the ASConfig, archive directory and the database we can usually recover Retain.

Once things are stable now is a good time to consider setting up the Retain server so this is far less likely to happen again. You should read our Best Practices document before you begin.

 So we are going to have to move things around. Retain is very happy to run in a virtual machine (VM) and it makes doing many things easier, especially increasing drive space.

The most important thing to do is to dedicate separate disks for the different components of the Retain server.

going from one disk to many disks

In the worst case you have Retain on the boot drive with the archive and database inside the Retain program directory. This is dangerous because the archive and database directories could be overwritten during an upgrade or reinstall.

We want these components on different disks for two reasons: to increase performance by spreading reads and writes across disks and to protect the data and OS so the entire server does not fail if, for example, the log volume is filled up.

We have knowledgebase (kb) articles on how to move Retain from one server to another

This is going to assume that you are keeping the current server and are able to make changes easily. Please read through the following and the associated kbs for more detailed information to determine what is required for your situation. You are allowed to call support for help.

Important: You will want to make a backup of your Retain server before doing any of this.

At the very start you will have to stop tomcat and you will not be able to use Retain until this is complete.

•The first and most challenging move will be the archive directory as it will have millions of items in, up to, 16.7 million subdirectories. This will take the most time.
  If you are using Linux, we recommend using rsync to move the archive directory
  If you are using Windows, we recommend using robocopy to move the archive directory

•The next most challenging move would be the database. You can ignore this if you have the database installed on a separate database server.
Moving MySQL Database to Another Server

•Then you need to move the indexes. This is less critical as they can be recreated.
How to Move Indexes to a Different Volume

•Then you move the logs and change the log location.
Location of Retain Log Files
Changing the Location of Retain Logs

•Finally you need to update the ASConfig, detailed instructions can be found in the appropriate kb below.
Moving Retain To New Server (Linux to Linux)
Moving Retain To New Server (Linux to Windows)
Moving Retain To New Server (Windows to Linux)
Moving Retain To New Server (Windows to Windows)

When everything is in place start tomcat, log into Retain, browse messages, search messages and run a job to make sure things are running smoothly again.

If some of this doesn't make sense or you run into hurdles you cannot overcome, please feel free to call support for further assistance.

Additional Information

This article was originally published in the GWAVA knowledgebase as article ID 2721.