Environment
Retain 4.x
Situation
- The Retain server reports the error: java.io.IOException: No space left on device
13:41:03, 884[Lucene Merge Thread #17306] [ERROR] NIOServerCnxnFactory: Thread Thread[Lucene Merge Thread #17306,5,main] died
org.apache.lucene.index.MergePolicy$MergeException: java.io.IOException: No space left on device
at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:673)
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:641)
Caused by: java.io.IOException: No space left on device
- The Retain Index directory shows what seems to be plenty of disk space remaining. See the results of the 'df' command:
df -Th
Filesystem Type Size Used Avail Use% Mounted on
/dev/sda2 ext3 97G 19G 74G 21% /
udev tmpfs 14G 144K 14G 1% /dev
tmpfs tmpfs 14G 648K 14G 1% /dev/shm
/dev/mapper/retaingrp1-archive1 ext3 7.9T 5.0T 2.6T 67% /data
/dev/sdb1 xfs 500G 200G 301G 40% /data/mysql
/dev/sdj1 xfs 350G 206G 145G 59% /data/mysql/mysql-retain/backup
/dev/sdc1 xfs 1.2T 616G 582G 52% /data/retain/index
Resolution
When Retain runs the Index Optimization routine, it needs more disk space beyond the size of the current index directory contents. Specifically it needs at least double the amount of disk space used; available for use during the index optimization routine.
A better rule of thumb is to allocate 2X of disk space taken in free space. So for example, if the Retain index directory contents is 50 Gigabytes in size, then make sure that the volume that contains the Retain index directory is 150 Gigabytes.
NOTE: If your Retain server was upgraded from Retain 3.x, then you can free up disk space by Removing old indexes after Retain 4.0 has completed re-indexing