Environment
Retain (all versions)
Linux
Situation
The Retain server has stopped archiving users' e-mail and the Retain Server is sending error messages to the Retain admin every 10 minutes.
The messages looked like this:
EXCEPTION : com.gwava.datastore.exceptions.DataStoreException: java.io.IOException: No such file or directory com.gwava.datastore.exceptions.DataStoreException: java.io.IOException: No such file or directory at com.gwava.datastore.impl.DataStoreImpl.add(DataStoreImpl.java:128) at com.gwava.datastore.client.SingleProcessStorageEngine.archiveEmail(SingleProcessStorageEngine.java:83) at com.gwava.archive.StoreEmail.handlePUT(StoreEmail.java:324) at org.apache.jsp.Server.index_jsp._jspService(index_jsp.java:150) at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70) at javax.servlet.http.HttpServlet.service(HttpServlet.java:722) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:432) at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:390) at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:334) at javax.servlet.http.HttpServlet.service(HttpServlet.java:722) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at com.gwava.Filters.ResponseFilter.doFilter(ResponseFilter.java:48) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at com.gwava.Filters.MainFilter.doFilter(MainFilter.java:301) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:225) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:927) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407) at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:200) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:585) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: com.gwava.datastore.exceptions.BlobStoreException: java.io.IOException: No such file or directory at com.gwava.datastore.blob.LargeBlobStore.store(LargeBlobStore.java:98) at com.gwava.datastore.impl.DataStoreImpl.add(DataStoreImpl.java:109) ... 30 more Caused by: com.gwava.exceptions.FatalIOException: java.io.IOException: No such file or directory at com.gwava.engine.standard.ArchiveFile.addDocument(ArchiveFile.java:274) at com.gwava.datastore.blob.LargeBlobStore.store(LargeBlobStore.java:95) ... 31 more Caused by: java.io.IOException: No such file or directory at java.io.UnixFileSystem.createFileExclusively(Native Method) at java.io.File.checkAndCreate(File.java:1705) at java.io.File.createTempFile0(File.java:1726) at java.io.File.createTempFile(File.java:1803) at com.gwava.engine.standard.ArchiveFile.addDocument(ArchiveFile.java:205) ... 32 more
Troubleshooting Methodology
The Retain server is on a Suse Linux Enterprise server. We went to the Retain server and viewed the current "Retain Server Log" in the following manner:
a.) Change to the directory /var/log/retain-tomcat7
b.) View the current Retain-Server-<date>.log file. For example: RetainServer.2014-07-28.log
c.) We used the less utility in the following manner:
less RetainServer.2014-07-28.log
d.) Once we were viewing the Retain Server Log, we used the search command, which is a forward slash: /
e.) We searched for the word: exception <Enter key>
f.) The less command does not require a case specific search, so the search found the exception that helped us to find the error.
g.) Once we found the word "Exception" we did a "Page Up" and found the following error:
ERROR com.gwava.utils.Files - Warning: /var/opt/beginfinite/retain/archive/63/73/43 couldn't be created.
h.) This indicates a rights issue. Specifically Tomcat, which hosts Retain, does not have sufficient rights at the path: /var/opt/beginfinite/retain/archive in order to create directories.
Resolution
Grant ownership to Tomcat in the following manner:
1. Go to the directory: /var/opt/beginfinite/retain/archive
2. Execute this command: chown -R tomcat:tomcat *
NOTE: The above-mentioned command could run for many hours.