Environment
Sentinel 7.4 Sentinel Server and Above
Situation
A situation where WebUI is inaccessible due to following errors below:The most obvious error I see in the log is the following:
Tue Oct 17 17:00:02 GMT+00:00 2017|WARNING|qtp227445057-602 Selector0|org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:714)
at org.eclipse.jetty.util.thread.QueuedThreadPool.startThread(QueuedThreadPool.java:441)
at org.eclipse.jetty.util.thread.QueuedThreadPool.dispatch(QueuedThreadPool.java:366)
at org.eclipse.jetty.server.nio.SelectChannelConnector$ConnectorSelectorManager.dispatch(SelectChannelConnector.java:300)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.dispatch(SelectChannelEndPoint.java:239)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.schedule(SelectChannelEndPoint.java:198)
at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:745)Thu Oct 12 19:00:06 GMT+00:00 2017|SEVERE|pool-7-thread-1|esecurity.ccs.comp.event.rawdata.RawEventSource.updateHashInDB
Error RAWDATA0001(RawData): Error generating hash for archived raw data file /var/opt/novell/sentinel/data/rawdata/online/682EA4B1-551B-1035-BA14-10604B9C359C/2017-10/12-1800.gz.
Root cause: /var/opt/novell/sentinel/data/rawdata/online/682EA4B1-551B-1035-BA14-10604B9C359C/2017-10/12-1800.gz (Too many open files) (java.io.FileNotFoundException)
com.novell.sentinel.util.logging.error.rawdata.RAWDATA0001: Error generating hash for archived raw data file /var/opt/novell/sentinel/data/rawdata/online/682EA4B1-551B-1035-BA14-10604B9C359C/2017-10/12-1800.gz.
Resolution
Add the following to limits.conf & then reboot. Filename: /etc/security/limits.confnovell soft as unlimited
novell hard nofile 65536
novell soft rss unlimited
novell hard nproc 32000
novell soft nproc 32000
novell soft nofile 32768
Cause
The file /etc/security/limits.conf was inadvertently modified.