java.lang.OutOfMemoryError: unable to create new native thread

  • 7022158
  • 24-Oct-2017
  • 12-Mar-2018

Environment

Sentinel 7.4 Sentinel Server and Above

Situation

A situation where WebUI is inaccessible due to following errors below:
The most obvious error I see in the log is the following:

Tue Oct 17 17:00:02 GMT+00:00 2017|WARNING|qtp227445057-602 Selector0|org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect

java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:714)
at org.eclipse.jetty.util.thread.QueuedThreadPool.startThread(QueuedThreadPool.java:441)
at org.eclipse.jetty.util.thread.QueuedThreadPool.dispatch(QueuedThreadPool.java:366)
at org.eclipse.jetty.server.nio.SelectChannelConnector$ConnectorSelectorManager.dispatch(SelectChannelConnector.java:300)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.dispatch(SelectChannelEndPoint.java:239)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.schedule(SelectChannelEndPoint.java:198)
at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:745)
 
 Thu Oct 12 19:00:06 GMT+00:00 2017|SEVERE|pool-7-thread-1|esecurity.ccs.comp.event.rawdata.RawEventSource.updateHashInDB
 Error RAWDATA0001(RawData): Error generating hash for archived raw data file /var/opt/novell/sentinel/data/rawdata/online/682EA4B1-551B-1035-BA14-10604B9C359C/2017-10/12-1800.gz.
 Root cause: /var/opt/novell/sentinel/data/rawdata/online/682EA4B1-551B-1035-BA14-10604B9C359C/2017-10/12-1800.gz (Too many open files) (java.io.FileNotFoundException)
 com.novell.sentinel.util.logging.error.rawdata.RAWDATA0001: Error generating hash for archived raw data file /var/opt/novell/sentinel/data/rawdata/online/682EA4B1-551B-1035-BA14-10604B9C359C/2017-10/12-1800.gz.
 
 

Resolution

Add the following to limits.conf  & then reboot. Filename: /etc/security/limits.conf
novell  soft    as      unlimited
novell  hard    nofile  65536
novell  soft    rss     unlimited
novell  hard    nproc   32000
novell  soft    nproc   32000
novell  soft    nofile  32768

Cause

The file /etc/security/limits.conf  was inadvertently modified.