Operations Connector WebServer (WS) taking to long to process the POST calls

  • KM03600539
  • 17-Feb-2020
  • 17-Feb-2020

Summary

Operations Connector WS is taking up to 10 minutes to process a single POST call.

Error

After a restart of the agent, when doing POST calls to Operations Connector it was taking 15 ms to process each call, after some WS calls the time for each POST increased to almost 600000 ms.

Using streamweaver can see the execution time of the WS Call:

 

HttpResponse(200 OK,List(Cache-Control: no-cache, Date: <Date>, senderid: <ID>, server: BBC 12.06.013; opcgeni 01.00.000),HttpEntity.Strict(application/octetstream,ByteString()),HttpProtocol(HTTP/1.1)) (596494ms)
HttpResponse(200 OK,List(Cache-Control: no-cache, Date: <Date>, senderid: <ID>, server: BBC 12.06.013; opcgeni 01.00.000),HttpEntity.Strict(application/octetstream,ByteString()),HttpProtocol(HTTP/1.1)) (596610ms)

 

Cause

Large environment sending a lot of POST calls, agent with default socket set i.e. 60 and served by 6 threads(1:10), even if  59 connections are closed whole socket set is active in the memory.

Fix

In large environments with more connections, keeping the socket set smaller is more efficient. In such scenarios if the number of threads is 3 and the socket set size is 10(1:3), the requests will be processed faster and results in faster release of the socket set. Due to this reason in large environments where there are many connections and high traffic, keeping the value of SOCKETS_PER_SOCKETSET to 10 and  MAX_SOCKETSET_THREADS to 3 is recommended.

How to configure the variables:

/opt/OV/bin/ovconfchg -ns bbc.http -set MAX_SOCKETSET_THREADS 3

/opt/OV/bin/ovconfchg -ns xpl.net -set SOCKETS_PER_SOCKETSET 10

/opt/OV/bin/ovc -kill

/opt/OV/bin/ovc -start

/opt/OV/bin/ovconfchg -ns xpl.net -set SocketPoll TRUE

 

Variables:

MAX_SOCKETSET_THREADS (6 default) / SOCKETS_PER_SOCKETSET (60 default): To achieve concurrency, ovbbccb keeps all the connections in a pool, called socket_set. Per socket set, there are 60 connections by default and 6 threads serving them by default. On an average 10 connections are served by 1 thread.

MAX_CONNECTIONS :After these many connections, ovbbccb tries to close the unused connections. Say if the total connections reaches > 2000, it will close the unused set of connections from the existing 2000 connections.