OBM/OMi HAC status flipping over/changing from STARTING to STOPPED state and vice versa

HAC services have been flipping over from STARTED to STOPPED state and vice versa, the state change happens in a matter of once a second and it happens constantly for especially the OPR, BIZ IMPACT, DASHBOARD services but it can happen with any other HAC.

In the hac-*.log of the respective service you will see that it started fine and you don't see when the HAC state changes from running to stopped entries like the actual service is going through an initialization or shutdown process.

The list HAC services from Jboss JMX console shows the HACs as RUNNNING fine even though you see the strange behavior on the OBM status page.

If you switch the services to the secondary OBM they will switch over but the same behavior will be seen.

You might see the HAC pings in negative time like -1m:38s