Creation of Downtime is not working in Operations Bridge Manager (OBM)

  • KM03693954
  • 25-Aug-2020
  • 25-Aug-2020

Summary

Is not possible to create new Downtimes because limit was breached

Error

When trying to create a new downtime in Downtime Manager it is not possible, an error is received:

An Error Occurred

The service encountered an unexpected condition which prevented it from fulfilling the request.

 

In <OMi_Home>/log/jboss/downtime.log:

 

[ajp-/127.0.0.1:8009-11] (DTManagerImpl.java:217) ERROR - downtime fuse was breached

[ajp-/127.0.0.1:8009-2] (DowntimesResource.java:142) ERROR - Downtime validation failed for customer 1: The number of CI downtime definitions exceeds the limit of <number> CIs in downtimes configured for your deployment. This downtime cannot be created.

 

Cause

The limit of downtimes depends on the deployment of the server:

deployment="SMALL" CI limit=600

deployment=" MEDIUM" CI limit=25000

deployment=" LARGE" CI limit=150000

Check OBM UI, Administration > Setup and Maintenance > Infrastructure Settings > DOWNTIME - General Settings

Max. number of downtimes: Defines the maximum number of downtimes for this deployment.

Max. total number of CIs in downtimes: Defines the maximum total number of all CIs in downtimes for this deployment. This number includes both directly defined and impacted CIs.

If any of those is breached, then no more downtimes could be created.

Using JMX Console to get the Downtime Model shows that affected CI is higher than the limit

MBean operation: invoke method on MBean Topaz:service=DowntimeModelService

Invocation successful

Result value:

Number of downtimes 101

Number of downtimes in CMDB TQL result 100

Number of affected CIs in CMDB 150273

That’s why is not possible to create more Downtimes. In this example, the amount of Downtimes is fine but the Affected CIs limit was breached so no more downtimes are allowed.

 

Fix

Access the OBMJMX console

  • Locally from GWS or DPS

http(s)://localhost:29000

  • or remotely from client PC:

http(s)://<GWS or DPS FQDN>:29000

 

NOTE: To access JMX console remotely the following infrastructure setting must be set to false (default value is true):

Infrastructure Settings > Foundation=Security > Restrict remote access to JMX console=false à A restart would be needed to apply the changes.

  • Use Admin credentials to access.

 

Go to:

>Topaz:service=DowntimeManager

>invoke java.lang.String purgeDowntimes

customerID=1

minimumPeriod= <value of your choice in Days>

 

The above would purge the downtimes that have completed before the time period specified in <value of your choice in Days>.

 

NOTE: By default, the minimum period of days is 365 and even if you specify any lower value the method would still use the default value. The default period could be modified as follows:

<OMi Home>/opr/support/opr-support-utils.sh –set_setting –context downtime –set downtime.purging.period <days> -c 1

 

followed by a restart of mercuryAS on all servers:

<OMi Home>/opr/support/opr-support-utils.sh -restart mercuryAS

 

Deactivate and Reactivate BSMDowntime_topology TQL

customerID as 1

patternName as BSMDowntime_topology

Invoke

 

After this, search for activateTql

customerID as 1

patternName as BSMDowntime_topology

Invoke

 

Rsync Downtime Model Service

  • Go to http://localhost:29000
  • Search for Topaz:service=DowntimeModelService
  • Search for java.lang.String resyncModel

customerID as 1

Invoke