Summary
Error
When trying to create a new downtime in Downtime Manager it is not possible, an error is received:
An Error Occurred
The service encountered an unexpected condition which prevented it from fulfilling the request.
In <OMi_Home>/log/jboss/downtime.log:
[ajp-/127.0.0.1:8009-11] (DTManagerImpl.java:217) ERROR - downtime fuse was breached
[ajp-/127.0.0.1:8009-2] (DowntimesResource.java:142) ERROR - Downtime validation failed for customer 1: The number of CI downtime definitions exceeds the limit of <number> CIs in downtimes configured for your deployment. This downtime cannot be created.
Cause
The limit of downtimes depends on the deployment of the server:
deployment="SMALL" CI limit=600
deployment=" MEDIUM" CI limit=25000
deployment=" LARGE" CI limit=150000
Check OBM UI, Administration > Setup and Maintenance > Infrastructure Settings > DOWNTIME - General Settings
Max. number of downtimes: Defines the maximum number of downtimes for this deployment.
Max. total number of CIs in downtimes: Defines the maximum total number of all CIs in downtimes for this deployment. This number includes both directly defined and impacted CIs.
If any of those is breached, then no more downtimes could be created.
Using JMX Console to get the Downtime Model shows that affected CI is higher than the limit
MBean operation: invoke method on MBean Topaz:service=DowntimeModelService
Invocation successful
Result value:
Number of downtimes 101
Number of downtimes in CMDB TQL result 100
Number of affected CIs in CMDB 150273
That’s why is not possible to create more Downtimes. In this example, the amount of Downtimes is fine but the Affected CIs limit was breached so no more downtimes are allowed.
Fix
Access the OBMJMX console
- Locally from GWS or DPS
http(s)://localhost:29000
- or remotely from client PC:
http(s)://<GWS or DPS FQDN>:29000
NOTE: To access JMX console remotely the following infrastructure setting must be set to false (default value is true):
Infrastructure Settings > Foundation=Security > Restrict remote access to JMX console=false à A restart would be needed to apply the changes.
- Use Admin credentials to access.
Go to:
>Topaz:service=DowntimeManager
>invoke java.lang.String purgeDowntimes
customerID=1
minimumPeriod= <value of your choice in Days>
The above would purge the downtimes that have completed before the time period specified in <value of your choice in Days>.
NOTE: By default, the minimum period of days is 365 and even if you specify any lower value the method would still use the default value. The default period could be modified as follows:
<OMi Home>/opr/support/opr-support-utils.sh –set_setting –context downtime –set downtime.purging.period <days> -c 1
followed by a restart of mercuryAS on all servers:
<OMi Home>/opr/support/opr-support-utils.sh -restart mercuryAS
Deactivate and Reactivate BSMDowntime_topology TQL
- Go to https://localhost:8443/jmx-console
- Search for UCMDB:service=TQL Services
- Search for deactivateTql
customerID as 1
patternName as BSMDowntime_topology
Invoke
After this, search for activateTql
customerID as 1
patternName as BSMDowntime_topology
Invoke
Rsync Downtime Model Service
- Go to http://localhost:29000
- Search for Topaz:service=DowntimeModelService
- Search for java.lang.String resyncModel
customerID as 1
Invoke