Environment
Novell ZENworks Configuration Management 11.2 Business Objects Reporting
Novell ZENworks 10 Configuration Management with Support Pack 3 - 10.3 Database
Novell ZENworks 11 Configuration Management Support Pack 1 Database
Novell ZENworks Reporting Server - ZRS
Situation
Deadlock errors showing on ZRS server caused by reporting loader module
The following error is seen in the loader-messages.log (see Additional Information for full stack trace):
ERROR:
[DEBUG] [10/25/13 1:51:57 PM] [] [Novell.Zenworks.Reporting] [5346] [] [Exceptioncom.novell.zenworks.reports.exception.ReportException: com.novell.zenworks.datamodel.exceptions.LockAcquisitionException: org.hibernate.exception.LockAcquisitionException: error performing bulk update] [] []
The following error is seen in the Oracle alert log:
ERROR:
ORA-00060: Deadlock detected. More info in file path\to\db_ora_7692.trc.
The following errors are seen in the Oracle .trc log:
ERROR:
DEADLOCK DETECTED ( ORA-00060 )
[Transaction Deadlock]
The following deadlock is not an ORACLE error. It is a
deadlock due to user error in the design of an application
or from issuing incorrect ad-hoc SQL. The following
information may aid in determining the deadlock:
...
current SQL:
update zReportFolder set BoeUID=:1 , ReportType=:2 , IsUpdated=:3 where ZUID=:4
The following error is seen in the loader-messages.log (see Additional Information for full stack trace):
ERROR:
[DEBUG] [10/25/13 1:51:57 PM] [] [Novell.Zenworks.Reporting] [5346] [] [Exceptioncom.novell.zenworks.reports.exception.ReportException: com.novell.zenworks.datamodel.exceptions.LockAcquisitionException: org.hibernate.exception.LockAcquisitionException: error performing bulk update] [] []
The following error is seen in the Oracle alert log:
ERROR:
ORA-00060: Deadlock detected. More info in file path\to\db_ora_7692.trc.
The following errors are seen in the Oracle .trc log:
ERROR:
DEADLOCK DETECTED ( ORA-00060 )
[Transaction Deadlock]
The following deadlock is not an ORACLE error. It is a
deadlock due to user error in the design of an application
or from issuing incorrect ad-hoc SQL. The following
information may aid in determining the deadlock:
...
current SQL:
update zReportFolder set BoeUID=:1 , ReportType=:2 , IsUpdated=:3 where ZUID=:4
Resolution
For ZCM 10.x: This is fixed in version 10.3.4 - see KB 7008244 "ZENworks Configuration Management 10.3.4 - update information and list of fixes" which can be found at https://www.novell.com/support
For ZCM 11.x: This is fixed in version 11.1 - see KB 7008746 "ZENworks Configuration Management 11.1 - update information and list of fixes" which can be found at https://www.novell.com/support
For ZCM 11.x: This is fixed in version 11.1 - see KB 7008746 "ZENworks Configuration Management 11.1 - update information and list of fixes" which can be found at https://www.novell.com/support
After applying the fix above, ensure that ZCC/Configuration/Reporting Services/Folder Sync Schedule. Ensure that the setting is no more frequent than 1 day.
This issue has also been seen in 11.2.x versions of ZCM. Any time you apply an update to ZCM, please follow the workaround below.
Workaround:
Linux
-
Goto /etc/opt/novell/zenworks/loader
-
Rename SyncBOE.xml to SyncBOE_xml_bkup
-
Restart novell-zenloader.
Additional Information
This workaround should fix the db high cpu utilization issue.
The only functionality loss is that ZCM super administrator will not be able to assign explicit reporting rights for any ZCM administrator over the reports or folder that is created after loader module has stopped. Report creation and running or scheduling reports will not have any impact.
After applying the workaround, it is still possible to see deadlock messages in the Oracle logs, but they won't be current deadlock messages. Pay careful attention to the date of the deadlock as compared to when the workaround was done. If the workaround was successful, there will be no further deadlock messages in the loader-messages.log on the ZENworks Reporting Server.
Full stack trace from loader-messages.log:
[DEBUG] [10/25/13 1:51:54 PM] [] [Folder Synchronization] [5346] [] [ReportUIds size: 0] [] []
[DEBUG] [10/25/13 1:51:54 PM] [] [Folder Synchronization] [5346] [] [ReportUIds: [[]]] [] []
[DEBUG] [10/25/13 1:51:57 PM] [] [Novell.Zenworks.Reporting] [5346] [] [Exceptioncom.novell.zenworks.reports.exception.ReportException: com.novell.zenworks.datamodel.exceptions.LockAcquisitionException: org.hibernate.exception.LockAcquisitionException: error performing bulk update] [] []
[DEBUG] [10/25/13 1:51:57 PM] [] [Novell.Zenworks.Reporting] [5346] [] [Exception Message:com.novell.zenworks.datamodel.exceptions.LockAcquisitionException: org.hibernate.exception.LockAcquisitionException: error performing bulk update] [] []
[DEBUG] [10/25/13 1:51:57 PM] [] [Novell.Zenworks.Reporting] [5346] [] [Exception cause:com.novell.zenworks.datamodel.exceptions.LockAcquisitionException: org.hibernate.exception.LockAcquisitionException: error performing bulk update] [] []
[DEBUG] [10/25/13 1:51:57 PM] [] [Novell.Zenworks.Reporting] [5346] [] [Exception stacktrace:[[Ljava.lang.StackTraceElement;@40ea011c] [] []
[DEBUG] [10/25/13 1:51:57 PM] [] [Novell.Zenworks.Reporting] [5346] [] [Transaction RolledBack] [] []
The only functionality loss is that ZCM super administrator will not be able to assign explicit reporting rights for any ZCM administrator over the reports or folder that is created after loader module has stopped. Report creation and running or scheduling reports will not have any impact.
After applying the workaround, it is still possible to see deadlock messages in the Oracle logs, but they won't be current deadlock messages. Pay careful attention to the date of the deadlock as compared to when the workaround was done. If the workaround was successful, there will be no further deadlock messages in the loader-messages.log on the ZENworks Reporting Server.
Full stack trace from loader-messages.log:
[DEBUG] [10/25/13 1:51:54 PM] [] [Folder Synchronization] [5346] [] [ReportUIds size: 0] [] []
[DEBUG] [10/25/13 1:51:54 PM] [] [Folder Synchronization] [5346] [] [ReportUIds: [[]]] [] []
[DEBUG] [10/25/13 1:51:57 PM] [] [Novell.Zenworks.Reporting] [5346] [] [Exceptioncom.novell.zenworks.reports.exception.ReportException: com.novell.zenworks.datamodel.exceptions.LockAcquisitionException: org.hibernate.exception.LockAcquisitionException: error performing bulk update] [] []
[DEBUG] [10/25/13 1:51:57 PM] [] [Novell.Zenworks.Reporting] [5346] [] [Exception Message:com.novell.zenworks.datamodel.exceptions.LockAcquisitionException: org.hibernate.exception.LockAcquisitionException: error performing bulk update] [] []
[DEBUG] [10/25/13 1:51:57 PM] [] [Novell.Zenworks.Reporting] [5346] [] [Exception cause:com.novell.zenworks.datamodel.exceptions.LockAcquisitionException: org.hibernate.exception.LockAcquisitionException: error performing bulk update] [] []
[DEBUG] [10/25/13 1:51:57 PM] [] [Novell.Zenworks.Reporting] [5346] [] [Exception stacktrace:[[Ljava.lang.StackTraceElement;@40ea011c] [] []
[DEBUG] [10/25/13 1:51:57 PM] [] [Novell.Zenworks.Reporting] [5346] [] [Transaction RolledBack] [] []