HTTP 500 Error after updating to Filr 3.4

  • 7023200
  • 24-Jul-2018
  • 26-Jul-2018

Environment

Micro Focus Filr 3.x

Situation

After applying Filr updates to the Database, Search, and Filr servers, attempting to load the :8443 (or the redirected port, in most cases :443) Filr web client results in an HTTP 500 Error. 
HTTP Status 500

type Exception report

message

description The server encountered an internal error that prevented it from fulfilling this request.

exception

java.lang.NullPointerException
    org.hibernate.cache.impl.bridge.RegionFactoryCacheProviderBridge.nextTimestamp(RegionFactoryCacheProviderBridge.java:93)
    org.hibernate.impl.SessionFactoryImpl.openSession(SessionFactoryImpl.java:637)
    org.hibernate.impl.SessionFactoryImpl.openSession(SessionFactoryImpl.java:646)
    org.springframework.orm.hibernate3.SessionFactoryUtils.doGetSession(SessionFactoryUtils.java:321)
    org.springframework.orm.hibernate3.SessionFactoryUtils.getSession(SessionFactoryUtils.java:202)
Review the /var/log/zypp/history fileEnsure the 500 Error is caused by a failure of running the required update scripts.
In most cases, due to the fact that a filr service had the SQL database in use, inaccessible or not running, the database and database schema update failed to execute properly, as shown here:
# Liquibase Update Failed: Network error IOException: Cannot assign requested address (Address not available)
# SEVERE 7/17/18 8:21 AM:liquibase: Network error IOException: Cannot assign requested address (Address not available)
# liquibase.exception.DatabaseException: java.sql.SQLException: Network error IOException: Cannot assign requested address (Address not available)
A few important items to consider:
  • A 500 error is highly generic, so this TID may not apply to your situation.  The key to this TID is that the error happens as a result of applying an update.
  • The symptom could occur with updates other than the 3.4.  The 3.4 is mentioned only because this issue was reported by multiple customers when going to that patch level.
  • While the 500 error is highly generic, the steps listed in the Resolution are also generic. Therefore the Resolution steps have the potential to resolve a myriad of problems that result from an update gone wrong.
  • When in doubt, contact the Micro Focus Customer Support.

Resolution

Manually run the Filr update scripts by following these steps.
  1. Ensure the indexserver service is running on each of the Search appliance and the Database is up and running (be it in the MySQL appliance or the MySQL, MariaDB or MSSQL database environment.
    • rcindexserver restart
  2. Stop the Filr service on all Filr appliances.
    • rcfilr stop
  3. Run the .sh files available in /var/adm/update-scripts/ in alphabetical order:
    • Example:
      • /var/adm/update-scripts/filr-db-3.0.0-278.2-update-database.sh
      • /var/adm/update-scripts/filr-webapps-3.0.0-493.2-update-server-xml.sh
      • /var/adm/update-scripts/web-filr-3.0.0-208.2-restart-9443-services.sh
  4. The scripts should start the filr service, however you can verify the Filr service on all Filr servers.
    • rcfilr status

Part of the upgrade scripts is to stop the filr service. As the filr service was stopped manually before executing the update scripts, these errors can be ignored:
# SEVERE: Catalina.stop:
# java.net.ConnectException: Connection refuse

Cause

There are several reasons  the update scripts do not executed properly. For instance,
  • At the time the update was executed, the Filr service was running and not halted in time.
  • The index appliances were not fully up and running.
  • The database was not running or accessible 
  • The update was not able to complete because the Tomcat Filr service was running.
    • Note: the update shuts down the Filr server (which is a requirement for the update to succeed).

Manually running the update while Tomcat is shutdown and the other required service are running often gets past the problem.

Additional Information

As instructed in the Filr Update TID or the patch finder page the release of a Filr update and in the regular documentation, the upgrade needs to be done in the same sequence as starting an expandable or large Filr setup.
The recommended start up sequence for the different types of the Filr appliances is: MySQL > Search > Filr

Basically, in brief this is the procedure and sequence the appliances are to be updated:
  • Stop the filr service on all Filr appliances (rcfilr stop)
  • Apply the available online updates on the MySQL appliance, as documented, and restart the appliance if required.*
  • Apply the available online updates on the Search Appliances (aka Index or Lucene appliance) and restart the appliance if required.
  • As soon as the database and index services are up and running and accessible, apply the available online updates on the Filr appliance and restart the appliance if required.
* In case the Filr environment was setup using a database environment (be it MySQL, MariaDB or MSSQL) and not the MySQL appliance there are no online updates available via the Filr update channel.
It is then up to the database administrators to update the database servers.
However it's required that the database is running and accessible while applying the updates to the Filr appliances.
Please be aware that it is impossible to perform a "rolling upgrade". All Filr appliances in an expandable Filr setup need to be updated one by one at the same moment and need to run the same release of Filr.