LAG does not start up anymore after the disk was out of space

  • 7002537
  • 04-Feb-2009
  • 14-Nov-2013

Environment

Novell Access Manager 3 Support Pack 4 Interim Release 1
Novell Access Manager 3 Linux Access Gateway
Novell Access Manager 3.1 Service Pack 5

Linux Access Gateway has been installed under VMWare
Standard installation process has been used for the LAG installation

Situation

A cronjob has been created on the LAG to force NTP time-sync more frequently due to the known VMWare time-slip problem
This generated a huge amount of log files in addition to the fact that debug logging had been enabled for the LAG as well
The disk got completely filled up with log and mail messages which cause the LAG to stop working under this condition.
Cleaning up the disk and rebooting the LAG did not help to the the proxy services operational again.



Resolution

This issue has been addressed to engineering

Workaround:
  • rm -f /var/novell/cfgdb/.current/virtual.xml
  • rm -f /var/novell/cfgdb/.current/.virtual.xml.lock
  • cp /var/novell/cfgdb/factory/virtual.xml  /var/novell/cfgdb/.current/.
  • /opt/novell/cli/bin/nash /var/novell/vms/diskcommands.cfg
  • /etc/init.d/novell-vmc restart
Note 1: This will not work with Novell Access Manager 3.1 Service Pack 3 or greater
In order to run the above steps the "/var/novell/vms/diskcommands.cfg" needs to by modified before by changing the following entry from "'vm scan'  to  'vm scan 0'

Note 2: /opt/novell/cli/bin/nash /var/novell/vms/diskcommands.cfg will only work if there are up to 15 CPU cores activated on the server. If the server has more than 15 cores enabled, the additional command must be run first ie. before the above nash command:

 

echo 0 > /sys/devices/system/cpu/cpu<cpu_number>/online

 

This deactivate the additional cores. After the above nash command is run, the deactivated cores can be activated again.


Additional Information

In oder to get more details about the root cause not starting the proxy services set the required log level to debug as documented in the Novell Access Manager adminguide and review the "/var/log/ics_dyn.log" file. in the above mentioned situation the following log statements had been written:

404514000: : 0: 0: VmController Server Process is coming up (27414, log level 7)
504514000: : 0: 0: Virtualization Configuration not available, start stand alone mode