Environment
Open Enterprise Server 2018 (OES 2018) Linux
Open Enterprise Server 2015 (OES 2015) Linux Support Pack 1
Open Enterprise Server 2015 (OES 2015) Linux
SUSE Linux Enterprise Server 12
VMware ESX Server 6.5
SUSE Linux Enterprise Server 12
VMware ESX Server 6.5
Situation
When performing normal work such as email, document/spreadsheet updates, etc., the server operates very well. However, when a heavy data workload -- such as a database load or database re-index – was performed from multiple workstations, at least one workstation would start to get file or record lock errors; or contention with opening files. Further, it was observed that the holder of the lock on the file/record was a different connection number for the same user.
Resolution
The default kernel IO scheduler is ‘cfq’ which performs IO optimization primarily designed for locally attached disks and tries to be “fair” with all IOs. These optimizations are redundant, when using a disk from a storage system (i.e. a SAN), as the storage system does its own IO optimizations – thereby creating unnecessary overhead on both the server and the connected storage system. This symptom may not be specific to VMware but could also be seen on any other Hypervisor such as Hyper-V, XEN or KVM.
To address this behavior, a different IO scheduler can employed which should make disk IO more efficient.
Temporary change to I/O scheduler
For temporary testing, it is possible to change the IO scheduler for one or more individual disks without rebooting the server. To do so, execute the following as root:
echo SCHEDNAME > /sys/block/DEV/queue/scheduler
replacing:
SCHEDNAME with the scheduler name (noop or deadline) and
DEV with the device the scheduler should be used on (i.e. sdc).
If you want temporarily change the scheduler for multiple disks, you will need to run the “echo” command (above) against each device; replacing the SCHEDNAME and DEV; for each device.
Note: when the server is restarted, the default scheduler will return for each of these devices.
More permanent change across all devices
To implement a non-cfq scheduler across all disks, that remains after the host is restarted, you will need to:
- Go into YaST
- Select “System” -> “Boot Loader” and press <enter>
- Go to “Kernel Parameters” tab and go into the “Optional Kernel Command Line Parameter” section.
- Add “elevator=SCHEDNAME” (where SCHEDNAME is the scheduler desired).
- Save/OK and exit out of YaST.
- Restart the server to activate the new scheduler.
This will turn off the SLES’s disk IO optimization making the virtual machine's IO a FIFO (first in, first out) to the underlying storage system. All IO optimizations, with this setting, will be accomplished on the storage system.
For temporary testing, it is possible to change the IO scheduler for one or more individual disks without rebooting the server. To do so, execute the following as root:
echo SCHEDNAME > /sys/block/DEV/queue/scheduler
replacing:
SCHEDNAME with the scheduler name (noop or deadline) and
DEV with the device the scheduler should be used on (i.e. sdc).
If you want temporarily change the scheduler for multiple disks, you will need to run the “echo” command (above) against each device; replacing the SCHEDNAME and DEV; for each device.
Note: when the server is restarted, the default scheduler will return for each of these devices.
More permanent change across all devices
To implement a non-cfq scheduler across all disks, that remains after the host is restarted, you will need to:
- Go into YaST
- Select “System” -> “Boot Loader” and press <enter>
- Go to “Kernel Parameters” tab and go into the “Optional Kernel Command Line Parameter” section.
- Add “elevator=SCHEDNAME” (where SCHEDNAME is the scheduler desired).
- Save/OK and exit out of YaST.
- Restart the server to activate the new scheduler.
This will turn off the SLES’s disk IO optimization making the virtual machine's IO a FIFO (first in, first out) to the underlying storage system. All IO optimizations, with this setting, will be accomplished on the storage system.
Note: in SLES 11 based Open Enterprise Server 2015|2015SP1, the DEFAULT_APPEND variable in /etc/sysconfig/bootloader would need to be updated to ensure the “elevator=” parameter would not get overwritten during a kernel update. This is not necessary with SLES 12/OES 2018 as the system will retain "elevator=" parameter after a kernel update.
Additional Information
For more information see the following documentation files:
/usr/src/linux/Documentation/kernel-parameters.txtThese files are provided by the kernel-source RPM; available on the installation media or via regular SLES online update repositories.
/usr/src/linux/Documentation/block/switching-sched.txt
/usr/src/linux/Documentation/block/as-iosched.txt
/usr/src/linux/Documentation/block/deadline-iosched.txt
Additional info from VMware: https://kb.vmware.com/s/article/2011861