iSCSI Overview - Optimization and Performance tuning.

  • 3754710
  • 27-Oct-2006
  • 27-Apr-2012


Internet Small Computer Systems Interface - iSCSI
Novell NetWare 6.5 Support Pack 5


Many customers today are finding interest in an affordable storage solution over the higher cost
FC (fiber channel) SAN solutions. There are many third party vendors delivering these types of
storage solutions and many more on the horizon. iSCSI has even began to gain support among some
SAN vendors which may have an iSCSI interface available on their SAN's. Along with this new found
interest comes a host of questions concerns and "alphabet soup". We hope to address all of these within
this document.
First, it's important for us to understand what iSCSI is and the main difference there is between a SAN.
iSCSI - Internet SCSI (small computer systems interface) provides the means of accessing external storage
arrays via Ethernet or LAN based connectivity. Legacy systems such as SAN's are generally accessible via
fiber channel. SAN connectivity can involve multiple fiber connection from a host (server) to a FC switch
which then has a FC connection to the SAN. Historically faster, this is due to the 4 Gbps fiber connection
versus 1 Gbps Ethernet and the limitation of iSCSI drives being SATA only. With the fast approach of
10 Gbps Ethernet and the expected availability of iSCSI/SAS drives iSCSI is becoming more and more a viable
solution for businesses small to large in size.


To address the here and now we will discuss iSCSI optimization to provide the best possible throughput
for most customers. Keeping in mind that infrastructure, hardware, and configuration can drastically affect
iSCSI performance as well as the iSCSI delivery implementation, meaning the end services provided using
Let us start with connectivity ( NIC's, Switches, varying technologies) which is the cornerstone for a successful
iSCSI implementation. Contrary to some reports and beliefs high network traffic and/or "collisions" does not
result in data corruption, it does however result in poor throughput and performance. In a properly configured switch environment this shouldn't be an issue. For the best possible performance we recommend using Jumbo frames (Ethernet enhancement that allows for up to 9k frames as opposed to 1.5k frames) especially in an environment where high iSCSI traffic is to be expected ex. Web servers, mail servers, backups etc.
The use of jumbo frames we drastically reduce the number of packets traversing the iSCSI channel/link. The reduction of packet traffic will also reduce the overhead associated to packet transfer ex. checksum operations, Acknowledgements and the overall operation of packet formatting which greatly affects the CPU. So in a"nutshell" - A larger payload will obviously decrease the amount of traffic on the wire and the amount of work
both the iSCSI initiator and target would have to perform.
When implementing jumbo frames verify that your NIC's have the latest drivers and BIOS available.
Switches should have the latest firmware available as well as any IOS specific to the switch. And of course,
jumbo frames can only be implemented on devices (NIC's and Switches) that support jumbo frames. On
NetWare we recommend setting the following parameters -
Set maximum physical receive packet size = 18000
This parameter is specific to the frame size and in the case aboveit is double the size of the expected frame which is 9k. Had the jumbo size specified by the NIC been 4500 then the above setting would be at least 9000.
We also recommend using a higher Minimum and Maximum number of packet receive buffers. These values can be set at your own discretion.
At the onset of iSCSI development there was initially an issue where during data transfer the NetWare
initiator could only achieve an 11K maximum window size. This greatly affected the overall performance of iSCSI and caused high demand Apps and data transfer to become exceedingly slow. This drawback has since
been addressed and corrected in the latest Win-sock code available as well as implementing the set parameter listed below which will increase the window size during the initial connection establishment -
Set TCP Maximum Initial Receive Window = 64
This will cause the initial receive window that is advertised by TCP to be 64K. In most cases the window
starts out at a smaller size and grows to 65k if possible.
Failure to set this parameter may result in a "stuck" window size of"0". This is because the default setting
is for 6k and a window size greater than 6k is attempted before the connection has been established. In this
state the window can become frozen/stuck. This will be corrected in a post SP6 TCPIP.
Another important factor to consider are the load line parameters associated with the NIC. Many provide offloading ( Checksum, segmentation, etc.) to optimize throughput by moving these processes from the machines CPU to a processor directly on the NIC. Take extreme care and test before implementing some of these load line parameters in a production environment. Many have been found to cause some problems and anomalies within a NetWare environment. We recommend setting flow control and duplex manually on each device or port, we do not recommend using"auto". We have not done any testing with TOE (TCP Offloading Engine) which is touted to be an exceeding improvement for higher throughput. TOE accomplishes this by offloading many CPU intensive processes that are TCP specific such as Checksum and Sequence number calculations, handling Ack's, Windowing (sliding window/data receive size capabilities), and even connection establishment and termination not excluding encapsulation. We believe that TOE along with 10Gbps Ethernet (once both technologies mature and become more available) users can see a tremendous improvement in throughput which could eventually exceed that of fiber channel .
NetWare 6.5 specific recommendation irrespective to iSCSI target being used -
1. Net Ware server should be running support pack 5 or later (this is due to TCP updates and optimization that directly affects iSCSI performance.
2. Server should be running the latest available version of Win-sock which at the time of this writing SP6 is due for release and that is the recommended version. Since Win-sock provides the interface between iSCSI and TCP it is imperative that this piece is updated. Many enhancements and fixes have been provided to the later versions of Win-sock and many of these are directly specific to iSCSI.
3. Server should have sufficient memory available and memory should be optimally tuned as well.
See TID 3920657
4. It is not recommended to run mission critical apps across iSCSI due to the delays that can very well
occur during a high utilization period.
5. It is not recommended to attempt backups during production times or periods of high user activity.
Continuing work is being done to improve the overall performance and reliability of iSCSI.
Some of these include improving TCP/Win-sock/iSCSI interaction with specifics to windowing
initial connection negotiation and packet ACK'ing. At this time only fail-over is supported for
multiple NIC usage but due to customer request we are considering adding support for MPIO
(multi-path I/O) to create a larger pipe, 2 NIC's running 1Gig Ethernet in a round robin config to
theoretically achieve a 2 Gig pipe which would provide to distinctive paths to a volume. This would
improve overall performance and throughput greatly. Also, there has been much interest in SAN boot
capability which could be achieved through HBA support on iSCSI. This is also a considered future
enhancement .