FIPS Connection Fails If Certificate Key Length Is Too Short

  • 7021861
  • 01-Mar-2012
  • 01-Apr-2018

Environment

INFOConnect Enterprise Edition version 9.x

Situation

Sessions that are configured to use Attachmate FIPS 140-2 encryption with an x.509v3 certificate will fail to connect if the RSA public key length does not meet the FIPS minimum requirement.

Resolution

If you are not required to operate in FIPS mode, configure the session to use Attachmate TLS 1.0 encryption. Attachmate TLS 1.0 supports PCI DSS security standards, providing a range of encryption from medium to strong. The actual encryption strength is the maximum that both the host and client can support (in the order of preference specified by the client).

To change the encryption for a session:

  1. In the Accessory Manager, open a session, and click Edit > Settings.
  2. On the left, select Connection, and then do one of the following:
    • For IBM AS/400 sessions, on the General tab, for Security type, select Attachmate TLS v1.0 and leave Encryption Strength set to Auto.
    • For IBM Mainframe sessions, on the General tab, click Add. In the Configure Connection dialog box, for Security type, select Attachmate TLS v1.0 and leave Encryption Strength set to Auto.
    • For Unisys sessions, on the Connection tab, click Advanced. In the Path Wizard, follow the prompts to add your settings. On the page that displays your IP address or host name and port, select Attachmate TLS. Complete the Path Wizard.
    • For VMS/UNIX/Asynchronous sessions, on the General tab, click Advanced. In the Reflection Secure Shell Settings dialog box, click the Encryption tab and then select Attachmate TLS v1.0.
  1. (Optional) To determine the active encryption strength for the session, hold the cursor over the padlock icon until the tooltip appears.

Cause

The RSA key length of the RSA server certificate is less than 2048 bits. FIPS 140-2 encryption requires the key length to be 2048 bits or greater.

The length of public keys used to exchange symmetric keys must correspond to the strength of the symmetric key algorithm in use. For example, if you use the 3DES encryption algorithm, which has an effective key strength of 112 bits, the RSA key in the server’s certificate must be at least 2048 bits in length. Similarly, if you use AES encryption (128- or 256-bit), the RSA key in the server’s certificate must be 3072 bits or greater in length. The client is unable to determine whether the RSA key meets these requirements until the server sends its certificate during the SSL/TLS negotiation process. If the key doesn’t meet the minimum required length, the connection fails.

Additional Information

Legacy KB ID

This document was originally published as Attachmate Technical Note 2613.