Environment
Novell Access Manager 3 Linux Access Gateway
Situation
A working Access Manager 3.0 environment was about to be upgraded to 3.1. Before doing this, the expiring Identity (IDP) server certificate on 3.0 had to be renewed until the 3.1 environment was fully ready. AFter renewing the IDP server certificate, users hitting the Linux Access Gateway (LAG) protected resources would get a 100101043 error on the browser.
The catalina.out file showed that the LAG had an issue processing the new IDP server certificate:
Creating a new self signed IDP certificate using the did not make any difference - it appeared that any new certificate sent by the IDP server would trigger the error:
"Received fatal alert: bad_record_mac"
The catalina.out file showed that the LAG had an issue processing the new IDP server certificate:
// Snippet of log showing issue
<amLogEntry> 2011-04-14T09:55:03Z INFO NIDS Application: AM#500105025: AMDEVICEID#D251D7AF16E764A4: AMAUTHID#AC3796493E22B23BA69ECA6C33B40AED: IDP is requesting metadata from ESP https://dp.namit.mccgo.com:443/nesp/idff/metadata</amLogEntry>
<amLogEntry> 2011-04-14T09:55:03Z NIDS Trace: Method: URLUtil.connectToURL()
Thread: http-10.117.89.131-8443-Processor49
(1 of 2):
Initiating WAIT for HTTP Outgoing Request object!
(2 of 2):
WAIT on HTTP Outgoing Request interrupted due to an InterruptedException!
</amLogEntry>
<amLogEntry> 2011-04-14T09:55:03Z SEVERE NIDS IDFF: AM#100106001: AMDEVICEID#D251D7AF16E764A4: Unable to load metadata for Embedded Service Provider: https://dp.namit.mccgo.com:443/nesp/idff/metadata, error: Received fatal alert: bad_record_mac </amLogEntry>
Thread: http-10.117.89.131-8443-Processor49
(1 of 2):
Initiating WAIT for HTTP Outgoing Request object!
(2 of 2):
WAIT on HTTP Outgoing Request interrupted due to an InterruptedException!
</amLogEntry>
<amLogEntry> 2011-04-14T09:55:03Z SEVERE NIDS IDFF: AM#100106001: AMDEVICEID#D251D7AF16E764A4: Unable to load metadata for Embedded Service Provider: https://dp.namit.mccgo.com:443/nesp/idff/metadata, error: Received fatal alert: bad_record_mac </amLogEntry>
<amLogEntry> 2011-04-14T09:55:03Z INFO NIDS Application: AM#500105039: AMDEVICEID#D251D7AF16E764A4: AMAUTHID#AC3796493E22B23BA69ECA6C33B40AED: Error on session id AC3796493E22B23BA69ECA6C33B40AED, error 100101043-D251D7AF16E764A4, Unable to complete authentication request. AM#100101043: AMDEVICEID#D251D7AF16E764A4: : Identity Provider failed to load Embedded Provider metadata </amLogEntry>
// End of snippet
Creating a new self signed IDP certificate using the did not make any difference - it appeared that any new certificate sent by the IDP server would trigger the error:
"Received fatal alert: bad_record_mac"
Resolution
Change communication between the IDP server and LAG to SSLv3 instead of TLS by doing the following:
1. edit /var/opt/novell/tomcat4/conf/tomcat4.conf on the LAG
1. edit /var/opt/novell/tomcat4/conf/tomcat4.conf on the LAG
2. go to bottom of file where the JAVA_OPTIONS are located eg.
JAVA_OPTS="${JAVA_OPTS} -Daxis.EngineConfigFactory=com.novell.nidp.liberty.wsf.axis.NIDPAxisEngineConfigFactory"
3. add a new line with
JAVA_OPTS="${JAVA_OPTS} -Dhttps.protocols="SSLv3
JAVA_OPTS="${JAVA_OPTS} -Dhttps.protocols="SSLv3
4. save and restart tomcat4 on the box.
Additional Information
When debugging SSL errors in a tomcat environment, there's a useful JAVA SSL debug option that can be enabled to give more verbose output. Go to /var/opt/novell/tomcat5/conf/tomcat5.conf (tomcat4 with Access Manager 3.0) on the LAG or IDP server and add
After restarting tomcat, we get the following type of output that helps identify what part of the SSL handshake the error occured at:
*** ServerHelloDone
JsseJCE: Using JSSE internal implementation for cipher RSA/ECB/PKCS1Padding
*** ClientKeyExchange, RSA PreMasterSecret, SSLv3
Random Secret: { 3, 0, 50, 46, 199, 32, 211, 8, 6, 167, 90, 172, 88, 204, 133, 7, 208, 62, 133, 19, 207, 116, 171, 235, 244, 177, 46, 175, 77, 252, 45, 174, 61, 77, 101, 78, 209, 95, 222, 123, 41, 94, 60, 233, 160, 150, 248, 85 }
HTTPRequest-gmyc09pv7ma3o, WRITE: SSLv3 Handshake, length = 260
SESSION KEYGEN:
PreMaster Secret:
0000: 03 00 32 2E C7 20 D3 08 06 A7 5A AC 58 CC 85 07 ..2.. ....Z.X...
0010: D0 3E 85 13 CF 74 AB EB F4 B1 2E AF 4D FC 2D AE .>...t......M.-.
0020: 3D 4D 65 4E D1 5F DE 7B 29 5E 3C E9 A0 96 F8 55 =MeN._..)^<....U
CONNECTION KEYGEN:
Client Nonce:
0000: 4D B6 49 2A 93 7E 53 77 EB F2 51 12 00 6D A5 99 M.I*..Sw..Q..m..
0010: D5 53 FC 76 45 A2 23 8E 0F 9D 5F C3 C2 5C 2F AE .S.vE.#..._..\/.
Server Nonce:
0000: 4D B6 49 27 81 89 83 0C 1F B5 FE E2 B7 C9 DB C7 M.I'............
0010: 72 3C 8C 23 75 10 44 81 BD 49 47 33 8A 0C 00 58 r<.#u.D..IG3...X
Master Secret:
0000: 5C BF 40 AA 49 1F A2 6C 74 7E E4 0A 86 3F CC 3A \.@.I..lt....?.:
0010: 74 45 FE 87 1B 28 CE 56 4D D2 33 DA A1 33 93 5E tE...(.VM.3..3.^
0020: 8F A8 3E DF 58 58 6A 8D B4 34 17 0E 91 19 B1 44 ..>.XXj..4.....D
Client MAC write Secret:
0000: 66 25 0F BF 49 81 C6 60 79 F3 4F 88 3A 02 1F 14 f%..I..`y.O.:...
Server MAC write Secret:
0000: BE 3E E4 B6 C2 34 B6 18 6F 38 9B 88 3C A2 E6 69 .>...4..o8..<..i
Client write key:
0000: A3 BD 27 FE 56 90 89 A4 21 80 1E 5E 41 96 76 F7 ..'.V...!..^A.v.
Server write key:
0000: 4D 0E 1D CC 4A 91 5D 35 06 40 9F 06 72 31 BB 22 M...J.]5.@..r1."
... no IV for cipher
HTTPRequest-gmyc09pv7ma3o, WRITE: SSLv3 Change Cipher Spec, length = 1
JsseJCE: Using JSSE internal implementation for cipher RC4
*** Finished
verify_data: { 224, 14, 101, 138, 3, 220, 96, 202, 135, 1, 137, 30, 4, 184, 57, 22, 19, 203, 85, 84, 118, 96, 249, 232, 26, 142, 185, 63, 49, 40, 19, 1, 182, 202, 147, 172 }
***
HTTPRequest-gmyc09pv7ma3o, WRITE: SSLv3 Handshake, length = 56
HTTPRequest-gmyc09pv7ma3o, READ: SSLv3 Alert, length = 2
HTTPRequest-gmyc09pv7ma3o, RECV SSLv3 ALERT: fatal, bad_record_mac
HTTPRequest-gmyc09pv7ma3o, called closeSocket()
HTTPRequest-gmyc09pv7ma3o, handling exception: javax.net.ssl.SSLException: Received fatal alert: bad_record_mac
JAVA_OPTS="${JAVA_OPTS} -Djavax.net.debug=ssl"
After restarting tomcat, we get the following type of output that helps identify what part of the SSL handshake the error occured at:
*** ServerHelloDone
JsseJCE: Using JSSE internal implementation for cipher RSA/ECB/PKCS1Padding
*** ClientKeyExchange, RSA PreMasterSecret, SSLv3
Random Secret: { 3, 0, 50, 46, 199, 32, 211, 8, 6, 167, 90, 172, 88, 204, 133, 7, 208, 62, 133, 19, 207, 116, 171, 235, 244, 177, 46, 175, 77, 252, 45, 174, 61, 77, 101, 78, 209, 95, 222, 123, 41, 94, 60, 233, 160, 150, 248, 85 }
HTTPRequest-gmyc09pv7ma3o, WRITE: SSLv3 Handshake, length = 260
SESSION KEYGEN:
PreMaster Secret:
0000: 03 00 32 2E C7 20 D3 08 06 A7 5A AC 58 CC 85 07 ..2.. ....Z.X...
0010: D0 3E 85 13 CF 74 AB EB F4 B1 2E AF 4D FC 2D AE .>...t......M.-.
0020: 3D 4D 65 4E D1 5F DE 7B 29 5E 3C E9 A0 96 F8 55 =MeN._..)^<....U
CONNECTION KEYGEN:
Client Nonce:
0000: 4D B6 49 2A 93 7E 53 77 EB F2 51 12 00 6D A5 99 M.I*..Sw..Q..m..
0010: D5 53 FC 76 45 A2 23 8E 0F 9D 5F C3 C2 5C 2F AE .S.vE.#..._..\/.
Server Nonce:
0000: 4D B6 49 27 81 89 83 0C 1F B5 FE E2 B7 C9 DB C7 M.I'............
0010: 72 3C 8C 23 75 10 44 81 BD 49 47 33 8A 0C 00 58 r<.#u.D..IG3...X
Master Secret:
0000: 5C BF 40 AA 49 1F A2 6C 74 7E E4 0A 86 3F CC 3A \.@.I..lt....?.:
0010: 74 45 FE 87 1B 28 CE 56 4D D2 33 DA A1 33 93 5E tE...(.VM.3..3.^
0020: 8F A8 3E DF 58 58 6A 8D B4 34 17 0E 91 19 B1 44 ..>.XXj..4.....D
Client MAC write Secret:
0000: 66 25 0F BF 49 81 C6 60 79 F3 4F 88 3A 02 1F 14 f%..I..`y.O.:...
Server MAC write Secret:
0000: BE 3E E4 B6 C2 34 B6 18 6F 38 9B 88 3C A2 E6 69 .>...4..o8..<..i
Client write key:
0000: A3 BD 27 FE 56 90 89 A4 21 80 1E 5E 41 96 76 F7 ..'.V...!..^A.v.
Server write key:
0000: 4D 0E 1D CC 4A 91 5D 35 06 40 9F 06 72 31 BB 22 M...J.]5.@..r1."
... no IV for cipher
HTTPRequest-gmyc09pv7ma3o, WRITE: SSLv3 Change Cipher Spec, length = 1
JsseJCE: Using JSSE internal implementation for cipher RC4
*** Finished
verify_data: { 224, 14, 101, 138, 3, 220, 96, 202, 135, 1, 137, 30, 4, 184, 57, 22, 19, 203, 85, 84, 118, 96, 249, 232, 26, 142, 185, 63, 49, 40, 19, 1, 182, 202, 147, 172 }
***
HTTPRequest-gmyc09pv7ma3o, WRITE: SSLv3 Handshake, length = 56
HTTPRequest-gmyc09pv7ma3o, READ: SSLv3 Alert, length = 2
HTTPRequest-gmyc09pv7ma3o, RECV SSLv3 ALERT: fatal, bad_record_mac
HTTPRequest-gmyc09pv7ma3o, called closeSocket()
HTTPRequest-gmyc09pv7ma3o, handling exception: javax.net.ssl.SSLException: Received fatal alert: bad_record_mac