Environment
- Access Manager 4.5.x
- Access Manager 5.x
Situation
- Micro Focus Access Manager Gateway has been configured to protected a Filr server
- uploading a files to Filr more than 1 GB in size through NAM Access Gateway fails.
- uploading files without having the Access Gateway proxy service in the communication path works without any problems
- the symptom does not depend on any browser type
- non formfill policy has been defined on the sub path: "filr/user/my-files"
- the HTTP "Content-Type: multipart/form-data" used to upload files will not be evaluated by the proxy rewriter process
Resolution
- Increase the Proxy Service => Web Servers = > TCP Connection Options Data => Data Read Timeout to a higher value. you can start with doubling the value to 240
(4 minutes) and run some tests - Investigate the filr server load
Cause
The persistent TCP session to upload the file is slows down caused by the Filr server not keeping up ack-ing the revived data fast enough. At the moment the TCP ACK for some send data has not been revived for more than 120 Seconds the proxy service will close the connection.
The name of the parameter "Data Read Timeout" might be misleading, In fact it is not only used while downloading web objects but as well for reviving Acks for send data
Additional Information
Troubleshooting
- If you are running an Access Gateway Cluster make sure you have a workstation configured to access just one node by using a local hosts file for DNS resolution. This will make sure you do not have to investigate multiple LANB trace or logfiles
- enable the following Access Gateway logging using the Access Gateway Advanced Options setting in order to have the required data in:
+++++++++++++++++++++++++++++++++++++++++++++++++++++
"/var/log/novell-apache2/error_log"
"/var/log/novell-apache2/httpheaders"
+++++++++++++++++++++++++++++++++++++++++++++++++++++
Access Gateway Advanced Options
+++++++++++++++++++++++++++++++++++++++++++++++++++++
# Apache Proxy Service Logging
LogLevel warn
LogLevel novell_ag_module:debug
Loglevel cache:debug ssl:crit core:crit
LogLevel proxy:debug proxy_balancer:crit proxy_ajp:info proxy_http:crit
# Debug HTTP Messages
DumpHeaders on
DumpResponseHeaders on
NAGGlobalOptions DebugHeaders=on
DumpHeadersFacility local6
DumpResponseHeadersFacility local6
+++++++++++++++++++++++++++++++++++++++++++++++++++++ - create a LAN trace which does not capture the full packet size suing the "-s" parameter.
For example: tcpdump -i eth0 -s 66 host [IP Address of Filr Server] -w filr-trace.cap - clear out all logfiles and start the file upload
- From the log files you need to find the failing session by for example looking for the upload file name in the logs. From the "httpheaders" log you can find the "TCP source port" used by the proxy service and match this with the taken LAN trace to filter the given session out. Note: "to-ws" = To Webserver. IN the following example you will see
2021-07-15T12:00:09.785492+02:00 lxnamgw1 httpd[25590]: ID:14269:2148:to-ws [192.168.20.10:52964->192.168.20.21:443]
POST /rest/self/my_files/library_files?file_name=netware.ova.zip&include_user_quota=true&mod_date=2020-**************Z HTTP/1.1
2021-07-15T12:00:09.785518+02:00 AccessGateway1 httpd[25590]: ID:14269:2148:to-ws Host:filr.ema.corp
2021-07-15T12:00:09.785538+02:00 AccessGateway1 httpd[25590]: ID:14269:2148:to-ws User-Agent: ***********************************
2021-07-15T12:00:09.785557+02:00 AccessGateway1 httpd[25590]: ID:14269:2148:to-ws Accept: */*
2021-07-15T12:00:09.785577+02:00 AccessGateway1 httpd[25590]: ID:14269:2148:to-ws Accept-Language: en-US,en;q=0.8,es;q=0.5,es-ES;q=0.3
2021-07-15T12:00:09.785596+02:00 AccessGateway1 httpd[25590]: ID:14269:2148:to-ws Accept-Encoding: gzip, br
2021-07-15T12:00:09.785615+02:00 AccessGateway1 httpd[25590]: ID:14269:2148:to-ws Authorization: Basic
2021-07-15T12:00:09.785635+02:00 AccessGateway1 httpd[25590]: ID:14269:2148:to-ws X-Requested-With: XMLHttpRequest
2021-07-15T12:00:09.785654+02:00 AccessGateway1 httpd[25590]: ID:14269:2148:to-ws Content-Type: multipart/form-data; boundary=---------------------------2588227729365
2021-07-15T12:00:09.785673+02:00 AccessGateway1 httpd[25590]: ID:14269:2148:to-ws Referer: https://filr.ema.corp/filr/user/my-files
2021-07-15T12:00:09.785693+02:00 AccessGateway1 httpd[25590]: ID:14269:2148:to-ws Cookie:******************************************
2021-07-15T12:00:09.785713+02:00 AccessGateway1 httpd[25590]: ID:14269:2148:to-ws X-Forwarded-Proto: https
2021-07-15T12:00:09.785732+02:00 AccessGateway1 httpd[25590]: ID:14269:2148:to-ws Via: 1.1filr.microfocus.com (Access Gateway-ag-***********-14269)
2021-07-15T12:00:09.785752+02:00 AccessGateway1 httpd[25590]: ID:14269:2148:to-ws X-Forwarded-For: 192.168.20.100
2021-07-15T12:00:09.785771+02:00 AccessGateway1 httpd[25590]: ID:14269:2148:to-ws X-Forwarded-Host: filr.novell.com
2021-07-15T12:00:09.785791+02:00 AccessGateway1 httpd[25590]: ID:14269:2148:to-ws X-Forwarded-Server: filr.novell.com
2021-07-15T12:00:09.785816+02:00 AccessGateway1 httpd[25590]: ID:14269:2148:to-ws Content-Length:3054159508
2021-07-15T12:19:37.046473+02:00 AccessGateway1 httpd[25590]: ID:14269:2148:cres status:502 502 Proxy Error