Uploading files more than 1GB in size to a filr service protected by an Access Manager Gateway fails

  • 7025199
  • 20-Jul-2021
  • 20-Jul-2021

Environment

  • Access Manager 4.5.x
  • Access Manager 5.x

Situation

  • Micro Focus Access Manager Gateway has been configured to protected a Filr server
  • uploading a files to Filr more than 1 GB in size through NAM Access Gateway fails.
  • uploading files without having the Access Gateway proxy service in the communication path works without any problems
  • the symptom does not depend on any browser type
  • non formfill policy has been defined on the sub path: "filr/user/my-files"
  • the HTTP "Content-Type: multipart/form-data" used to upload files will not be evaluated by the proxy rewriter process

Resolution

  • Increase the Proxy Service => Web Servers = > TCP Connection Options Data => Data Read Timeout to a higher value. you can start with doubling the value to 240
    (4 minutes) and run some tests

  • Investigate the filr server load

Cause

The persistent TCP session to upload the file is slows down caused by the Filr server not keeping up ack-ing the revived data fast enough. At the moment the TCP ACK for some send data has not been revived for more than 120 Seconds the proxy service will close the connection.

The name of the parameter "Data Read Timeout" might be misleading, In fact it is not only used while downloading web objects but as well for reviving Acks for send data

Additional Information

Troubleshooting

  1. If you are running an Access Gateway Cluster make sure you have a workstation configured to access just one node by using a local hosts file for DNS resolution. This will make sure you do not have to investigate multiple LANB trace or logfiles

  2. enable the following Access Gateway logging using the Access Gateway Advanced Options setting in order to have the required data in:
    +++++++++++++++++++++++++++++++++++++++++++++++++++++
    "/var/log/novell-apache2/error_log"
    "/var/log/novell-apache2/httpheaders"

    +++++++++++++++++++++++++++++++++++++++++++++++++++++

    Access Gateway Advanced Options
    +++++++++++++++++++++++++++++++++++++++++++++++++++++
    # Apache Proxy Service Logging
    LogLevel warn
    LogLevel novell_ag_module:debug
    Loglevel cache:debug ssl:crit core:crit
    LogLevel proxy:debug proxy_balancer:crit proxy_ajp:info proxy_http:crit

    # Debug HTTP Messages
    DumpHeaders on
    DumpResponseHeaders on
    NAGGlobalOptions DebugHeaders=on
    DumpHeadersFacility local6
    DumpResponseHeadersFacility local6

    +++++++++++++++++++++++++++++++++++++++++++++++++++++

  3. create a LAN trace which does not capture the full packet size suing the "-s" parameter.
    For example: tcpdump -i eth0 -s 66 host [IP Address of Filr Server] -w filr-trace.cap

  4. clear out all logfiles and start the file upload

  5. From the log files you need to find the failing session  by for example looking for the upload file name in the logs. From the "httpheaders" log you can find the "TCP source port" used by the proxy service and match this with the taken LAN trace to filter the given session out. Note: "to-ws" = To Webserver. IN the following example you will see

    2021-07-15T12:00:09.785492+02:00 lxnamgw1 httpd[25590]: ID:14269:2148:to-ws [192.168.20.10:52964->192.168.20.21:443]
    POST /rest/self/my_files/library_files?file_name=netware.ova.zip&include_user_quota=true&mod_date=2020-**************Z HTTP/1.1
    2021-07-15T12:00:09.785518+02:00 AccessGateway1 httpd[25590]: ID:14269:2148:to-ws Host:filr.ema.corp
    2021-07-15T12:00:09.785538+02:00 AccessGateway1 httpd[25590]: ID:14269:2148:to-ws User-Agent: ***********************************
    2021-07-15T12:00:09.785557+02:00 AccessGateway1 httpd[25590]: ID:14269:2148:to-ws Accept: */*
    2021-07-15T12:00:09.785577+02:00 AccessGateway1 httpd[25590]: ID:14269:2148:to-ws Accept-Language: en-US,en;q=0.8,es;q=0.5,es-ES;q=0.3
    2021-07-15T12:00:09.785596+02:00 AccessGateway1 httpd[25590]: ID:14269:2148:to-ws Accept-Encoding: gzip, br
    2021-07-15T12:00:09.785615+02:00 AccessGateway1 httpd[25590]: ID:14269:2148:to-ws Authorization: Basic
    2021-07-15T12:00:09.785635+02:00 AccessGateway1 httpd[25590]: ID:14269:2148:to-ws X-Requested-With: XMLHttpRequest
    2021-07-15T12:00:09.785654+02:00 AccessGateway1 httpd[25590]: ID:14269:2148:to-ws Content-Type: multipart/form-data; boundary=---------------------------2588227729365
    2021-07-15T12:00:09.785673+02:00 AccessGateway1 httpd[25590]: ID:14269:2148:to-ws Referer: https://filr.ema.corp/filr/user/my-files
    2021-07-15T12:00:09.785693+02:00 AccessGateway1 httpd[25590]: ID:14269:2148:to-ws Cookie:******************************************
    2021-07-15T12:00:09.785713+02:00 AccessGateway1 httpd[25590]: ID:14269:2148:to-ws X-Forwarded-Proto: https
    2021-07-15T12:00:09.785732+02:00 AccessGateway1 httpd[25590]: ID:14269:2148:to-ws Via: 1.1filr.microfocus.com (Access Gateway-ag-***********-14269)
    2021-07-15T12:00:09.785752+02:00 AccessGateway1 httpd[25590]: ID:14269:2148:to-ws X-Forwarded-For: 192.168.20.100
    2021-07-15T12:00:09.785771+02:00 AccessGateway1 httpd[25590]: ID:14269:2148:to-ws X-Forwarded-Host: filr.novell.com
    2021-07-15T12:00:09.785791+02:00 AccessGateway1 httpd[25590]: ID:14269:2148:to-ws X-Forwarded-Server: filr.novell.com
    2021-07-15T12:00:09.785816+02:00 AccessGateway1 httpd[25590]: ID:14269:2148:to-ws Content-Length:3054159508
    2021-07-15T12:19:37.046473+02:00 AccessGateway1 httpd[25590]: ID:14269:2148:cres status:502 502 Proxy Error



Feedback service temporarily unavailable. For content questions or problems, please contact Support.