GWIA Worker Threads Hung/Stuck

• Inbound Email to the GWIA is not received.
• All or some GWIA's inbound/Receive email worker threads are used up.
• Receive worker threads are not releasing.
• Receive worker threads are being used for long periods of time, hours, or even days.
• Restarting the GWIA, or adding extra Receive threads only buys time until the problem happens again.
• X-Files in the GWIA's receive directory were not growing, these corresponded to the threads that were hung open for extended periods of time.
  

Possible solutions include:
1)  A Firewall was upgraded and thought the SMTP packets going to the GWIA were suspicious and quit allowing them through.  This prevented the GWIA from ever receiving the final packets including the closing packet to have the GWIA route the email.
2)  Possible SPAM filter not closing out the connection correctly or at all.
3)  Stuck message in the GWIA's receive queue that was causing the GWIA to choke.
4)  Low disk space preventing the GWIA from properly handling the message, and not able to process any more data.

As the problem came out of the blue, it was not believed to be a code issue.  Creating a second GWIA on another box can help determine where the problem may be.  In one instance, the second GWIA was able to determine that the problem was not the original GWIA or something on the GWIA's box.