Autodiscover: How Retain Connects to Your Exchange Mailboxes

This article discusses how Retain connects to your Exchange mailboxes.  It is through a Microsoft process called Autodiscover.  It is important for the reader to understand that this is not a process developed by GWAVA, but a process developed by Microsoft for client (i.e., Outlook or third party applications like Retain) connections to the Exchange system.  Every client must go through Autodiscover. AutoDiscover errors can appear in Retain Worker logs when it attempts to login to a mailbox.  This article is designed to:

1. Help the reader understand how Autodiscover works so that the Exchange setup makes more sense.
2. To provide a foundation for troubleshooting issues as they arise, as they are almost always indicative of setup issues in the Exchange system (see also "AutoDiscover Errors on Some Mailboxes"), (Autodiscover)

The process for connecting to an Exchange mailbox was specified by Microsoft (see "Autodiscover for Exchange"). This article provides a general overview of the interaction between the Retain Worker and Autodiscover as well as tips on reading the Worker logs to better understand the information Retain is receiving from Autodiscover to help you troubleshoot issues.

General Overview


1.  Retain Worker queries DNS to find the AD DS server.

2.  DNS server returns the location.

3.  Retain Worker queries AD for Autodiscover SCP records.

4.  AD provides the Autodiscover URLs.

5.  Retain Worker attempts to connect to the Autodiscover service on the CAS based on the URL it was provided.  It finds the server through DNS.

6.  Autodiscover connects Retain Worker to the EWS.

7.  EWS connects Retain Worker the the user mailbox.

8.  Exchange provides the mailbox data and passes it through EWS.

9.  EWS passes the mailbox data to Retain Worker which passes the data to the Retain Server.

Autodiscover Process

In Retain version 3.x and 4.0, we would use the Autodiscover library provided by Microsoft; however, in some instances - due to configuration issues in the customer's environment, this would cause lag time in connection with each mailbox, sometimes resulting in 2-minute delays for each mailbox.  In Retain 4.0.1, we will use our own adaptation of the Autodiscover process, which has eliminated the delays.  If our own process fails, then Retain tries the Microsoft Autodiscover library.

The order of which Autodiscover code gets used can be changed.  See, "Changing the Autodiscover Mode".

The Microsoft Autodiscover process has three phases, but only the first two are used if successful after phase 2:

PHASE 1:  Define the Candidate Pool

This is the part where the client (Retain) has to locate the right Autodiscover server for the mailbox to which it is attempting to login.  In the case of Retain, we are using the impersonation account you set up in Exchange for Retain to use (see Exchange Module Setup Instructions, "Preparatory Steps" section, steps 2 - 4).

1.  It first looks in Active Directory Domain Services (AD DS) for a service connection point (SCP) object, which can redirect Retain to the Autodiscover server.

You can find your SCP record(s) in AD DS.  However, you can also run the Get-ClientAccessServer cmdlet in an Exchange Management Shell by typing:  Get-ClientAccessServer | fl

If it returns a value for "AutoDiscoverServiceInternalUri", then an SCP record exists and it is pointing to one of the Autodiscover servers.  If you have multiple client access servers (CAS), each should have its own SCP record and unique Autodiscover URL.

This allows Autodiscover requests to be routed to servers based on Active Directory sites.  So, if the company is GWAVA and they have several sites, SCP might provide a number of Autodiscover servers:

https://orem.gwava.com/autodiscover/autodiscover.xml
https://montreal.gwava.com/autodiscover/autodiscover.xml
https://ahuas.gwava.com/autodiscover/autodiscover.xml

2.  Next, it looks at the domain part of the user's email address (for jdoe@gwava.com, gwava.com is the domain) and comes up with two Autodiscover standard endpoint URL forms using that domain.

Using jdoe@gwava.com, it comes up with:

https://gwava.com/autodiscover/autodiscover.xml
https://autodiscover.gwava.com/autodiscover/autodiscover.xml

Now it has a compiled list of 5 potential Autodiscover servers:

https://orem.gwava.com/autodiscover/autodiscover.xml
https://montreal.gwava.com/autodiscover/autodiscover.xml
https://ahuas.gwava.com/autodiscover/autodiscover.xml
https://gwava.com/autodiscover/autodiscover.xml
https://autodiscover.gwava.com/autodiscover/autodiscover.xml

PHASE 2:  Try each potential Autodiscover server

It goes through the list of potential Autodiscover servers trying to find the endpoint mailbox.  It uses the first Autodiscover server that produces a successful connection.  Retain uses the POX Autodiscover service as defined by Microsoft, so an HTTP POST with an Autodiscover request body.

If each of those potential Autodiscover server URLs fail, then we move onto phase 3.

PHASE 3:

Since the URLs gathered in Phase 1 failed during Phase 2, the Microsoft library next tries other alternatives.

1.  Sends an unauthenticated GET request derived from the user's email address in this format:

Using jdoe@gwava.com, it comes up with:

http://autodiscover.gwava.com/autodiscover/autodiscover.xml

This is not an SSL encapsulated request; however, if the server returns a 302 redirect response, we attempt to resend the Autodiscover request to the endpoint URL.

2.  If step #1 fails, we try qurying DNS for an SRV record for the Autodiscover service.  See also "Creating a DNS SRV Record for Exchange".

Using jdoe@gwava.com, the record takes the form of _autodiscover._tcp.gwava.com.  This query might return multiple records, but we only use records that point to an SSL endpoint and that have the highest priority and weight.

If it fails after completing all three phases, then Retain returns an error, "No Autodiscover/Endpoint found!" error.

TESTING CONNECTIVITY

Microsoft provides a connectivity tester utility.  For more details, see https://support.microfocus.com/kb/doc.php?id=7020687

LOG READING TIPS

The log will show the method it uses to discover Autodiscover.  Note in the following example where it states "class=scp3".  In this case, the Autodiscover server was found through SCP.  Retain's logging changes over time, but here are some examples from a log in Retain 3.2:

09:18:02,899 AutoDiscover - The Autodiscover Server is in cache, so that's good: url=https://[server DNS hostname]/autodiscover/autodiscover.xml,class=scp3

In the following example, the AutoDiscover server is found but the endpoint (or mailbox) could not be found.  This means that Retain found AutoDiscover, but AutoDiscover did not provide a valid EWS URL for that mailbox.  The reason for this issue is documented in the KB article, "AutoDiscover Errors on Some Mailboxes" (note: this occurred in Retain 3.1 and logging messages may have changed since then for this particular situation):

09:18:02,977 AutodiscoverFunctions - No EWSURL or ASURL in response request for jdoe@xyzcompany.com
09:18:02,977 AutodiscoverFunctions - This will mean we will fail to find an EWS service suitable for this user.
09:18:02,977 AutoDiscover - Not good. No good EWS URL returned... code=connectButNoUrl,payload=[null, -1]
09:18:02,977 ExchangeDredger - No Autodiscover/Endpoint found!

Here is a list of log messages from Retain 3.5 under various circumstances as noted:

Setup Issue:  Retain Server's Primary DNS Not the Same as the CAS Server(s)

2015-03-24 14:16:21,039 ERROR [RTWQuartzScheduler_Archive_Worker-1] com.gwava.retain.worker.rest.client.JobStatusTracker: Unable to send request to server.
org.springframework.web.client.HttpServerErrorException: 503 Service Unavailable

Setup Issue:  Basic Authentication Not Enabled on Autodiscover, Impersonation User Password Expired or SRV Record was not setup correctly/missing.

2015-03-24 14:28:41,076 ERROR [RTWQuartzScheduler_Archive_Worker-6] com.gwava.utils.StandardErrorHandleStrategy: reportError: Exception during discover of SRVURLs :: com.gwava.ews.autodiscover.AutoDiscover.getEWSUrl:114 :: javax.naming.NameNotFoundException: DNS name not found [response code 3]; remaining name '_autodiscover._tcp.ad.gwavasupport.com'

Setup Issue:  Basic Authentication Not Enabled on EWS (but Enabled on Autodiscover)

You will not even get this far for this error unless basic authentication is enabled on Autodiscover; otherwise, you get the error listed for when basic authentication is not enabled on Autodiscover (above).  It does not even get to the point of logging in to EWS because it cannot get past authenticating to the Autodiscover service.  So, again, if you get the following error, you know that basic authentication is enabled at least for Autodiscover; but, it is not enabled for EWS.  The following are snippets of various lines in the Worker log that you'll find under this error condition:

2015-03-24 14:32:15,453 INFO  [RTWQuartzScheduler_Archive_Worker-8] com.gwava.ews.archiveimpl.process.ExchangeUser: discovered: https://exmail.ad.gwavasupport.com/EWS/Exchange.asmx
2015-03-24 14:32:27,199 ERROR [RTWQuartzScheduler_Archive_Worker-8] com.gwava.ews.archiveimpl.process.ExchangeArchiveFactory:
com.sun.xml.ws.client.ClientTransportException: The server sent HTTP status code 401: Unauthorized
    2015-03-24 14:32:27,224 DEBUG [RTWQuartzScheduler_Archive_Worker-8] com.gwava.ews.archiveimpl.process.ExchangeArchiveFactory: EWS request failed (checkConnection). Will retry after 2 seconds
 

This article was originally published in the GWAVA knowledgebase as article ID 2242.