Exchange 2019 and XG WAF Publishing with MAPI/RPC

Hello all,

I have just deployed a personal Exchange 2019 setup for myself and published through the XG WAF and had some trouble getting it to work on a clean setup through the pre-configured templates. Now, I'm not an Exchange engineer by any means so a lot of my Exchange configuration is surface knowledge and extrapolation from the variety of guides out there. But when I set up the templates I ran into two issues preventing proper WAF setup and successful Microsoft Remote Connectivity Anlyzer (MRCA) from giving me the endorphin hit of green ticks across the board.

Firstly, I encountered an issue in two areas and they were:

  • MAPI HTTP authentication error
  • RPC over HTTPS error

Firstly, some notes on the build:

  • I am using the Microsoft recommended 3 domain setup which is autodiscover, webmail and Outlook Anywhere (oa)
  • XG is v18.0.0GA-354
  • Default WAF templates as provided by Sophos utilised as standard
  • Exchange 2019 CU5
  • I am not using reverse authentication proxy at this time (may set up and add notes later)

On a clean Destination NAT on HTTP/HTTPS to the Exchange server, it is a clean pass from the MRCA.

Regarding the MAPI error, I noted that the connectivity analyzer was trying to reach a site path with:

[Wed Apr 8 15:10:02.359432 2020] timestamp="1586355002" srcip="13.74.35.9" localip="x.x.x.x" user="-" method="POST" statuscode="404" reason="-" extra="-" exceptions="SkipURLHardening, SkipFormHardeningMissingToken" duration="1752" url="/mapi/emsmdb/" server="webmail.domain.blah" referer="-" cookie="-" set-cookie="-" recvbytes="659" sentbytes="4402" protocol="HTTP/1.1" ctype="text/html" uagent="Microsoft Office/15.0 (Windows NT 6.2; Microsoft Outlook 15.0.4615; Pro; MS Connectivity Analyzer)" querystring="?MailboxId=7de65457-7532-4823-aaa0-15b604ea733d@domain.blah" websocket_scheme="-" websocket_protocol="-" websocket_key="-" websocket_version="-" ruleid="6"

And on the MRCA:

  

Testing HTTP Authentication Methods for URL webmail.domain.blah/.../
The HTTP authentication test failed.

Additional Details

A Web exception occurred because an HTTP 404 - NotFound response was received from Unknown.
HTTP Response Headers:
Keep-Alive: timeout=15, max=100
Connection: Keep-Alive
Content-Length: 196
Content-Type: text/html; charset=iso-8859-1
Date: Wed, 08 Apr 2020 15:33:10 GMT
Server: Apache
Elapsed Time: 102 ms.

The following I tested in various permutations until finally all 3 elements were needed to make this work (/MAPI may not be required but I added anyway for consistency).

So I added to the webmail WAF profile in the site paths /mapi and /MAPI as site paths pointed to my Exchange, added to the exceptions with /mapi/* and /MAPI/* alongside "/owa/*,/OWA/*,/ews/*,/EWS/*,/ecp/*,/ECP/*,/oab/*,/OAB/*,/oma/*,/OMA/*,/Microsoft-Server-ActiveSync?". I also had to edit the "Exchange General" protection profile to add /mapi and /MAPI to the Static URL hardening entry urls, screenshots below:

Entry into site path routing on the rule:

Entry into the exceptions for the rule:

Static URL Hardening entries for the Exchange General:

This resolved the MAPI authentication issue but then I received the following error post this being fixed on the MRCA:

"Attempting to ping RPC proxy oa.domain.blah.
RPC Proxy can't be pinged.

Additional Details

An unexpected network-level exception was encountered. Exception details:
Message: The remote server returned an error: (405) Method Not Allowed.
Type: Microsoft.Exchange.Tools.ExRca.Extensions.MapiTransportException
Stack trace:
at Microsoft.Exchange.Tools.ExRca.Extensions.MapiRpcTestClient.PingProtocolProxy(String endpointIdentifier)
at Microsoft.Exchange.Tools.ExRca.Tests.MapiPingProxyTest.PerformTestReally()
Exception details:
Message: The remote server returned an error: (405) Method Not Allowed.
Type: System.Net.WebException
Stack trace:
at System.Net.HttpWebRequest.GetResponse()
at RpcPingLib.RpcPing.PingProxy(String internalServerFqdn, String endpoint)
at Microsoft.Exchange.Tools.ExRca.Extensions.MapiRpcTestClient.PingProtocolProxy(String endpointIdentifier)
Elapsed Time: 115 ms."

Looking in the WAF log, this is shown up:

[Wed Apr 8 14:34:10.425381 2020] timestamp="1586352850" srcip="13.74.35.9" localip="x.x.x.x" user="-" method="RPC_IN_DATA" statuscode="405" reason="-" extra="-" exceptions="-" duration="231" url="/Rpc/RpcProxy.dll" server="oa.domain.blah" referer="-" cookie="-" set-cookie="-" recvbytes="589" sentbytes="4432" protocol="HTTP/1.1" ctype="text/html" uagent="MSRPC" querystring="?7de65457-7532-4823-aaa0-15b604ea733d@domain.blah:6001" websocket_scheme="-" websocket_protocol="-" websocket_key="-" websocket_version="-" ruleid="7"

To resolve this, I modified my rule that was using the Exchange Outlook Anywhere for /rpc & /RPC and added /Rpc to Exceptions, Site Path Routing and the Static URL Hardening in the Exchange Outlook Anywhere Profile. It was very odd and I checked IIS and lo and behold all references to RPC were in /Rpc. Now, I've not deployed many Exchange installations (one before and this being the second) but have done many WAF configurations and they are all either "/rpc" or "/RPC". Screenshots below:

Rpc Site-Path Routing addition

Rpc Exceptions:

Exchange Outlook Anywhere Protection policy:

After this:

Now, I strongly suspect there is no further requirement for the old /rpc and /RPC as doing some further reading led me to these articles (available at time writing of 2020-04-08): https://cdn2.hubspot.net/hubfs/38080/Understand%20how%20MAPI%20over%20HTTP%20is%20changing%20your%20Outlook%20.pdf

The need for the MAPI in Exchange General and the adjustment to the RPC site path for the Outlook Anywhere seems to be directly related to the replacement/migration to MAPI-over-HTTP instead of RPC-over-HTTP.

I may test the removal of /rpc and /RPC at a later date but after a lot of pain, sleeping dogs and all that. I will be reporting this to Sophos Support shortly to highlight as there may need to be a new Exchange 2019 template added to the XG.

Emile