HTTPS Inspection problem requires a tricky regular expression

I am using https inspection.   The Sophos-supplied configuration excludes https inspection for everything on microsoft.com.   I am having a problem with pages from support.microsoft.com not loading correctly.   On investigation, I determined that the missing content is being retrieved from akamaized.com, but the website refuses the connection.   If I turn off https inspection, the connection is accepted and the page loads correctly.

My inference is the following:

  • support.microsoft.com bypasses https inspection
  • *.akamaized.com does not bypass https inspection, so it looks to the server like a primary query rather than a referral
  • The server is secured so that it only accepts content when the referral comes from an authorized source, such as support.microsoft.com

The obvious workaround options:

  • Always reference both sites with https inspection.   This requires overriding the Sophos-supplied exception
  • Always reference both sites without https inspection.

I assume that akamaized.com serves many clients, so if I disable https inspection for it, the problem will reappear in reverse somewhere else -  the referring site has inspection on and the akamaized site has inspection off, so the connection is refused.     This is likely to become an endless chain that eventually makes https inspection impossible.

Therefore, the best solution seems to be to change the Sophos-suppied Regex to apply to microsoft.com except when it is support.microsoft.com    I know this is theoretically possible with RegEx, but is beyond my experience.   The Sophos KB article about regex syntax does not address this type of construct, so I don't know for sure if the chosen syntax will work with their implementation.

Hoping that someone knows the regex command and knows whether the Sophos implementation supports it.

  • Try this.

    ^https?://(?!.*(support)).*\.?microsoft\.com/

    This should include every subdomain of microsoft.com, excluding support.microsoft.com.  

    I tested this and you'll see https://microsoft.com and https://test.microsoft.com get caught by this exception, while https://support.microsoft.com does not.

    timhansen:/root # tail -f /var/log/http.log | grep https://microsoft.com
    2018:12:08-16:09:50 timhansen httpproxy[30155]: id="0001" severity="info" sys="SecureWeb" sub="http" name="http access" action="pass" method="CONNECT" srcip="192.168.2.10" dstip="13.77.161.179" user="" group="" ad_domain="" statuscode="200" cached="0" profile="REF_DefaultHTTPProfile (Default Web Filter Profile)" filteraction="REF_DefaultHTTPCFFAction (Default content filter action)" size="7618" request="0xdb67e600" url="https://microsoft.com/" referer="" error="" authtime="0" dnstime="668" cattime="50043" avscantime="0" fullreqtime="5391720" device="0" auth="0" ua="" exceptions="av,sandbox,ssl,fileextension,size" category="105" reputation="trusted" categoryname="Business" country="United States"

    timhansen:/root # tail -f /var/log/http.log | grep https://test.microsoft.com
    2018:12:08-16:10:16 timhansen httpproxy[30155]: id="0001" severity="info" sys="SecureWeb" sub="http" name="http access" action="pass" method="CONNECT" srcip="192.168.2.10" dstip="52.161.161.117" user="" group="" ad_domain="" statuscode="200" cached="0" profile="REF_DefaultHTTPProfile (Default Web Filter Profile)" filteraction="REF_DefaultHTTPCFFAction (Default content filter action)" size="8277" request="0xdc3d5200" url="https://test.microsoft.com/" referer="" error="" authtime="0" dnstime="6" cattime="287" avscantime="0" fullreqtime="216389" device="0" auth="0" ua="" exceptions="av,sandbox,ssl,fileextension,size" category="105" reputation="trusted" categoryname="Business" country="United States"

    2018:12:08-16:11:06 timhansen httpproxy[30155]: id="0001" severity="info" sys="SecureWeb" sub="http" name="http access" action="pass" method="CONNECT" srcip="192.168.2.10" dstip="184.29.84.173" user="" group="" ad_domain="" statuscode="200" cached="0" profile="REF_DefaultHTTPProfile (Default Web Filter Profile)" filteraction="REF_DefaultHTTPCFFAction (Default content filter action)" size="568" request="0xdc4ca000" url="support.microsoft.com/" referer="" error="" authtime="0" dnstime="10" cattime="296" avscantime="0" fullreqtime="53703" device="0" auth="0" ua="" exceptions="" category="105" reputation="trusted" categoryname="Business" country="United States" application="micrsoft" app-id="1151"

    Tim

  • In reply to TimHansen:

    Sorry this will exclude the word 'support' anywhere in the subdomain, eg. mysupport.microsoft.com would be excluded too.

    You can probably just get rid of the leading and trailing .*'s to just exclude support.microsoft.com.

    ^https?://(?!support)\.?microsoft\.com/   

    I didn't test the above but I wouldn't see why it wouldn't work.  

    Tim

  • In reply to TimHansen:

    Thanks.    I will squirrel this away, because I have not done much with negation in regular expressions.

    After posting this, I realized that I had not really tested my theory about the cause of the problem and its resolution.   Yesterday I had the same problem with splunk.com. 

    It actually appears that these sites are blocking me because I am running an outdated version of OpenSSL (UTM v9.506 = OpenSSL 1.0.2j-fips).   At home, I downloaded the latest version of OpenSSL and it connected to Splunk easily, while an old version did not.   I plan to do some more testing with intermediate versions.

    Anybody willing to do OPENSSL VERSION from the shell prompt on UTM 9.6?

  • In reply to DouglasFoster:

    I haven't upgrading my home appliance to 9.6 yet.  I would do it now but I'd risk an open revolt by the children because their show on the t.v. freezes for 2 minutes.  :)

    Tim