Sophos Firewall: v19.5 GA: Feedback and experiences

Hello LHerzog,

It seems you are facing similar issue as mentioned in  HA flop on manual firmware upgrade to 19.5 .

firmware upgrade on XGS4500 from SFOS 19.0.1 MR-1 to 19.5 immediately broke MTA, all email in/out ceased. Simultaneously we received a / A hard drive has failed and requires immediate attention error.  Upgraded backup unit's firmware and restored a backup from the primary. Same results MTA broken.  Downgraded the firmware MTA resolved.  However when the unit downgraded the firmware it also reverted the configuration.  Had to also restore the configuration from a current backup again.

Case 06043592 opened with support, spent ~10 hours and went through 3 Level I's, one Level II, and one Level III. 

At this point we are running on a backup device, on v19.0.1.  Waiting to hear why MTA breaks. At this point the MTA failure is the only considered issue. The backplane was sadly just a coincidence.

The 10 hour marathon had a lot of variables that made it so time consuming. 

Thank you for your both answers. I don't believe it is a Switch issue as the firewall reported The issue also for Port1. Both firewalls have an IP on that port but it is not wired / unplugged. So it is more a SFOS software issue than something with external hardware.

We did not have a HA failover loop due to no ports monitored.

I want to add, that the Port flapping was happening for hours, started right after the upgrades and stopped exactly with the reboot of the primary node. We have HA fall back enabled (Fail back to primary device after it recovers). So the same node that had the port flapping is primary now again. The issue did not reappear during the last days.

Hi Together,(Firstpost)

facing following issue, straight after the upgrade.

Websites in Chrome/Edge with Certificate Chain Issues, while having a Exception of the type "HTTPS cerificate validation", get the error "NET::ERR_CERT_AUTHORITY_INVALID" in the Browser. 

Hi Baml,

In 19.0 additional protection was added to DPI mode, to ensure that traffic passing through the Firewall did not appear to be more trustworthy than sites accessed directly. To rephrase, if a browser would likely give a TLS or certicate warning to the user when accessed directly, we should also make sure that the browser gives a warning to the user when accessed through the firewall. In some cases we are not able to give the exact same error, but we do give an error. This feature is called Forward Certificate Invalidity.

As far as I know, there were no changes between 19.0 and 19.5.

Can you please go to one of the sites with the error (and let me know which one), and look at the certificate and certificate authority. Does the authority have an unusual/descriptive name such a "Sophos SSL Untrusted"? Does the certificate have an unusual/descriptive name like "Original cert sha1 in chain"?

Hi Michael Dunn,

first of all, i see you under close to every community discussion while searching around. I highly appreciate what you are doing for this community.

Next:

My update was from 18.5.4 to 19.5, could be some changes like you mentioned.

One Website is: www.pofm.com (Here is the ssllabs-check https://www.ssllabs.com/ssltest/analyze.html?d=www.pofm.com )

The SSLLABS check is showing following: 

About the Chain issue: Extra download is usually no problem for browsers if they access the internet directly(tested with Safari/Chrome/Edge etc.) Only Sophos sees it as a "security risk" and blocks it with  "SSL error: BLOCKED_BY_TLS_PROFILE_CERT_VALIDATION" while accessing with the Browser using the DPI-Engine. To get rid of this i have configured a "HTTPS certificate validation" under "Web -> Exceptions" for this site. With that it was possible to access the website without an error or blocking from Sophos nor Chrome.

After the Update if i try to access the website i get following form Chrome:

The "SecurityAppliance_SSL_CA" on my firewall is matching with the one i have in my trusted-root-ca-store in Windows. But the "Sophos SSL Untrusted CA_..." (You can see in the screenshot) is something different.

If i add a exclusion for SSL/TLS decryption for this website, i don´t get any warning from my Browser. The Cert looks like this:

Maybe it is like You say and the firewall is now forwarding the info about a security risk to the browser with changing the cert to a not trusted one. This is a cool thing, if the original servers cert really has a problem, even if i make a exception for "HTTPS certificate validation".  But I´m using this method to get rid of the problem sophos has with webservers needing "extra download"-ed Certificates.(To get rid of this false error)

Does somebody know a way to get around it without disabling ssl-decryption for the website?

This is indeed expected behavior of the FCI feature.  What follows is a Draft of a KB Article I'm writing (feedback welcome).

In XG 18.0 the DPI proxy was introduced, with many more SSL/TLS scanning options and certificate protection. There are some certificate security concerns with that are blocked in most configurations, however they are allowed without any warning when using DPI mode with the decryption profile "Maximum compatibility". In these cases the XG decrypts and re-signs, creating a new certificate with its own Certificate Authority that hides potential problems that the end user should know about which that would be blocked in other configurations.

In 19.0 a new feature was added called Forward Certificate Invalidity (FCI). This feature detects certain types of certificate invalidity and "forwards" (tells the user) about them. Because we cannot create a certificate with the same error, so we create one in a special way that we know will cause browsers to warn users. A CA is used that us unique and untrusted, and the certificate Common Name is used as the error message to tell the end user what the problem is. This changes the behavior of DPI mode with Maximum Compatibility. By signing it in the way, browsers will warn users that there is a certificate problem but will allow users to proceed and load the pages as they did in 18.0/18.5.

The most common issue that FCI catches on the public internet is websites that do not send their entire certificate chain. When a website provides their certificate, they usually provide the certificate, the CA that signed it (usually an intermediate CA), the CA that signed that one, up to the root CA. The root CA is trusted by the browser, and the browser can verify the entire chain.

However some websites do not provide the chain. They may provide only the certificate, or they provide the certificate and the root CA, but they do not provide the intermediate CA. While this is valid, it is not best practice and sites like ssllabs.com which rate sites will cap their score.

If a website does not provide the chain they usually implement AIA (Authority Information Access). This is a link within the certificate that says where to download the CA that signed it.

Some browsers (Chrome 58+, Edge, Safari) will automatically use the AIA to download the intermediate and store it for future use. Firefox 75+ uses a different mechanism called intermediate CA preloading (wiki.mozilla.org/.../Intermediate_Preloading). Older Android (pre Oreo) devices do not support any mechanism. The XG does not currently support AIA.

If the website does not provide the full chain, the XG behavior depends on the configuration. It is important to note that v19 did not introduce blocking of sites that do not provide the full chain. The XG has always blocked these sites when using normal security, and the resolution below has always worked. The difference is that when using a decryption profile that does not block self signed certificate, invalid issuers, or many other security concerns (such as "Maximum compatibility") we used to allow the connection and sign it in a way that made it appear more secure. Now we allow but sign it in a way that appears insecure.

How to configure the system to allow access to sites that do not provide a complete certificate chain.

Method 1 - Do not decrypt
Configure the domain to not be decrypted. Adding the domain to the Local TLS Exclusion List is the best option for DPI mode. Adding it to a Web Exception will exclude it for both DPI and Proxy mode.

Method 2 - Add the Intermediate certificate to the XG CA store

Option 1:
Test the site in www.ssllabs.com/.../
You should see that the grade is capped to B and the Certification Path includes an "Extra Download". In the section under Issuer there should be an AIA link.

Option 2:
Use a browser that is not going through the XG, or is going through the XG with HTTPS not decrypted so that you get the original certificates as presented by the site. Ask the browser to display the certificate information and the AIA link should be there. How this is displayed is browser specific.

Once you have the AIA link, download the certificate to your computer. Then in WebAdmin go to Certificates > Certificate Authorities and Add. Choose the file.
With the Intermediate in the CA store, the page will load in all configurations.

Hi Franz-Josef Heidkamp ,

Good day and thanks for reaching out to Sophos Community.

The KB Link you shared is already updated and proposed a resolution already for the specific matter

Resolution

• The issue has been resolved, and the fix will be deployed automatically in Sophos Central. The firmware upgrade will be available only after the staging phase is completed. 

• In the meantime, you can manually download SFOS v19.5 from the Licensing Portal and update individual firewalls manually. 

Thanks for your time and patience and thank you for choosing Sophos.

Cheers,

Hi Michael Dunn,

Thanks for your Reply. I´m using now Method2 Option1 and it works fine. Because the Certpath(for the XG) is now complete, I don´t need to add those websites to my "HTTPS certificate validation"-Exception anymore.

Feedback:

The KB is good. My suggestion: Method 1 shouldn´t be highlighted as "best option", disabling decryption leads to a higher securityrisk.

Hello,

Checking internally in the case, I can see now the case is with DEV under NC-113038, and they’ll be working on an RCA.

Regards,