XG v18 SSL/TLS inspection interfering with Veeam Cloud Provider Replication

Hi,

try turning the on the web proxy and then create exceptions for the site.

What restrictions do you have in your firewall rule the VEEAM application or traffic?

Ian

Hmm, yeah I guess, but I think my ultimate goal would be to use the new DPI engine (and not have it break things ;) ). The firewall rule just looks at the port number, not the application.

Hi Bjoern,

because the backup is a streaming connection, DPI will not be able to touch it and you will need an exception in the DPI setup.

Ian

Bjoern,

Please share the firewall rule used. While the issue is occurring, please take a packet capture on XG.

Decrypting backups don’t not make sense and I understand how even slower the backup can go if decryption occurs (when it works!).

Regards

The rule is very basic, just source IPs, services (using the TCP port), nothing else.

This is strange as DPI does not scan traffic from WAN to LAN. Check IPS if it is blocking something.

Oh, really? Wasn't aware of that. But somehow it's doing something.

I checked all the logs and there was absolutely nothing being denied etc. from the source where I'm having the issue :( That's why it lead me to try turning off the SSL/TLS inspection as it was the last resort of what it could be. I noticed I was rejecting SSL compression and SSL 2.0 and 3.0. I doubt that Veeam would use those, but I set them to allow and not decrypt and turned inspection back on. We'll see if that makes a difference. I don't expect it to.

So you access via DNAT your Veeam Infrastructur? 

Or does your Veeam build up the Connection?

In Log Viewer - TLS, you do not see matching connections of this Client?

Could you show your TLS Rules? 

Yes the remote site establishes the connection to the replication target at the main site. I don't see anything logged under TLS for the remote site IP address, which makes sense if inspection only happens on outbound connections.

I don't have any TLS rules as I'm not doing any decryption currently.

Still leaves the question why turning off TLS inspection fixes the connectivity issue if it's not even scanning inbound connections.

There is an Issue still open in XG, which maybe could be also causing this issue.

Michael Dunn Could this be the same as the Camera Issue?

Do you see any packets with source IP of the Veeam Solution? 

Also + Advice to open a Case to get this logged. 

Related to this topic, I wanted to mention the great post that Michael published:

• Sophos XG Firewall v18: Troubleshooting problems with the DPI engine

Bjoern Freiherr I'd suggest to take a look here as well before raising a case.

If you do end up raising one, please share the number with me.

Thanks,

In the firewall logs I do, but not in the SSL/TLS logs.

I'll call a ticket in next week when I have some time. I can also pretty easily test this so traffic can be captured.

Thanks for your help so far!

In the firewall logs I do, but not in the SSL/TLS logs.

I'll call a ticket in next week when I have some time. I can also pretty easily test this so traffic can be captured.

Thanks for your help so far!

Perfect! Please let us know.

As I said, DPI should not be responsible for this traffic unless it is a bug.

Regards

Hey guys, I opened a case: #9744383

We've just been testing this with multiple different combinations of settings and nothing but disable SSL/TLS has worked so far. Even when the DNAT firewall rule is set to use the Proxy rather than DPI engine it still doesn't work.

We upgraded to the latest release of v18 also.

Hi,

please upgrade to the v18 SR2 which will enable you to disable SSL/TLS on your specific rules and hopefully overcome your issue.

Ian

We upgraded to build 339 if that's what you mean, but I don't see an option to disable SSL/TLS inspection for a firewall rule. There's an on/off toggle for the inspection which is global which solves our problem, but that's not the point because I'm disabling the entire feature that's being promoted in V18. Making a SSL/TLS rule with don't decrypt is not applicable here because the WAN zone can't be selected as the source and we're not even decrypting anything either.

To me it looks like a severe bug that's interfering with traffic that shouldn't be inspected. It's being escalated to GES now after troubleshooting and testing for 4h straight.

Hi,

my apologies, you are correct. I need to go back and review the thread on the subject to find out why I misunderstood the intent of that switch.

Ian

No worries. I'll post an update once I hear more from the escalations team.

I have followed up with the response from Michael Dunn in the thread where the feature is questioned as well.

A bit disappointing because that was an issue I raised doing EAP about not being able to create simple firewall rule as was the case in v17.5.8

Ian

The DPI engine is always looking and inspecting at traffic, even if you don't want it to. It is a very poor implementation IMHO.

So, you can disable the DPI completely and continue to use the Proxy or wait until Sophos get's a clue and realizes they need to make a change.

Many issues have come up because of their decisions. Hopefully they change their awful approach to DPI.

What most don't realize is that XG cannot meet stated performance metrics because of how Sophos chose to implement DPI. I questioned the performance of XG because of the change but, did not get a response. The XG cannot meet the performance metrics they quote in the current V18 config. As an example, Sophos states 16,000 mbps for the XG210. It cannot come close to that on v18 because the DPI engine is looking at all traffic.

If this is production, I would downgrade back to v17.5 if I were you.

MichaelBolton said:

The DPI engine is always looking and inspecting at traffic, even if you don't want it to. It is a very poor implementation IMHO.

Actually the "DPI engine" has always looked at all traffic since XG's very beginning.

What is new in XG v18 is that the DPI engine is doing more than it used to.

 - detecting HTTP traffic on any port

 - detecting TLS on any port

    - enforcing TLS policy on any port

    - potentially decrypting TLS

 - enforcing Web policy on HTTP and HTTPS on any port (not just 80/443)

When the XG is detecting HTTP and TLS on any port and enforcing TLS policy on any port it is slower.  When it is offloading connections (fastpath), decrypting TLS and enforcing using the DPI engine it is significantly faster.  The speed improvements that are made more than make up for the areas where speed is reduced.  In affect we are doing more (which slows things down) but we are doing it faster (which speeds things up) resulting in a net faster speed.

MichaelBolton said:

So, you can disable the DPI completely and continue to use the Proxy or wait until Sophos get's a clue and realizes they need to make a change.

Clue received and GA Soft Release 2 contains a switch that disables TLS enforcement unless the firewall rule contains a web policy.  Please see this post:

https://community.sophos.com/products/xg-firewall/b/blog/posts/xg-firewall-v18-ga_2d00_build339-is-now-available

One of the side effects of this is that customers who upgrade (and therefore TLS inspection is Off) or who turn it Off should get similar performance between 17.5 and 18.0.

Customers who configure their boxes to take advantage of v18.0 DPI mode and fastpath should see much greater performance.

MichaelBolton said:

Many issues have come up because of their decisions. Hopefully they change their awful approach to DPI.

Many issues are fixed in GA, GA SR2, and will be fixed in MR1.  Our approach is to fix issues in DPI, not bypass DPI.

MichaelBolton said:

What most don't realize is that XG cannot meet stated performance metrics because of how Sophos chose to implement DPI. I questioned the performance of XG because of the change but, did not get a response. The XG cannot meet the performance metrics they quote in the current V18 config. As an example, Sophos states 16,000 mbps for the XG210. It cannot come close to that on v18 because the DPI engine is looking at all traffic.

I agree that performance metrics are hard to state because lab test environment don't match real world conditions.  In reality in my personal opinion, published metrics (of any vendor/product) are most useful for comparing two models, not for seeing the real world usage.  I would stand behind whatever the Sophos published metrics are for comparing 17.5 and 18.0 for correctly configured systems.  In other words if Sophos says model xyz does 1,000 foobar in 17.5 and 1,200 foobar in 18.0 I believe that v18.0 will be 20% faster in foobar.

Whether you believe that the foobar numbers are achievable or not based on your understanding of DPI is immaterial.  Internal and independent testing is proving otherwise.

MichaelBolton said:

If this is production, I would downgrade back to v17.5 if I were you.

Thanks for the additional insight. I'm in the same boat and think that good improvements have been made and I'm hoping for all these bugs to get ironed out.

I'll report back after working further with GES and hopefully can determine the root cause of this issue.