Web filtering still blocks sites that are allowed AND pass policy check

If the UTM is bridged, then "Full Transparent" should be chosen instead of "Transparent" mode as the Proxy then will not masquerade the traffic.

When you see statuscode="504", you know that the server doesn't like the Proxy.  The first thing to try is an Exception for Antivirus for the site.  If that doesn't solve the problem, the only choice is to skip the Proxy.  The same thing would happen in Standard mode, but skipping the Proxy must be done in the client's browser settings.

I do prefer Standard mode with Authentication, but also like to have that in a Profile with the Default Profile in Transparent mode.

Cheers - Bob

Hi Bob,

Thanks for the feedback and I did find much of what you said here in my earlier searches.  The one thing in your post that does stand out is that many of these issues would persist if using Standard mode - that is very disappointing as I do not see much value in having a proxy if you then need to create a bunch of exceptions for it.  I mean what's the point...Install the UTM to only proxy for a handful of sites?  Not seeing much value there - just use pfSense + MASQ-NAT...BOOM...DONE...functionally it is the same thing and it works for literally every target site.

Also, I did try exceptions (please see my original post) and they also failed to resolve the issue - and AV is not even running.  In fact, the only things NOT disabled are Web Protection and FW - everything else is turned off.  And finally, reiterating what I said originally - all of the website checks do pass the policy test only to fail miserably when actually trying to access those sites but I guess this makes sense if the host server is having some unknown issue with the proxy.

Also, why would a proxy even be getting accessed in Full Transparent mode?  I thought the main difference between Transparent and Full Transparent was the former is a transparent proxy and the latter is no proxy - correct?  If so, then why do I see literally no improvement between using Transparent and Full Transparent?

I really wanna like this device but as I said, not seeing much use for it when several bypass and exception mechanisms are required to get it to semi-operate.  If there is a use-case showing how the UTM adds value despite having to configure workarounds for the very services it is supposed to provide, I'd love to see it.

Thanks...

PS: The XG seems to be a far more functional box - at least it aspires to be.  As we know it has a whole other set of issues and flaws...hopefully those will get stomped out in 2018...

Hi Bob,

Thanks for the feedback and I did find much of what you said here in my earlier searches.  The one thing in your post that does stand out is that many of these issues would persist if using Standard mode - that is very disappointing as I do not see much value in having a proxy if you then need to create a bunch of exceptions for it.  I mean what's the point...Install the UTM to only proxy for a handful of sites?  Not seeing much value there - just use pfSense + MASQ-NAT...BOOM...DONE...functionally it is the same thing and it works for literally every target site.

Also, I did try exceptions (please see my original post) and they also failed to resolve the issue - and AV is not even running.  In fact, the only things NOT disabled are Web Protection and FW - everything else is turned off.  And finally, reiterating what I said originally - all of the website checks do pass the policy test only to fail miserably when actually trying to access those sites but I guess this makes sense if the host server is having some unknown issue with the proxy.

Also, why would a proxy even be getting accessed in Full Transparent mode?  I thought the main difference between Transparent and Full Transparent was the former is a transparent proxy and the latter is no proxy - correct?  If so, then why do I see literally no improvement between using Transparent and Full Transparent?

I really wanna like this device but as I said, not seeing much use for it when several bypass and exception mechanisms are required to get it to semi-operate.  If there is a use-case showing how the UTM adds value despite having to configure workarounds for the very services it is supposed to provide, I'd love to see it.

Thanks...

PS: The XG seems to be a far more functional box - at least it aspires to be.  As we know it has a whole other set of issues and flaws...hopefully those will get stomped out in 2018...

One other question - Recall that I have the "Do not proxy HTTPS traffic in transparent mode" setting checked - so then why would HTTPS traffic be proxied regardless of which Transparent mode is selected?

This seems to be a workaround that is just ignored...

As I tried to indicate before, your "timeout" error indicates that the reply is not being received by the web proxy, it is definitely NOT being blocked by the web proxy.  One possibility is a NAT or routing problem, so that the server never receives the packet or the reply never reaches UTM.   The other possibility is that another part of UTM blocks the reply.   I think the only other place that UTM can block the packet is IPS, and you have ruled that out.   You also said that the problem goes away if you bypass the proxy, so the server does not seem to be blocking your IP for reputation reasons.   (This might be a false assumption if you have multiple IP addresses.)

Your URL does not indicate whether the problem is with HTTP or HTTPS.   This might be significant information, because HTTPS requires that the client and the server agree on an encryption rule.   I don't think I have seen an encryption problem show up as a timeout problem, however.

On proxy types:

In Standard Mode, your browsers says, "Hey UTM, please fetch the following website for me!"  The initial packet is address to UTM on the proxy port.   UTM does the DNS lookup to determine the URL, and UTM issues the query from its IP address.    Standard Mode proxy can handle non-standard ports because the non-standard port number is in the URL.   However, it blocks non-standard ports until you specifically allow them.   You enable Standard Mode with a client configuration setting, usually a proxy script deployed using a Windows Group Policy.   You bypass the proxy by changing the proxy script (or by configuring a client-level exception if not using a proxy script.)   Standard Mode Web proxy also handles FTP traffic.

In Transparent Mode, UTM intercepts traffic as it passes through, even though the packets are not addressed to UTM.   Because it has no information from the client device, it only examines traffic heading to port 80 or 443.   Exceptions are configured with the Transparent Proxy Destination skip list.   When you configure a Transparent Destination exception, the proxy ignores the packet, which means that firewall rules check the packet, so you use firewall rules to allow or deny the packet.   If it is denied at the firewall, the host is effectively blacklisted.   If it is allowed at the firewall, it is effectively whitelisted.

Transparent Mode FTP works similarly to Transparent Mode Web, but checks packets on port 21 (only).  The two transparent proxies should be used together.

Any Standard Mode Source IP that does not match a Filter Profile Allowed Network list is handled using the Base Policy, which is configured for Block All by default, but can be changed easily.

Any Transparent Mode Source IP that does not match a Filter Profile Allowed Network list is handled by the Firewall Rules, as described above.

Any Transparent Mode traffic on a non-standard port is ignored, and handled by firewall rules only.

Both proxies have added considerations when HTTPS inspection is involved.   In general, Standard Mode is preferred.

Your log line was for HTTP, not HTTPS, so I think you're misinterpreting something else you're seeing.

The difference between Transparent and Full Transparent is only that the Proxy forwards the packet with the client's IP in Full  In simple Transparent, it forwards the packet as coming from its own IP - it masquerades the packet.

Cheers - Bob

Hi Bob & Douglas,

First - thanks to Bob for alerting me to the log message being for HTTP vs. HTTPS.  That got me to re-examine what I was seeing and yes, this appears to be an issue with just HTTP as when HTTPS is involved, things seem to flow much better and the "Do not proxy for Transparent mode" option seems to work as expected.

As to the proxy differences, you have both confirmed my understanding of what the UTM is supposed to be doing in each of the different modes - thanks to you both for that.

However this still leaves HTTP as problematic...  

I keep hearing two consistent points in the feedback from both of you as well as elsewhere in the Sophos forums:

1. Standard mode is preferred.
2. If using Transparent, then often times the Skiplist needs to be utilized.

This kinda leads me back to my original question of how is this a useful device when being used in Transparent mode.  I totally see the value if using in Standard mode.  But Transparent seem to be fraught with issues if using anything other than FW which, also as I mentioned, there are much simpler devices to accomplish that goal.

Thanks again for all of your feedback and time...

Whether in Standard or Transparent, a "50?" means that you first try an Exception for Antivirus and if that doesn't work, you must skip the Proxy.  In Transparent, that's done with the 'Skip Transparent Mode Destination Hosts/Nets' list.  In Standard, that's done in the client browser in 'LAN Settings' 'Advanced' (in IE for example).  The advantage of doing this in Standard is that you can use wildcards in the client browser - in Transparent, you must use IP addresses.  See Configuring HTTP/S proxy access with AD SSO.

Cheers - Bob

Hi Bob...

I think it's clear on what has to be done to get traffic flowing, what isn't clear is:

• If this really is an issue with target hosts "not liking" the UTM proxy, why would it be for nearly every site I attempt?  Proxies aren't a new thing so it's kinda unbelievable that most of the hosts we access would all not like the UTM.

• Adding a bunch of Skiplist entries isn't really a fix - it's a workaround.  So then what is the actual root cause - meaning, what is it about the UTM proxy that target hosts "do not like"?

• What is the value of the UTM when the very features that provide value are essentially neutered?

In any event, I took the UTM out of service and stuck with the XG.  It does work pretty well but as I mentioned earlier, it does have some significant issues so fingers are crossed for some of those getting knocked down in 2018.

Thanks.