Web Policy and Filtering Not Working at All

XG V16 - It seems yet another thing real simple in other firewalls just doesn't want to work.  I'm not sure if the KB article I found isn't complete, but if I have the default web filtering policy or Default Workplace Policy applied on the only LAN-to-WAN network rule, nothing gets blocked, nor does anything show up in the log viewer.  Also, while I can see the value of doing it on a rule basis, is there a way to just filtering on a zone like with other firewalls?

  • In reply to Michael Dunn:

    Thanks Michael for perfectly explaining things. This is the kind of interaction that I miss from astaro days. The devs/mods didn't just show you a workaround but the logic and thinking that went behind a decision to do things a certain way. Most people get caught up in the basic gui differences between UTM9 and XG and I don't blame them but by doing so they don't give the true gems like firewall policies a fair chance. I was not a fan of v15 but gave v16 a serious chance after reading 's praises of XG. Although I still miss certain things from UTM9, after using XG v16 exclusively for a few months, I realized that UTM9 is indeed showing its age.

    Although you mostly deal with the webfiltering aspect, I am hoping you guys can inspire other teams with your forward thinking into other areas of XG. If we keep getting the XG firmwares at the pace we have been getting, I am very optimistic and can't wait to test what v17 will bring.

    Thanks again and happy holidays


  • In reply to Billybob:

    v16.5 will bring in Sandstorm for web and email.

    v17.0 I can't give you any hints.  :)

  • In reply to Michael Dunn:

    Michael Dunn

    If it helps, you can think of it a different way.  An incoming packet can only go to one Web Policy.  So you cannot define two Web Policies and have them both apply to that connection.  You need to have one Web Policy.  Then you have the freedom of doing user selection within the Web Policy or the Firewall Rule.

    Sorry to bump such an old thread but this thread has some great info on how policies work. If I'm understanding this correctly, it really boils down to the statement quoted above but I'm still a bit confused as to the purpose behind "Allow All" or "None" and why policies are applied to individual firewall rules. Here's an example I created to illustrate what I'm confused about:

    I created two firewall rules that apply to the same computer, we'll call it Firewall Rule 1 and Firewall Rule 2.

    Firewall Rule 1 - Applies to Computer X - Allows all traffic - Web Policy set to 'None'

    Firewall Rule 2- Applies to Computer X (same computer) - Allows all traffic - Web Policy set to 'Block Shopping Websites'

    In this example, Firewall Rule 1 is applied when accessing the internet, therefore Firewall Rule 2 will never be applied and I can still access shopping related websites. However, I thought the purpose of setting a policy to 'None' is so that it can still be "eligible" for a web policy to apply, but how is that possible if it will never be assessed against another firewall rule?

  • In reply to shred:

    There are two other factors you need to consider - the "Services" that the rule applies (which is really the port numbers), and the Malware scan HTTP and scan HTTPS setting.

    Think of it as two sections - an "The rule applies to traffic matching criteria" section and an "enforcement" section.


    Assuming that Firewall Rule 1 applies to the HTTP and HTTPS service (or TCP or Any), then Rule 1 would apply to HTTP traffic.

    The Firewall looks at both the Malware setting and the Web Policy.  Assuming that Malware scan was on for HTTP and Web Policy was None, then it would go through the web proxy in order to do antivirus scanning but the web proxy would not enforce any policy (eg around website categories or filetypes).  If Malware Scan was off, then it would not go though the web proxy at all and would just be allowed through the firewall.


    Once the matching part is done to select a firewall rule, then the firewall decides "Does this need to be sent to the web proxy".  If the malware scan is on, then it is sent to proxy.  If the Web Policy is not "None" then it is sent to Proxy.  If you want to create a rule that passes the traffic with no interference from the proxy, then you turn malware scan off and Web Policy None.


    Web Policy "None" (which is part of enforcement) does not cause it to fall through to another firewall rule (which would be a part of traffic matching).  


    Pretend for example that None did allow a fall through from Rule 1 to Rule 2.  Which rule's Malware setting would apply?  Would rule's Application policy?


    If you do want to create a "Firewall Rule 1 - Applies to Computer X - Allows all traffic - Web Policy set to 'None'"  it needs to not apply to the service HTTP/HTTPS (eg not really Allows all traffic).  Or if you really need it to be the Service Any then you need a higher priority rule for HTTP/HTTPS.

  • In reply to Michael Dunn:

    Michael Dunn, thanks for taking the time to respond. I think I'm understanding everything you're saying, but it seems to further support the fact that there seems to be no practical purpose for setting a Web Policy to "None" versus "Allow All". The only difference it would make (assuming Malware scanning is off) is the fact that traffic would pass through the web proxy if it's set to "Allow All" but nothing would be blocked... meaning, there is no reason to NOT set it to "None" - why have traffic pass through the web proxy for no reason?

    If I'm understanding you correctly, Sophos XG determines which Firewall Rule traffic will apply to based on "Source Zone", "Source Networks and Devices", "Destination Zone", "Destination Networks" and "Services". If traffic matches based on these variables, then whatever falls under that Firewall Rule will apply, to include the "enforcement" section as you mentioned (malware scanning, policies, etc.). If the traffic does NOT match based on these variables, then nothing in that Firewall Rule will apply.

    So again, I'm still not understanding why "Allow All" versus "None" would ever be used.

    As a side note, would this also mean you need to have the appropriate "Services" selected for the "enforcement" you're trying to utilize? For example, if I want to Scan HTTP for malware or utilize a web policy, "Services" would have to be set to either "Any" or "HTTP". I'm assuming the answer to this is yes (just seems to make sense logically as a different service/port would be a different connection even if it's from the same IP address).

    Thanks again for your time. I'm just trying to understand how Sophos XG works and not actually having any specific issues with my setup so I hope I'm not wasting your time!

  • In reply to shred:

    None can be used for a few things.  One of the basic ones is to create an HTTP/S rule that bypasses the proxy.
    When traffic matches and goes through the proxy with an Allow All policy, it get sent to the proxy for processing.  The proxy validates that the HTTP headers are correct.  It (optionally) enforces Pharming Protection.  It does additional checking for application.  It logs.  It...  does stuff.  And sometimes there is oddball traffic that just does not like going through a proxy.
    Example 1)
    I know that we have an old internal KVM switch that hosts a webpage that uses a Java applet to remotely show desktops.  And it just hates proxies (IIRC it uses wacky headers).  If you go through the firewall, through proxy to it with Allow All, it breaks.
    So you create a rule that is above your normal rules, destination IP is the KVM, applies to HTTP traffic, Web Policy None.  Now the traffic goes through the XG without ever going through the web proxy.
    Example 2)
    What if you purchase a product that you just want to use a Firewall.  You don't want to use it as a Web Proxy - maybe you've got another one you use.  So you want to put HTTP traffic through the XG without it ever touching the XG's Web Proxy.  Use a firewall HTTP rule with None.
    Example 3)
    Outlook 365.  I hate that product for the headaches it causes.  The way it works is it first tries to connect to one server using HTTPS.  If it fails to make an SSL connection, it tries the next server, and so on.  Makes sense right?  But when you go through the XG we have "friendly error messages", web pages that we generate the describe the error.  So when Outlook 365 tries to go to the first server, it succeeds on the SSL connection and we send a webpage saying "the server you are trying to connect to does not respond".  Outlook then halts.  It does not see that as a failed connection and does not go to the next server.  This occurs on "Allow All" and does not on "None".
    Creating a rule for that destination with None allows the raw connection attempt to be made, and fail, so that Outlook then tries the next server.  (Note we changed things in v17 so that we no longer present certain error pages in transparent mode HTTPS when Decrypt is off in order to fix this rather than doing a None rule)
    Basically Allow All still proxies the traffic, does logging and other stuff.  None is a true "don't touch this traffic, don't log it, just allow it and pretend you don't exist".
    If you are familiar with the UTM and the "transparent mode skiplist", a firewall rule with None is basically the same thing.
    I'm not 100% sure on what you mean in your side note, but yes.  Every TCP connection made to a different destination IP and port is treated as a separate connection and the firewall rule is chosen independently.  In this context a "Service" is a port.  See Hosts and Services | Services.
    Often when I do one of these lengthy explanation posts, I see that several people "like" it.  That tells me that people are reading and appreciating them.  KB articles and such are not always a good way of transmitting this information.
  • In reply to Michael Dunn:

    If I’m understanding this correctly, using “Allow All” passes traffic through the web proxy which might be required to allow other features to work that utilize the web proxy such as Pharming Protection or validating HTTP headers. For example,  if I wanted to enable Pharming Protection but I don’t necessarily want to enable “Scan HTTP” (malware scanning), then I just set the web policy to “Allow All” so traffic goes through the web proxy so Pharming Protection can work.

    Personally, I don’t think this is very intuitive in Sophos XG (for me at least). For example, Pharming Protection is enabled in the general settings under the ‘Web’ section. It’s enabled by default but if I didn’t have a firewall that routes traffic through the web proxy, then Pharming Protection isn’t actually doing anything but there’s nothing telling me that. Additionally, it’s not easy to ascertain the difference between “None” and “Allow All” without having had read this thread. The help files don’t mention any of this. Even changing “None” to something like “None (Bypass Web Proxy)” is a little more clear in my mind.

    Anyways, thanks again for explaining everything!