This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

Web Proxy rules - lots of them?

Just wanted to get a heads up here to ensure I'm heading in the right direction.

With firewall rules and default drops, it's fairly easy to say via IP:
HOST A allowed to HOST B
HOST C allowed to HOST D

So in the above instance, the default drop would prevent HOST C communicating with HOST A etc etc.

Now as we move away from the firewall and more towards the web proxy, doing the same would require the same number of rules:
HOST A allowed to HOST B
HOST C allowed to HOST D

Now, imagine if you have 80 servers that are only allowed to access certain sites.

Would you create a GLOBAL site rule ie all servers can talk to xyz sites?

Like so:

Rule 1: SERVER A can talk to ABC sites
Rule 2: SERVER B can talk to DEF sites
Rule 3: GLOBAL Allow sites
Bottom rule: Default DROP rule



This thread was automatically locked due to age.
  • Web filtering requires a different mindset than firewall rules.  I have written at some length on this elsewhere, but it is worth reviewing, especially since you appear to be implementing a highly controlled configuration.

    With firewall rules, source and destination are evaluated together.   With Web Proxy, the source and destination are evaluated at different stages of the process, which can lead to unexpected results if you are not careful.  Here is a rough summary of the processing stages:

    • The Source IP and Proxy Mode determine the Filter Profile and Authentication Method.
    • The Filter Profile and Authentication Results determine the Policy.
    • The Policy determines the Filter Action
    • The Filter Action and the URL determine whether the request is allowed or blocked.

    Secondly, the destination is evaluated using the URL, not the IP address to which the URL translates.   This is a reasonable design.  One IP address can host many different websites, with different risk characteristics for each site.    Similarly, one DNS host name can be implemented on many different IP addresses.   In some cases, different paths within a website can have different category assignments, and most UTM customers are filtering based on categories.  If you want to block a site completely, you need to ensure that you have blocked both the IP address(es) and the host name(s).

    You may be intending to use only Transparent Mode, but every Transparent Mode Filter Profile also acts as a Standard Mode profile.   So for maximum defense, you want to ensure that your defenses will work even if a hostile user successfully enables Standard Mode on his client device.   Standard Mode has these major differences from Transparent Mode:  

    1. In Standard Mode, the DNS lookup occurs on UTM, not at the client.   In some configurations, this can produce a different result.   For example, a DMZ device may be unable to query an internal DNS server to obtain the address of an internal web server, and even if the address is known, it may not have a path to the internal web server.   In Standard Mode, UTM performs the DNS lookup, which may include results from an internal web server.
       
    2. In Standard Mode, UTM relays the packet, so webfilter may create a path from DMZ to Internal where Firewall Rules would block the same attempt.  Your Filter Actions need to block both IP and Host name for complete protection.   These blocks can be configured on the Websites tab of the Filter Action.  

    3. In Standard Mode, UTM detects all traffic from web browsers, regardless of the port number, for protocols HTTP, HTTPS, and FTP.   URLs with a custom protocol indicator will be blocked.   Non-standard ports will be blocked by default, but you can allow the ones that you need using the Allowed Target Services list.   In Transparent Mode, any traffic on non-standard ports is ignored by the proxy, causing the packet to be processed by firewall rules only.   Additionally, Transparent Mode has one proxy for HTTP/HTTPS, and a different proxy for Transparent FTP.   Standard Mode FTP is useless and will create problems for web browsers, so it should never be enabled.

    Web Proxy should never have a Filter Profile with an Allow Networks list that includes the Internet (ANY, ANY IPv4, etc.)   Web Proxy with semi-trusted sources, such as Site-to-site VPN, Client VPN, or DMZ, always requires special care.

    You must block UDP 443 with a firewall rule.   If not, Chrome's QUIC protocol will bypass your web filtering and pass through as unfiltered firewall traffic.

    There are very few reasons to use Regular Expressions.   Regex creates risk because it is too easy to create rules with unwanted allows or unwanted blocks.  Even if you get it right, your successor will not.   Most exceptions are based on FQDN or Domain name, not the full URL path.   As long as this is the case, it is much safer to create a Website object for the domain (with "include subdomains" checked") or the host name (with "include subdomains" unchecked.)   Assign a tag to the website, making up a name for the tag which documents the reason.    For example, I have one for "Allow program downloads" and another for "Web Proxy Bypass".   The assign the tag to an Exception Object or to the Websites section of a Filter Action.

    Most business traffic uses HTTPS.   If you have decrypt-and-scan disabled, you cannot filter on the path.  Only the FQDN is unencrypted.    Similarly, NTLM information is encrypted with HTTPS, so UTM guesses that the HTTPS user is the same as the last user seen on an HTTP packet.   If no HTTP traffic has occurred, the packet is handled as unauthenticated.   For this reason, I do not recommend blocking unauthenticated traffic, although I have suggested some alternatives in a forum topic from about a month ago.

    HTTPS inspection (decrypt-and-scan) requires really good log analysis tools, a high level of expertise to understand the logs, and a lot of network administration labor to deal with the problems.   I currently recommend against its use, even though I worry about the potential for hostile content delivered via an HTTPS session.  The risk for HTTPS attacks has almost certianly increased because of LetsEncypt.

    Do not underestimate the amount of non-browser traffic that uses HTTP or HTTPS.   Monitor your logs to prove me right or wrong.    Non-browser traffic will be intercepted by the Transparent Mode proxy but will not provide NTLM information.   This is another reason why you should probably not block unauthenticated traffic.

    I recommend against using SkipLists, which are based on IP addresses only.   I have an exception object named "Bypass All" that disables all of the proxy checks, which applies to any destination with the "Web Proxy Bypass" tag.   This approach has eliminated the need for modifying the Standard Mode proxy script or the Transparent Proxy Skip List.

    Because Standard Mode intercepts more ports, I recommend using both proxy modes.  I use Standard Mode to filter browser traffic, and Transparent Mode to filter non-browser traffic (or anything else that deliberately attempts to evade the Standard Mode proxy.)

    Hope this helps you get started safely.

  • That's a very good, clear and detailed post. Many thanks.

    I'm currently testing decrypt and scan and it does appear to be working well. As we have 2 egress points in our network, we've had to distribute 2 UTM's CA's via GPO's as we use a WPAD file for balancing/failover. It appears to be working well.

    The next step is to further regulate about 80 servers although a global rule will take care of most of them and about 10 of them will be different.

    I'm thinking the global rule will site just above the base deny rule and each profile for each server will sit above that ie it works from top down?

    Now, it looks to me as thought this could get a little crowded in there by the time we take our user lan's eg data, voice etc into the equation so I was wondering if others who have larger networks have done the same eg my UTM at home has 3 or 4 filter profiles but it looks that I may end up with somewhere between 10 and 20 here.

  • Things I have seen with decrypt-and-scan.

    • Some websites do not offer a ciphersuite compatible with UTM.   www.splunk.com is one example.  You have to bypass decrypt and scan for these sites.   The browsers do not h have this problem.

    • Some websites do not include the proper intermediate certificate.   The browsers use a gimmick called AIA to find and fetch the missing intermediate certificate, but UTM does not.   The workaround is to connect to the website without https inspection, check the certificate, and export the intermediate certificate from the certificate chain.  Then import it into UTM as a root certificate.   This becomes a risk factor only if the intermediate certificate is subsequently revoked.  After you build a collection of the most commonly used intermediate certificates, this problem goes down.

    • Many cloud sites have an https port open even though the site is not intended for https, and the offered certificate is a generic one provided by the hosting service.   Perhaps the search engines favor https results.  For this reason or some other, the user finds the https reference instead of http.  A browser will display a certificate warning and let the user proceed.  UTM will block the invalid certificate.   This is good or bad depending on your perspective.

    • US Military uses certificates that link to a DoD root which is not in any vendor's default certificate store, so they will fail certificate verification.   The root certificate bundle is available for download, but it changes with some regularity.  If you have employees who also have a military connection, this may cause some support calls.  As with missing intermediate certificates, the browsers return a warning and let the user proceed, while UTM throws an error.

    Yes, filter profiles are evaluated from top-to-bottom.   The first match wins and will determine the final result.   Subsequent filter profiles will never be considered.    Standard-mode profiles should appear before Transparent Mode ones, since Transparent Mode really implement both.    A filter profile for a few source IP addresses should appear above ones for a subnet, so that the subnet definition cannot override the more specific list.  

    Allowed Network entries can use Host, DNS Host, or Network objects, but not Network Range objects.  You can simulate any network range with multiple Network objects, but it is less convenient.

    I have a pretty large network but I do not have significant selection based on source IP, so I have only a few Filter Profiles.   Most of my configuration is based on group membership that triggers different features.  Marketing wants to see their own ads, so they need Web Ads and Social Networking allowed.   HR needs Job Search allowed.  Etc.