Web Proxy rules - lots of them?

Question

Just wanted to get a heads up here to ensure I'm heading in the right direction. 
 With firewall rules and default drops, it's fairly easy to say via IP: HOST A allowed to HOST B HOST C allowed to HOST D 
 So in the above instance, the default drop would prevent HOST C communicating with HOST A etc etc. 
 Now as we move away from the firewall and more towards the web proxy, doing the same would require the same number of rules: HOST A allowed to HOST B HOST C allowed to HOST D 
 Now, imagine if you have 80 servers that are only allowed to access certain sites. 
 Would you create a GLOBAL site rule ie all servers can talk to xyz sites? 
 Like so: 
 Rule 1: SERVER A can talk to ABC sites Rule 2: SERVER B can talk to DEF sites Rule 3: GLOBAL Allow sites Bottom rule: Default DROP rule

DouglasFoster · Answer

Web filtering requires a different mindset than firewall rules. I have written at some length on this elsewhere, but it is worth reviewing, especially since you appear to be implementing a highly controlled configuration. 
 With firewall rules, source and destination are evaluated together. With Web Proxy, the source and destination are evaluated at different stages of the process, which can lead to unexpected results if you are not careful. Here is a rough summary of the processing stages: 
 
 The Source IP and Proxy Mode determine the Filter Profile and Authentication Method. 
 The Filter Profile and Authentication Results determine the Policy. 
 The Policy determines the Filter Action 
 The Filter Action and the URL determine whether the request is allowed or blocked. 
 
 Secondly, the destination is evaluated using the URL, not the IP address to which the URL translates. This is a reasonable design. One IP address can host many different websites, with different risk characteristics for each site. Similarly, one DNS host name can be implemented on many different IP addresses. In some cases, different paths within a website can have different category assignments, and most UTM customers are filtering based on categories. If you want to block a site completely, you need to ensure that you have blocked both the IP address(es) and the host name(s). 
 You may be intending to use only Transparent Mode, but every Transparent Mode Filter Profile also acts as a Standard Mode profile. So for maximum defense, you want to ensure that your defenses will work even if a hostile user successfully enables Standard Mode on his client device. Standard Mode has these major differences from Transparent Mode: 
 
 In Standard Mode, the DNS lookup occurs on UTM, not at the client. In some configurations, this can produce a different result. For example, a DMZ device may be unable to query an internal DNS server to obtain the address of an internal web server, and even if the address is known, it may not have a path to the internal web server. In Standard Mode, UTM performs the DNS lookup, which may include results from an internal web server. 
 In Standard Mode, UTM relays the packet, so webfilter may create a path from DMZ to Internal where Firewall Rules would block the same attempt. Your Filter Actions need to block both IP and Host name for complete protection. These blocks can be configured on the Websites tab of the Filter Action. 
 In Standard Mode, UTM detects all traffic from web browsers, regardless of the port number, for protocols HTTP, HTTPS, and FTP. URLs with a custom protocol indicator will be blocked. Non-standard ports will be blocked by default, but you can allow the ones that you need using the Allowed Target Services list. In Transparent Mode, any traffic on non-standard ports is ignored by the proxy, causing the packet to be processed by firewall rules only. Additionally, Transparent Mode has one proxy for HTTP/HTTPS, and a different proxy for Transparent FTP. Standard Mode FTP is useless and will create problems for web browsers, so it should never be enabled. 
 
 Web Proxy should never have a Filter Profile with an Allow Networks list that includes the Internet (ANY, ANY IPv4, etc.) Web Proxy with semi-trusted sources, such as Site-to-site VPN, Client VPN, or DMZ, always requires special care. 
 You must block UDP 443 with a firewall rule. If not, Chrome's QUIC protocol will bypass your web filtering and pass through as unfiltered firewall traffic. 
 There are very few reasons to use Regular Expressions. Regex creates risk because it is too easy to create rules with unwanted allows or unwanted blocks. Even if you get it right, your successor will not. Most exceptions are based on FQDN or Domain name, not the full URL path. As long as this is the case, it is much safer to create a Website object for the domain (with "include subdomains" checked") or the host name (with "include subdomains" unchecked.) Assign a tag to the website, making up a name for the tag which documents the reason. For example, I have one for "Allow program downloads" and another for "Web Proxy Bypass". The assign the tag to an Exception Object or to the Websites section of a Filter Action. 
 Most business traffic uses HTTPS. If you have decrypt-and-scan disabled, you cannot filter on the path. Only the FQDN is unencrypted. Similarly, NTLM information is encrypted with HTTPS, so UTM guesses that the HTTPS user is the same as the last user seen on an HTTP packet. If no HTTP traffic has occurred, the packet is handled as unauthenticated. For this reason, I do not recommend blocking unauthenticated traffic, although I have suggested some alternatives in a forum topic from about a month ago. 
 HTTPS inspection (decrypt-and-scan) requires really good log analysis tools, a high level of expertise to understand the logs, and a lot of network administration labor to deal with the problems. I currently recommend against its use, even though I worry about the potential for hostile content delivered via an HTTPS session. The risk for HTTPS attacks has almost certianly increased because of LetsEncypt. 
 Do not underestimate the amount of non-browser traffic that uses HTTP or HTTPS. Monitor your logs to prove me right or wrong. Non-browser traffic will be intercepted by the Transparent Mode proxy but will not provide NTLM information. This is another reason why you should probably not block unauthenticated traffic. 
 I recommend against using SkipLists, which are based on IP addresses only. I have an exception object named "Bypass All" that disables all of the proxy checks, which applies to any destination with the "Web Proxy Bypass" tag. This approach has eliminated the need for modifying the Standard Mode proxy script or the Transparent Proxy Skip List. 
 Because Standard Mode intercepts more ports, I recommend using both proxy modes. I use Standard Mode to filter browser traffic, and Transparent Mode to filter non-browser traffic (or anything else that deliberately attempts to evade the Standard Mode proxy.) 
 Hope this helps you get started safely.