Sophos UTM: Securing Web Application Firewall (WAF)

Question

My experience with WAF has been disappointing. I have had so many false positives that about all I can reliably enforce is Site Path Routing. I have found no documentation that provides "hints and tricks" to minimize problems and maximize success. Level 1 support simply advises that if a feature causes problems, turn it off. 
 During the sales cycle, Forms Hardening seemed conceptually easy to implement with full transparency. WAF simply needs to ensure that the returned form has the same field names as the one sent out, that the max_length parameter is observed on all returned fields, and that the returned character set is consistent with the requested character set. In practice, I have never had a site which did not throw Form Hardening errors. 
 Form Hardening and Cookie Hardening have to be used together, so Cookie Hardening fails because Form Hardening fails. 
 Strict rules throw false positives so I add them to the ignore list without really understanding the implications of the exception, then I hit a wall because I have what appears to be a false positive on an infrastructure rule which should never be disabled. So instead I disable Strict Rules. 
 Bad Client checking uses the email RBLs. The email RBLs include residential IP addresses in their blocklists because residential users on dynamic IPs should not be running mail servers (but might be running mail-sending malware). I don't want to abandon those RBLs for email protection, so I have turned Bad Client checking off for WAF protection. (One person in this forum points out the Bad Client checking can be enabled without the RBL lookup, which performs some undocumented filtering based on GEO-IP.) 
 Of course, it is difficult to know during testing whether the tests have been extensive enough to expose all of the needed WAF exceptions. Once the site is in production, it is extremely difficult to know whether a log entry represents a false positive or not. The logs are obscure, but even after studying them carefully, it seems impossible to know how to respond to them. 
 Has anyone figured out how to make Form Hardening work, or figured out whether certain development practices that can either cause it to fail or allow it to succeed? 
 Do we even know whether WAF supports both HTML4 and HTML5? Are there different limitations based on the HTML version? 
 Does anyone have any advice for making the other features useful?

DouglasFoster · Accepted Answer

Since the Best Practice Guide has not been forthcoming, I decided to beat my head against this wall again. Below is my current understanding of the Webserver Protection subsystem and how to make it behave. I plan to make changes best on my learning or corrections received from others, so I will start with a revision timestamp: 
 Last Revisioin: 2019-08-20 
 Securing Web Application Firewall (WAF) 
 WAF detects and logs three types of incidents: 
 
 Path violations 
 Content threats 
 Session events 
 
 Session events include disconnects and timeouts. Their log entries are in the ModSecurity format, as explained below, but are of no particular importance for WAF configuration. 
 The proposed model for deployment is: 
 
 Deploy with maximum features enabled, using a Firewall Profile in Monitor mode. 
 Adjust settings until logs indicate no false positives. 
 Switch firewall profile to Reject mode. 
 
 Path Management 
 UTM WAF provides three aspects of path management: 
 
 Site Path Routing determines aspects of UTM behavior, including reverse authentication profile and firewall profile. A Site Path Route is selected on a best-match basis. Consequently, it does not do any filtering. A URL will always match a Site Path because the default root path is always used if a more complete match is not found. 
 URL Hardening is used to configure the allowed application entry points, and to prevent invalid navigation after the website has been entered. When URL Hardening is enabled, the Entry URL list indicates URLs that do not need to be signed. All other entry points must be signed, or the attempt to navigate to that URL will be blocked as a security violation. The signature is applied as a web page is passed from the web server to the web client, so that a signed link is returned when the link is selected. 
 A web page may reference content from multiple servers. URL Hardening only applies to links that reference the same server. If the embedded content is from another WAF-protected site, the URLs at that site will need to be configured as entry points. 
 
 Building the Entry Points List for URL Hardening 
 The list of entry points is more complicated than might appear at first glance. Obviously, the application design entry points should be included. For example, the guidelines for Outlook Web Access recommend site paths for /owa and /ecp. These should also be configured in the URL hardening entry point list. 
 However, there are several entry points that browsers may reference unilaterally. It is recommended that these be included in the manual entry points list, even if the file does not exist, so that missing files are handled as session errors rather than being handled as security threats. 
 
 /Favicon.ico &ndash; The icon to use for the browser tab. 
 Apple-touch-icon files &ndash; Used for browser tabs on Apple phones and tablets. I have observed these files being requested:
 
 /apple-touch-icon.png 
 /apple-touch-icon-120x120.png 
 /apple-touch-icon-120x120-precomposed.png 
 /apple-touch-icon-152x152.png 
 /apple-touch-icon-152x152-precomposed.png 
 /apple-touch-icon-precomposed.png

/Robots.txt &ndash; Instructions for web crawlers used by search engines. 
 /Ads.txt &ndash; Authorized Digital Seller containing instructions for programmatic advertising. 
 
 Problems will be reported in the log file as a URL Hardening error. If a false positive is detected, add the appropriate file to the Entry Points list. 
 What can go wrong with URL Hardening? 
 UTM WAF is only compatible with sites that use static entry points. If entry points are randomly generated to uniquely identify a user or an event, URL Hardening will not be viable for that website. I assume that manual entry points can have querystrings appended, and still be accepted, but I have not actually tested this scenario. 
 If a web browser script generates a link dynamically, the link will not be hardened, because UTM can only harden static links that it sees as the web page is being sent from the server to the browser. In these cases, URL hardening must be disabled or the application must be redesigned. 
 The documentation says that URL Hardening and Form Hardening need to be used together. My inference from the provided examples is the following: 
 
 URLs without a query string are tested by URL Hardening. 
 URLS with a query string are tested by Form Hardening. 
 
 Form Hardening 
 As stated above, From Hardening and URL Hardening should be used together. I have consistently experienced problems with unexplainable false positives, but I want to use URL Hardening, so I am currently re-testing. Stay tuned for details. 
 Cookie Hardening 
 I have experienced many problems with Cookie Hardening, so I have been running with it disabled, but I have begun retesting. 
 Common Threats Filter 
 Content Filtering is enabled by checking the option for &ldquo;Common Threats Filter&rdquo;. The common threats are divided into categories, which can be enabled or disabled individually: 
 
 Protocol violations 
 Protocol anomalies 
 Request limits 
 HTTP policy 
 Bad robots 
 Generic attacks 
 SQL injection attacks 
 XSS attacks 
 Tight security 
 Trojans 
 Outbound 
 
 Each of these categories is associated with a set of rules, with each rule uniquely identified by a rule ID. When a rule fires, a log entry is created. The log entry will include the Rule ID. 
 Handling false positives in Common Threats Filter 
 False positives are identified during log review. Details of log interpretation will be addressed later in this document. 
 The &ldquo;Rigid Filtering&rdquo; option causes all of the filtering rules to be applied more strictly. The documentation warns that this is likely to create false positives. So if it is enabled and false positives are occurring, the first step should be to disabled Rigid Filtering. 
 If the website does not use a SQL database, the SQL Injection Attacks can reasonably be disabled, which may prevent false positives related to this category. 
 For all other false positives, there are two options for creating an exception: 
 
 Create a &ldquo;Skip Filter Rules&rdquo; entry to ignore the specific Rule ID. 
 Guess the correct category, and disable that category. For example, if you find yourself exempting multiple rules related to cross-site scripting (XSS), it may be reasonable to disable the category rather than wait for every variation of this problem to occur. In a perfect world, Sophos would provide a cross-reference document that shows which rule ID is in which category. Unfortunately, that document does not seem to exist. 
 
 Block Clients with Bad Reputation 
 The documentation indicates that this option uses two sources: the Maxmind GeoIP database and the Sorbs reputation block list (RBL). 
 The minimum functionality, based on GeoIP, appears to be applicable to most situations, and should be enabled. 
 Like most other RBLs, the Sorbs list is optimized for email filtering, so it blocks dynamic IP addresses, especially dynamic IP addresses associated with residential ISP service. If you expect to receive connections from home users, or even from small businesses with a dynamic IP address, you should disable the RBL check by using the option to &ldquo;Skip remote lookups for clients with bad reputation.&rdquo; 
 Scanning 
 These options are applicable to websites that accept file uploads. 
 Application Customization 
 The &ldquo;Pass Outlook Anywhere&rdquo; should be enabled if (and only if) the firewall profile is being used for an Outlook Anywhere website. 
 Log Analysis 
 The WAF log has two types of entries, UTM standard-format entries and ModSecurity WAF entries. The two entries types have very different formats. 
 The UTM log entries simply document the request that was received and the status code that was returned, similar to the contents of an IIS webserver log file. The example below has been anonymized and reformatted onto multiple lines for readability. These substitutions have been used during anonymization: 
 
 c1.c2.c3.c4 = web client IP address 
 s1.s2.s3.s4 = web server IP address 
 
 Example 1: UTM-format log entry 
 2019:08:18-01:15:03 utmname httpd: 
 id="0299" 
 srcip="c1.c2.c3.c4" 
 localip="s1.s2.s3.s4" 
 size="0" 
 user="-" 
 host="s1.s2.s3.s4" 
 method="OPTIONS" 
 statuscode="200" 
 reason="-" 
 extra="-" 
 exceptions="-" 
 time="370" 
 url="*" 
 server="REF_RevFroSecureWaf_redirect_ssl" 
 port="80" 
 query="" 
 referer="-" 
 cookie="-" 
 set-cookie="-" uid="XVje18CobxAAAEwpnNEAAAAZ" 
 
 The ModSecurity WAF log entries use tokens enclosed in square brackets, and free-form text between the tokens, as in these examples: 
 Example 2: URL Hardening Error entry 
 2019:08:18-01:15:12 defense httpd[30132]: 
 [url_hardening:error] 
 [pid 30132] 
 [client c1.c2.c3.c4:49560] 
 No signature found, URI: www.example.com/webpage.asp 
 
 Example 3: Protocol violation Error Entry 
 2019:08:18-01:15:12 utmname httpd[30132]: 
 [security2:error] 
 [pid 30132] 
 [client c1.c2.c3.c4] 
 ModSecurity: Warning. String match "HTTP/1.1" at REQUEST_PROTOCOL. 
 [file "/usr/apache/conf/waf/modsecurity_crs_protocol_violations.conf"] 
 [line "399"] 
 [id "960020"] 
 [rev "1"] 
 [msg "Pragma Header requires Cache-Control Header for HTTP/1.1 requests."] 
 [severity "NOTICE"] 
 [ver "OWASP_CRS/2.2.7"] 
 [maturity "6"] 
 [accuracy "8"] 
 [tag] 
 [hostname "www.example.com"] 
 [uri "/webpage.asp"] 
 [unique_id "XVje4MCobxAAAHW0r0wAAAAY"] 
 
 The log entries will be broken into two records if any one record exceeds 1000 characters. It is challenging to determine how to recognize the difference between a Sophos-format entry, a OWASP-format entry, and a continuation line. Continuation lines will have the same header structure as regular entries. Once you identify that the data section is a continuation block, you append the data section to the data section of the record that it continues. 
 
 In the examples above: 
 For the URL Hardening error, a false positive is corrected by adding the file to the Entry Points List. 
 For the Protocol Violation error, the token [id "960020"] indicates the rule number. A false positive can be corrected either by adding a Skip Filter Rule entry for 960020, or by unchecking &ldquo;Protocol Violations&rdquo; from the Command Threat Filter Categories. Rule IDs are always a six-digit number.