Sophos UTM: Securing Web Application Firewall (WAF)

My experience with WAF has been disappointing.   I have had so many false positives that about all I can reliably enforce is Site Path Routing.   I have found no documentation that provides "hints and tricks" to minimize problems and maximize success.  Level 1 support simply advises that if a feature causes problems, turn it off.

During the sales cycle, Forms Hardening seemed conceptually easy to implement with full transparency.  WAF simply needs to ensure that the returned form has the same field names as the one sent out, that the max_length parameter is observed on all returned fields, and that the returned character set is consistent with the requested character set.   In practice, I have never had a site which did not throw Form Hardening errors.

Form Hardening and Cookie Hardening have to be used together, so Cookie Hardening fails because Form Hardening fails.

Strict rules throw false positives so I add them to the ignore list without really understanding the implications of the exception, then I hit a wall because I have what appears to be a false positive on an infrastructure rule which should never be disabled.   So instead I disable Strict Rules.

Bad Client checking uses the email RBLs.  The email RBLs include residential IP addresses in their blocklists because residential users on dynamic IPs should not be running mail servers (but might be running mail-sending malware).  I don't want to abandon those RBLs for email protection, so I have turned Bad Client checking off for WAF protection.   (One person in this forum points out the Bad Client checking can be enabled without the RBL lookup, which performs some undocumented filtering based on GEO-IP.)

Of course, it is difficult to know during testing whether the tests have been extensive enough to expose all of the needed WAF exceptions.   Once the site is in production, it is extremely difficult to know whether a log entry represents a false positive or not.   The logs are obscure, but even after studying them carefully, it seems impossible to know how to respond to them.

Has anyone figured out how to make Form Hardening work, or figured out whether certain development practices that can either cause it to fail or allow it to succeed?

Do we even know whether WAF supports both HTML4 and HTML5?   Are there different limitations based on the HTML version?

Does anyone have any advice for making the other features useful?

  • Hi  

    Thanks for sharing the concerns with details. I admit that there's not much documentation available for the WAF in UTM9. With WAF, we always have to take a careful approach. I'll like to introduce this WAF troubleshooting article for some of the basic issues a user might face.

    For the form hardening issues caused by the form hardening HTML-parser changing the initial web request, you need to use the "Never change HTML during Static URL Hardening or Form Hardening" option, which can be enabled in the exceptions. For the other concerns, it would be different for each situation.

    Regarding Rigid filtering, you're never supposed to add the Infrastructure rules in the exception, it is clearly mentioned here. If you face any issue where your requests are rejected due to this, it might be something that needs to be checked with the Senior Support team.

    Regarding limitations with HTML4 and HTML5, I don't know for sure if we Support it or not, I will try to get in touch with someone who knows and will update you here.

    Regarding documentation or the best practice guide, I think we can prepare something that can help everyone but that will take some time.


    Regards

    Jaydeep

  • Since the Best Practice Guide has not been forthcoming, I decided to beat my head against this wall again.  Below is my current understanding of the Webserver Protection subsystem  and how to make it behave.   I plan to make changes best on my learning or corrections received from others, so I will start with a revision timestamp:

    Last Revisioin: 2019-08-20

    Securing Web Application Firewall (WAF)

    WAF detects and logs three types of incidents:

    • Path violations
    • Content threats
    • Session events

    Session events include disconnects and timeouts.  Their log entries are in the ModSecurity format, as explained below, but are of no particular importance for WAF configuration.

    The proposed model for deployment is:

    • Deploy with maximum features enabled, using a Firewall Profile in Monitor mode.
    • Adjust settings until logs indicate no false positives.
    • Switch firewall profile to Reject mode.

    Path Management

    UTM WAF provides three aspects of path management:

    • Site Path Routing determines aspects of UTM behavior, including reverse authentication profile and firewall profile.   A Site Path Route is selected on a best-match basis.   Consequently, it does not do any filtering.   A URL will always match a Site Path because the default root path is always used if a more complete match is not found.

    • URL Hardening is used to configure the allowed application entry points, and to prevent invalid navigation after the website has been entered.   When URL Hardening is enabled, the Entry URL list indicates URLs that do not need to be signed.   All other entry points must be signed, or the attempt to navigate to that URL will be blocked as a security violation.  The signature is applied as a web page is passed from the web server to the web client, so that a signed link is returned when the link is selected.

    • A web page may reference content from multiple servers.   URL Hardening only applies to links that reference the same server.   If the embedded content is from another WAF-protected site, the URLs at that site will need to be configured as entry points.

    Building the Entry Points List for URL Hardening

    The list of entry points is more complicated than might appear at first glance.   Obviously, the application design entry points should be included.   For example, the guidelines for Outlook Web Access recommend site paths for /owa and /ecp.   These should also be configured in the URL hardening entry point list.

    However, there are several entry points that browsers may reference unilaterally.  It is recommended that these be included in the manual entry points list, even if the file does not exist, so that missing files are handled as session errors rather than being handled as security threats.

    • /Favicon.ico – The icon to use for the browser tab.
    • Apple-touch-icon files – Used for browser tabs on Apple phones and tablets.  I have observed these files being requested:
      • /apple-touch-icon.png
      • /apple-touch-icon-120x120.png
      • /apple-touch-icon-120x120-precomposed.png
      • /apple-touch-icon-152x152.png
      • /apple-touch-icon-152x152-precomposed.png
      • /apple-touch-icon-precomposed.png
    • /Robots.txt – Instructions for web crawlers used by search engines.
    • /Ads.txt – Authorized Digital Seller containing instructions for programmatic advertising.

    Problems will be reported in the log file as a URL Hardening error.   If a false positive is detected, add the appropriate file to the Entry Points list.   

    What can go wrong with URL Hardening?

    UTM WAF is only compatible with sites that use static entry points.   If entry points are randomly generated to uniquely identify a user or an event, URL Hardening will not be viable for that website.  I assume that manual entry points can have querystrings appended, and still be accepted, but I have not actually tested this scenario.

    If a web browser script generates a link dynamically, the link will not be hardened, because UTM can only harden static links that it sees as the web page is being sent from the server to the browser.  In these cases, URL hardening must be disabled or the application must be redesigned.

    The documentation says that URL Hardening and Form Hardening need to be used together.  My inference from the provided examples is the following:

    • URLs without a query string are tested by URL Hardening.
    • URLS with a query string are tested by Form Hardening.

    Form Hardening

    As stated above, From Hardening and URL Hardening should be used together.  I have consistently experienced problems with unexplainable false positives, but I want to use URL Hardening, so I am currently re-testing.   Stay tuned for details.

    Cookie Hardening

    I have experienced many problems with Cookie Hardening, so I have been running with it disabled, but I have begun retesting.

    Common Threats Filter

    Content Filtering is enabled by checking the option for “Common Threats Filter”.   The common threats are divided into categories, which can be enabled or disabled individually:

    • Protocol violations  
    • Protocol anomalies  
    • Request limits  
    • HTTP policy  
    • Bad robots  
    • Generic attacks  
    • SQL injection attacks  
    • XSS attacks  
    • Tight security  
    • Trojans  
    • Outbound

    Each of these categories is associated with a set of rules, with each rule uniquely identified by a rule ID.   When a rule fires, a log entry is created.  The log entry will include the Rule ID.  

    Handling false positives in Common Threats Filter

    False positives are identified during log review.   Details of log interpretation will be addressed later in this document.

    The “Rigid Filtering” option causes all of the filtering rules to be applied more strictly.   The documentation warns that this is likely to create false positives.   So if it is enabled and false positives are occurring, the first step should be to disabled Rigid Filtering.

    If the website does not use a SQL database, the SQL Injection Attacks can reasonably be disabled, which may prevent false positives related to this category.

    For all other false positives, there are two options for creating an exception:

    • Create a “Skip Filter Rules” entry to ignore the specific Rule ID.
    • Guess the correct category, and disable that category. For example, if you find yourself exempting multiple rules related to cross-site scripting (XSS), it may be reasonable to disable the category rather than wait for every variation of this problem to occur.  In a perfect world, Sophos would provide a cross-reference document that shows which rule ID is in which category.   Unfortunately, that document does not seem to exist.

    Block Clients with Bad Reputation

    The documentation indicates that this option uses two sources:   the Maxmind GeoIP database and the Sorbs reputation block list (RBL).  

    The minimum functionality, based on GeoIP, appears to be applicable to most situations, and should be enabled.

    Like most other RBLs, the Sorbs list is optimized for email filtering, so it blocks dynamic IP addresses, especially dynamic IP addresses associated with residential ISP service.   If you expect to receive connections from home users, or even from small businesses with a dynamic IP address, you should disable the RBL check by using the option to “Skip remote lookups for clients with bad reputation.”

    Scanning

    These options are applicable to websites that accept file uploads.

    Application Customization

    The “Pass Outlook Anywhere” should be enabled if (and only if) the firewall profile is being used for an Outlook Anywhere website.

    Log Analysis

    The WAF log has two types of entries, UTM standard-format entries and ModSecurity WAF entries.   The two entries types have very different formats.

    The UTM log entries simply document the request that was received and the status code that was returned, similar to the contents of an IIS webserver log file.   The example below has been anonymized and reformatted onto multiple lines for readability.  These substitutions have been used during anonymization:

    •   c1.c2.c3.c4 = web client IP address
    •   s1.s2.s3.s4 = web server IP address

    Example 1:  UTM-format log entry

    2019:08:18-01:15:03 utmname httpd:

     id="0299"

     srcip="c1.c2.c3.c4"

     localip="s1.s2.s3.s4"

     size="0"

     user="-"

     host="s1.s2.s3.s4"

     method="OPTIONS"

     statuscode="200"

     reason="-"

     extra="-"

     exceptions="-"

     time="370"

     url="*"

     server="REF_RevFroSecureWaf_redirect_ssl"

     port="80"

     query=""

     referer="-"

     cookie="-"

     set-cookie="-" uid="XVje18CobxAAAEwpnNEAAAAZ"

     

    The ModSecurity WAF log entries use tokens enclosed in square brackets, and free-form text between the tokens, as in these examples:

    Example 2:    URL Hardening Error entry

    2019:08:18-01:15:12 defense httpd[30132]:

     [url_hardening:error]

     [pid 30132]

     [client c1.c2.c3.c4:49560]

     No signature found, URI: www.example.com/webpage.asp

     

    Example 3:   Protocol violation Error Entry

    2019:08:18-01:15:12 utmname  httpd[30132]:

     [security2:error]

     [pid 30132]

     [client c1.c2.c3.c4]

     ModSecurity: Warning. String match "HTTP/1.1" at REQUEST_PROTOCOL.

     [file "/usr/apache/conf/waf/modsecurity_crs_protocol_violations.conf"]

     [line "399"]

     [id "960020"]

     [rev "1"]

     [msg "Pragma Header requires Cache-Control Header for HTTP/1.1 requests."]

     [severity "NOTICE"]

     [ver "OWASP_CRS/2.2.7"]

     [maturity "6"]

     [accuracy "8"]

     [tag]

     [hostname "www.example.com"]

     [uri "/webpage.asp"]

     [unique_id "XVje4MCobxAAAHW0r0wAAAAY"]

    The log entries will be broken into two records if any one record exceeds 1000 characters.   It is challenging to determine how to recognize the difference between a Sophos-format entry, a OWASP-format entry, and a continuation line.   Continuation lines will have the same header structure as regular entries.   Once you identify that the data section is a continuation block, you append the data section to the data section of the record that it continues.

    In the examples above:

    For the URL Hardening error, a false positive is corrected by adding the file to the Entry Points List.

    For the Protocol Violation error, the token  [id "960020"] indicates the rule number.   A false positive can be corrected either by adding a Skip Filter Rule entry for 960020, or by unchecking “Protocol Violations” from the Command Threat Filter Categories.   Rule IDs are always a six-digit number.



    Corrected typos and formatting
    [edited by: DouglasFoster at 5:20 PM (GMT -7) on 14 Mar 2021]
  • The bad news is that enabling form and cookie hardening broke the web interface to my email system - users received an error message of the form "oops, IIS server threw an error."    I have not yet parsed the log file to determine which type of hardening was most like the cause of the problem.   9Because someone will ask, I will mention that is not an OWA/Exchange environment, so the OWA configuration guide is not directly applicable.)

    Apparently monitor-mode hardening is not transparent to the back-end application, because UTM alters the cookie or form to apply the hardening, but does not successfully strip the information back off on the return trip.   Shame on me for not testing immediately after making the change, but if makes me afraid to try again on any website.

  • One of the issues in my original post was, "After I go into production, how do I know if an alarm is a true attack or merely a false positive that was not exposed during testing?"

    This is the approach I am taking right now:

    • One site requires a WAF login.   If an alarm is detected on an IP address that successfully completed a WAF login, I am assuming that the user is not hostile and the alarm is a false positive.   This requires matching the UTM-format log entries (which provide username and IP address) with the ModSecurity-format log files (which have the alarm information and IP address.)
    • On another site, login is not required but the number of source IP addresses is very small.   Some of the IP addresses I recognized as a known business partner.   The other IP address was not immediately recognized, but from the UTM-format log entries I could determine that the IP address went through a full transaction sequence to the confirmation page.   Based on completing a normal transaction, I concluded that the traffic was not hostile and alarms could be treated as false positives.
  • I have recently discovered these resources, which I am only beginning to digest:

    The ModSecurity Handbook.   Available online in HTML format.   Available as a PDF download if  you create a (free) account on the site.   I have also signed up for their SSL/TLS newsletter.

    https://www.feistyduck.com/library/modsecurity-handbook-2ed-free/

    OWASP Best Practice Recommendations for WAF

    https://www.owasp.org/index.php/Category:OWASP_Best_Practices:_Use_of_Web_Application_Firewalls

     

  • Neither of the previous documents have the information that I wanted.

    I found this page with the ModSecurity reference manuals for both v2 and v3.   I have not yet determined which version is in my UTM firmware.

    https://github.com/SpiderLabs/ModSecurity/wiki

    When I do a web search for ModSecurity and hardending, most of the links are to this forum.   

    I cannot find anything that explains when the three hardening methods (URL, Form, Cookie) will definitely work, when they will definitely fail, or how to predict when they might fail.   With all three disabled, it seems like I have lost a lot of protection.   But I do not seem to have a safe way to even find out if they are usable, because I cannot assume that Monitor mode is non-disruptive.

     

  • I have determined that every log entry with a rule ID will have tokens for the file and line number which define the rule.   An example:

    • Rule ID 96009 is from /usr/apache/conf/waf/modsecurity_crs_protocol_anomalies.conf, line 66

    The path is actually truncated.   Full path is /var/chroot-reverseproxy/usr/apache/conf/waf/modsecurity_crs_protocol_anomalies.conf

    There are other file types in that folder, but all of the WAF log entries refer to one of the *.conf files.

    I looked at one of those files to see if i could parse it to build a reference list of the form:   Rule Category, Rule ID, Rule Description

    Unfortunately, the rule definitions use a complex syntax which would be non-trivial to parse.   However, this does allow us to see that specific GUI options can be presumed to enable or disable specific *.conf files, as follows:

    Presumed relationships betwen Common Threat Filter Categories and *.conf files
    modsecurity_crs_protocol_violations.conf, Protocol Violations
    modsecurity_crs_protocol_anomalies.conf, Protocol Anomalies
    modsecurity_crs_request_limits.conf, Request Limits
    modsecurity_crs_http_policy.conf, HTTP Policy
    modsecurity_crs_bad_robots.conf, Bad Robots
    modsecurity_crs_generic_attacks.conf, Generic Attacks
    modsecurity_crs_sql_injection_attacks.conf, SQL Injection Attacks
    modsecurity_crs_xss_attacks.conf, XSS Attacks
    modsecurity_crs_tight_security.conf, Tight Security
    modsecurity_crs_trojans.conf, Trojans
    modsecurity_crs_outbound.conf, Outbound

    Presumed relationship between other options and *.conf files
    10-REQUEST-IP-REPUTATION.conf, Block Clients with Bad Reputation,Filtering

    Other *.conf files, purpose not investigated
    base.conf
    modsecurity_crs_common_exceptions.conf
    modsecurity_crs_correlation.conf,
    modsecurity_crs_inbound_blocking.conf
    modsecurity_crs_outbound_blocking.conf