Thread Info
  • State Not Answered
  • +2 person also asked this people also asked this
  • Locked Locked
  • Replies 16 replies
  • Subscribers 40 subscribers
  • Views 4308 views
  • Users 0 members are here
Options
Suggested
This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

Web Filter Bypass Issue with URL Groups

haydenspence

Hello.

I've come across a minor issue with the Web Filter where a user may still be able to load a webpage, even though it is 'blocked.'

Running the latest version of XG firewall (18.0.4 MR-4).

To replicate:

1. Create a URL group and add a domain, for example: example.com

2. Create a User Activity with the URL group added.

3. Create a web filter Policy, denying the above created User Activity.

4. Apply the web filter Policy to a firewall rule.

On a PC impacted by the firewall rule, load the URL with a period (.) at the end of the URL. For example: http://example.com.

Without the period at the end of the URL, the webpage is blocked (as expected). With the period, it loads in full.

One more thing, the PC points to the XG for DNS.

Is anyone else able to replicate this issue?



This thread was automatically locked due to age.
Parents Reply
  • 0 in reply to emmosophos

    Hi, Emmanuel. Thanks for your message.

    I'm using the DPI engine (not the proxy). I get the same results as mentioned by Prism in the above post.

    Could this have anything to do with the home version of the XG firewall (which is what I'm using)? I though the only difference was the licensing?

Children
  • 0 in reply to haydenspence

    Hello haydenspence and Prism,

    Shouldn't make any difference as far as I know, would it be possible to share a screenshot of your Web Policy, URL Group and Firewall Rule, just to double check I am doing the same as you both.

    Regards,

     
    Emmanuel (EmmoSophos)
    Technical Team Lead, Global Community Support
    Sophos Support Videos | Product Documentation  |  @SophosSupport  | Sign up for SMS Alerts
    If a post solves your question use the 'Verify Answer' link.
  • 0 in reply to emmosophos

    Sure thing, here are the screen shots, and I've included the logs as well.

    URL Group:

    User Activity:

    Web Filter Policy:

    Firewall Rule:

    Test Results:  

    For this test, I decided to use a different device (an iPhone). The above firewall rule was applied only to the test device. Note that previous tests were done with different firewall rules, different devices and different web filter policies. These new 'test' rules and policies were created just for this, from new.

    • 2021-02-03 10:02:43 - messageid="16002" log_type="Content Filtering" log_component="HTTP" log_subtype="Denied" status="" fw_rule_id="15" user="" user_group="" web_policy_id="8" web_policy="" category="Information Technology" category_type="Acceptable" url="http://example.com/" content_type="" override_token="" response_code="" src_ip="192.168.50.4" dst_ip="93.184.216.34" protocol="TCP" src_port="49570" dst_port="80" bytes_sent="377" bytes_received="0" domain="example.com" exception="" activity_name="Test User Activity" reason="" user_agent="Mozilla/5.0 (iPhone; CPU iPhone OS 10_3_4 like Mac OS X) AppleWebKit/603.3.8 (KHTML, like Gecko) Version/10.0 Mobile/14G61 Safari/602.1" status_code="403" transaction_id="" referer="" download_file_name="" download_file_type="" upload_file_name="" upload_file_type="" con_id="220829952" app_name="" app_is_cloud="0" override_name="" override_authorizer="" used_quota="0"
    • 2021-02-03 10:03:23 - messageid="16001" log_type="Content Filtering" log_component="HTTP" log_subtype="Allowed" status="" fw_rule_id="15" user="" user_group="" web_policy_id="8" web_policy="" category="InvalidUrl" category_type="" url="http://example.com./" content_type="text/html" override_token="" response_code="" src_ip="192.168.50.4" dst_ip="93.184.216.34" protocol="TCP" src_port="49572" dst_port="80" bytes_sent="378" bytes_received="1028" domain="example.com." exception="" activity_name="" reason="" user_agent="Mozilla/5.0 (iPhone; CPU iPhone OS 10_3_4 like Mac OS X) AppleWebKit/603.3.8 (KHTML, like Gecko) Version/10.0 Mobile/14G61 Safari/602.1" status_code="200" transaction_id="" referer="" download_file_name="" download_file_type="" upload_file_name="" upload_file_type="" con_id="653403712" app_name="" app_is_cloud="0" override_name="" override_authorizer="" used_quota="0"

    Web Categories:

    As mentioned, this is not just happening with URL groups, but web categories as well. I made one change to the User Activity (as shown below) to block advertisements, then tested again using sophostest.com.

  • 0 in reply to haydenspence

    Hello,

    Thank you for the screenshots, I was able to reproduce as per the way you configured, however, if you use the default "Blocked URLs for Default Policy" then this issue doesn't happen, I was able to reproduce only with these two websites "example.com" and "testsophos.com" for any other such as Facebook.com, cnn.com it would block both regular FQDN and FQDN +. at the end. I think this is because the URL group uses a string match, that is why it blocks the sites.

    If you’re able to replicate this with some different URLs let me know.

    I also found these links that explain a bit more the difference between example.com and example.com. and why the XG would allow it when adding a . at the end, but again when I did this following your screenshots and adding facebook.com and cnn.com  then they both get blocked after adding a . at the end, however, example.com wasn't blocked neither testsophos.com

    If add testsophos.com to (example.com is by default ) the Blocked URLs for Default Policy then it blocks with and without . 

    Regards,

     
    Emmanuel (EmmoSophos)
    Technical Team Lead, Global Community Support
    Sophos Support Videos | Product Documentation  |  @SophosSupport  | Sign up for SMS Alerts
    If a post solves your question use the 'Verify Answer' link.
  • 0 in reply to emmosophos

    I can replicate this with any Sophos XG Web Categories.

    I've sent you a video showcasing It on a private message.

    If a post solves your question use the 'Verify Answer' button.

    Ryzen 5600U + I226-V (KVM) v20 GA @ Home

    XG 115w Rev.3 8GB RAM v19.5 MR3 @ Travel Firewall

  • 0 in reply to emmosophos

    This is not just happening with the above two domains. It appears to be any web category or custom URL group lists.

    Here's an example:

    If I add the 'Peer-to-peer & torrents' web category to the user activity (as per my screenshots), then load https://www.utorrent.com. the website appears in full (without the "." at the end, it is blocked).

    However, at first glance it may appear that this does not work for all websites, but that is because when you load the first URL (with the "." at the end), the web server redirects you to a different domain (without the ".") and then the user sees the blocked page. But the first request was allowed.

    Here's an example of that:

    If I add 'youtube.com' to the URL group (as per screenshots), then load http://youtube.com. in a browser, I see the blocked page. But what has actually happened is that the request was allowed, then got redirected to www.youtube.com (without the "."). If I enter https://www.youtube.com. in a browser, then the YouTube website will load partially. Some of the scripts are referenced with a FQDN to www.youtube.com (without the "."), so that's why it only loads partially because those requests are being blocked (check the web inspector).

  • 0 in reply to haydenspence

    Hello Hayden,

    Thank you, I would try to bring this to the attention of GES, I was somewhat able to replicate, for a moment the behavior was the one mentioned in the post, but after a 2nd or 3rd try the site would remain blocked, even after clearing cookies or cache, or staring on incognito mode or trying different browsers.

    I should add also that the XG would catch this using the Category InvalidURL. (not in my Web Policy)

    As you mentioned only the first time accessing would be allowed, however clicking anything on the site, would then show the blocked Website.

    I will be updating this post once I hear back from GES.

    And thank you Prism for the video. 

    Regards,

     
    Emmanuel (EmmoSophos)
    Technical Team Lead, Global Community Support
    Sophos Support Videos | Product Documentation  |  @SophosSupport  | Sign up for SMS Alerts
    If a post solves your question use the 'Verify Answer' link.
  • 0 in reply to emmosophos

    Okay, thanks for helping out.

    Just a few more points:

    • My first thought was to just block the InvalidURL web category, but it turns out that there is no category named that which can be added to a User Activity.
    • Is it technically an invalid URL with the dot at the end? Your previous posts with the links indicate that it's actually a Fully-Qualified Domain Name.
    • This issue could be exploited by malware, as a way to bypass the web filter and communicate with a host on the Internet using HTTP.

    I look forward to your update.

  • 0 in reply to haydenspence

    It is technically a valid domain name - eg it matches the internet specification.  However they are almost never used.  Most web servers will redirect you to what they consider the "normal" FQDN is.  Several web browsers will automatically change the domain name to the one without the period.  Therefore in normal browsing the chance that this happens/matters is very low.  Yes a malware author could use this to bypass some checks, however it is not a common technique.


    The system that looks up categories based on the domain name incorrectly considers them to be invalid.

    The URL Group is used in several different areas.  Some areas may not do the matching correctly.

  • 0 in reply to Michael Dunn

    Hi, Michael. I've just looked over the release notes for v18 MR5, and there is no mention of this issue as either resolved or known. What is the status of this with the development teams?

  • 0 in reply to haydenspence

    Hello,

    This is being raised to GES as two different issues:

    1) Sites are coming back with incorrect category (Invalid).
    2) Sites are not matching URL groups when there is a trailing period
    Regards,
     
    Emmanuel (EmmoSophos)
    Technical Team Lead, Global Community Support
    Sophos Support Videos | Product Documentation  |  @SophosSupport  | Sign up for SMS Alerts
    If a post solves your question use the 'Verify Answer' link.
Unfiltered HTML