[BUG] Web Filter blocking random categories

l0rdraiden,

check the website via the Policy Tester under Diagnostic menu. Also, switch to Proxy mode and see if the websites are blocked.

Regards

The web filter blocks the same stuff the policy test allows. This is still the DPI mode. Anyway it's a bug no matter if with the web proxy works.

I have seen a similar issue -- scenario:

XG Home firewall (v18 latest build) with an IPSEC tunnel to home office (happens to be a SG UTM, latest build).  Tunnel is up, traffic routes fine... for the most part.  ON the XG side a rule is configured to allow access from home office subnet to main office subnet, allow ANY -> ANY, NO web filtering enabled, NO IPS, NO App Control, etc..  Globally DPI is disabled (this was done later as a troubleshooting step).  Still... when Outlook starts on a laptop on the home office subnet, and it goes to port 80 on the mail server at the other end (the autodiscover, etc. traffic) the Web Content filter shows the following (some info redacted):

2020-03-19 15:42:15Web filtermessageid="16002" log_type="Content Filtering" log_component="HTTP" log_subtype="Denied" status="" fw_rule_id="12" user="" user_group="" web_policy_id="0" web_policy="" category="" category_type="Acceptable" url="" content_type="" override_token="" response_code="" src_ip="IP_OF_LAPTOP_AT_HOME" dst_ip="EXCHANGE_2013_SERVER" protocol="TCP" src_port="65358" dst_port="80" bytes_sent="0" bytes_received="0" domain="" exception="" activity_name="" reason="HTTP parsing error encountered." user_agent="" status_code="403" transaction_id="" referer="" download_file_name="" download_file_type="" upload_file_name="" upload_file_type="" con_id="4070301696" app_name="" app_is_cloud="0" override_name="" override_authorizer="" used_quota="0"

Using OWA to the same IP works fine, etc.  It's like the web filtering setting is being ignored on some firewall rules, and not entirely.

See screenshots below of rule in question... since I'm sure that'll be asked :)

Can someone open a ticket?

Michael Dunn can you provide support or an answer for this issue?

Thanks

similar problem as brucekconvergent.

Ticket #9771627

This is the web filter policy that I have applied to some firewall rules where I used the DPI engine

The web filter is blocking other random categories for no reason. This is the log

And example

2020-03-16 13:30:39Web filtermessageid="16002" log_type="Content Filtering" log_component="HTTP" log_subtype="Denied" status="" fw_rule_id="10" user="samsungtv" user_group="Clientless Open Group" web_policy_id="13" web_policy="" category="Content Delivery" category_type="Acceptable" url="d25xi40x97liuc.cloudfront.net/.../70a53108-c477-4e1e-be16-2b5f2cc987fe_320w.bif content_type="" override_token="" response_code="" src_ip="192.168.1.102" dst_ip="13.225.84.68" protocol="TCP" src_port="60076" dst_port="80" bytes_sent="661" bytes_received="0" domain="d25xi40x97liuc.cloudfront.net" exception="" activity_name="" reason="HTTP pipelined request encountered." user_agent="Ignition/1.0 (samsungtv, arm)" status_code="403" transaction_id="" referer="" download_file_name="" download_file_type="" upload_file_name="" upload_file_type="" con_id="1501110912" app_name="" app_is_cloud="0" override_name="" override_authorizer="" used_quota="0"

Why is this happening?

reason="HTTP pipelined request encountered." 

HTTP pipelining is not supported in DPI mode.  It is not commonly used but then we discovered that some netflix are using it.  We are working on what we can do.

See this post for more

https://community.sophos.com/products/xg-firewall/f/firewall-and-policies/118757/v18---exclusions-by-application

I am surprised, however, to see pipelining on a non-netflix.  I would be even more surprised to see it being used with microsoft and some of the other things in your log.  Can you look at a few of them and see if they all have the same reason= ?

BrucekConvergent said:

 

XG Home firewall (v18 latest build) with an IPSEC tunnel to home office (happens to be a SG UTM, latest build).  Tunnel is up, traffic routes fine... for the most part.  ON the XG side a rule is configured to allow access from home office subnet to main office subnet, allow ANY -> ANY, NO web filtering enabled, NO IPS, NO App Control, etc..  Globally DPI is disabled (this was done later as a troubleshooting step).  Still... when Outlook starts on a laptop on the home office subnet, and it goes to port 80 on the mail server at the other end (the autodiscover, etc. traffic) the Web Content filter shows the following (some info redacted):

 

2020-03-19 15:42:15Web filtermessageid="16002" log_type="Content Filtering" log_component="HTTP" log_subtype="Denied" status="" fw_rule_id="12" user="" user_group="" web_policy_id="0" web_policy="" category="" category_type="Acceptable" url="" content_type="" override_token="" response_code="" src_ip="IP_OF_LAPTOP_AT_HOME" dst_ip="EXCHANGE_2013_SERVER" protocol="TCP" src_port="65358" dst_port="80" bytes_sent="0" bytes_received="0" domain="" exception="" activity_name="" reason="HTTP parsing error encountered." user_agent="" status_code="403" transaction_id="" referer="" download_file_name="" download_file_type="" upload_file_name="" upload_file_type="" con_id="4070301696" app_name="" app_is_cloud="0" override_name="" override_authorizer="" used_quota="0"

reason="HTTP parsing error encountered."

It is having trouble parsing the data on port 80, thinking that it is not valid HTTP for some reason.

When you say "Globally DPI is disabled" what do you mean?  You posted your firewall rule which indicates that you are using DPI not proxy.  If in the SSL/TLS inspection rules you set the SSL/TLS engine as disabled (use on for troubleshooting) it may be causing a problem with parsing some connections.

Hello Michael

Here you can donwload formated csv where you can see all the devices (windows 10 pc, android phones, alexa, SamsungTV, etc,) and urls, I have checked all of them and the reason is always "HTTP pipelined request encountered."

1drv.ms/.../s!Ar4JZUxFYoRDhNIxBeRWJN1FSF0W5A

Going back to the old web proxy will fix the issue?

Will the pipelined requests be supported? is there an estimation date? or is there any other workraound instead of going back to the old web proxy?

Please tell once you have downloaded the file becasue I plan to remove it, thanks.

First of all, the rule as shown in the screenshot -- you'll note NO web content filtering selected, DPI or Web Proxy, so the web content filter from either module should not be involved, but yet they are!  This is the rule referenced in the log snippet.  I've also tried disabling TLS/SSL inspection globally, and tried leaving it on with exceptions set.