[BUG] Web Filter blocking random categories

l0rdraiden,

check the website via the Policy Tester under Diagnostic menu. Also, switch to Proxy mode and see if the websites are blocked.

Regards

The web filter blocks the same stuff the policy test allows. This is still the DPI mode. Anyway it's a bug no matter if with the web proxy works.

I have seen a similar issue -- scenario:

XG Home firewall (v18 latest build) with an IPSEC tunnel to home office (happens to be a SG UTM, latest build).  Tunnel is up, traffic routes fine... for the most part.  ON the XG side a rule is configured to allow access from home office subnet to main office subnet, allow ANY -> ANY, NO web filtering enabled, NO IPS, NO App Control, etc..  Globally DPI is disabled (this was done later as a troubleshooting step).  Still... when Outlook starts on a laptop on the home office subnet, and it goes to port 80 on the mail server at the other end (the autodiscover, etc. traffic) the Web Content filter shows the following (some info redacted):

2020-03-19 15:42:15Web filtermessageid="16002" log_type="Content Filtering" log_component="HTTP" log_subtype="Denied" status="" fw_rule_id="12" user="" user_group="" web_policy_id="0" web_policy="" category="" category_type="Acceptable" url="" content_type="" override_token="" response_code="" src_ip="IP_OF_LAPTOP_AT_HOME" dst_ip="EXCHANGE_2013_SERVER" protocol="TCP" src_port="65358" dst_port="80" bytes_sent="0" bytes_received="0" domain="" exception="" activity_name="" reason="HTTP parsing error encountered." user_agent="" status_code="403" transaction_id="" referer="" download_file_name="" download_file_type="" upload_file_name="" upload_file_type="" con_id="4070301696" app_name="" app_is_cloud="0" override_name="" override_authorizer="" used_quota="0"

Using OWA to the same IP works fine, etc.  It's like the web filtering setting is being ignored on some firewall rules, and not entirely.

See screenshots below of rule in question... since I'm sure that'll be asked :)

Can someone open a ticket?

Michael Dunn can you provide support or an answer for this issue?

Thanks

similar problem as brucekconvergent.

Ticket #9771627

BrucekConvergent said:

 

XG Home firewall (v18 latest build) with an IPSEC tunnel to home office (happens to be a SG UTM, latest build).  Tunnel is up, traffic routes fine... for the most part.  ON the XG side a rule is configured to allow access from home office subnet to main office subnet, allow ANY -> ANY, NO web filtering enabled, NO IPS, NO App Control, etc..  Globally DPI is disabled (this was done later as a troubleshooting step).  Still... when Outlook starts on a laptop on the home office subnet, and it goes to port 80 on the mail server at the other end (the autodiscover, etc. traffic) the Web Content filter shows the following (some info redacted):

 

2020-03-19 15:42:15Web filtermessageid="16002" log_type="Content Filtering" log_component="HTTP" log_subtype="Denied" status="" fw_rule_id="12" user="" user_group="" web_policy_id="0" web_policy="" category="" category_type="Acceptable" url="" content_type="" override_token="" response_code="" src_ip="IP_OF_LAPTOP_AT_HOME" dst_ip="EXCHANGE_2013_SERVER" protocol="TCP" src_port="65358" dst_port="80" bytes_sent="0" bytes_received="0" domain="" exception="" activity_name="" reason="HTTP parsing error encountered." user_agent="" status_code="403" transaction_id="" referer="" download_file_name="" download_file_type="" upload_file_name="" upload_file_type="" con_id="4070301696" app_name="" app_is_cloud="0" override_name="" override_authorizer="" used_quota="0"

reason="HTTP parsing error encountered."

It is having trouble parsing the data on port 80, thinking that it is not valid HTTP for some reason.

When you say "Globally DPI is disabled" what do you mean?  You posted your firewall rule which indicates that you are using DPI not proxy.  If in the SSL/TLS inspection rules you set the SSL/TLS engine as disabled (use on for troubleshooting) it may be causing a problem with parsing some connections.

BrucekConvergent said:

 

XG Home firewall (v18 latest build) with an IPSEC tunnel to home office (happens to be a SG UTM, latest build).  Tunnel is up, traffic routes fine... for the most part.  ON the XG side a rule is configured to allow access from home office subnet to main office subnet, allow ANY -> ANY, NO web filtering enabled, NO IPS, NO App Control, etc..  Globally DPI is disabled (this was done later as a troubleshooting step).  Still... when Outlook starts on a laptop on the home office subnet, and it goes to port 80 on the mail server at the other end (the autodiscover, etc. traffic) the Web Content filter shows the following (some info redacted):

 

2020-03-19 15:42:15Web filtermessageid="16002" log_type="Content Filtering" log_component="HTTP" log_subtype="Denied" status="" fw_rule_id="12" user="" user_group="" web_policy_id="0" web_policy="" category="" category_type="Acceptable" url="" content_type="" override_token="" response_code="" src_ip="IP_OF_LAPTOP_AT_HOME" dst_ip="EXCHANGE_2013_SERVER" protocol="TCP" src_port="65358" dst_port="80" bytes_sent="0" bytes_received="0" domain="" exception="" activity_name="" reason="HTTP parsing error encountered." user_agent="" status_code="403" transaction_id="" referer="" download_file_name="" download_file_type="" upload_file_name="" upload_file_type="" con_id="4070301696" app_name="" app_is_cloud="0" override_name="" override_authorizer="" used_quota="0"

reason="HTTP parsing error encountered."

It is having trouble parsing the data on port 80, thinking that it is not valid HTTP for some reason.

When you say "Globally DPI is disabled" what do you mean?  You posted your firewall rule which indicates that you are using DPI not proxy.  If in the SSL/TLS inspection rules you set the SSL/TLS engine as disabled (use on for troubleshooting) it may be causing a problem with parsing some connections.

First of all, the rule as shown in the screenshot -- you'll note NO web content filtering selected, DPI or Web Proxy, so the web content filter from either module should not be involved, but yet they are!  This is the rule referenced in the log snippet.  I've also tried disabling TLS/SSL inspection globally, and tried leaving it on with exceptions set.

Just a FYI for anyone who's monitoring this thread:  my issue (the TLS engine interfering in traffic flow even when it is "turned off completely") is resolve in v18 MR-1 (released today).  Give it a try.

Hi,

I am following this thread because I have the exact opposite issue a site which is correctly classified in not blocked.

I have checked my default policy and it has a lot of block stuff in it which I don't believe I added. I will check the default on my test system which I haven't modified.

Ian

 fw_rule_id="6"

web_policy_id="2"

Web policy 2 is Deny All.  What is firewall rule 6, is that the one you posted?  You can see the id # in WebAdmin (this is not the first column).  If it is hitting the wrong firewall rule, check your ordering and source/destination matching logic.  You can also try using the policy tester (see tab of the log viewer).

Hi rfcat,

The default policy that comes out of the box has a lot of rules that are present but disabled (see status column), only the top two are enabled.  You have disabled them as well so for you this rule behaves as an Allow All.

Hi Michael,

I reviewed my test XG and found

1/. the default policy has two policies enabled

2/. the default business policy has all policies enabled

So in summary using the default web policies would block a lot of sites.

ian

rfcat_vk said:

Hi Michael,

I reviewed my test XG and found

1/. the default policy has two policies enabled

In the screenshot you provided, the last column that is "status" all the sliders are set to Off, and the "In Use" for the policy overall is 0, meaning you don't have it used by a firewall rule.

Can you show me screenshot of - the web policy you are trying to hit, the firewall rule you are trying to hit, the list of all firewall rules, the detailed Log Viewer log, and the results of the Policy Tester for the source/destination.

Hi  Michael,

you misunderstand my post, I was talking about the default policies in regard to this thread, because the poster was using the default policy and seeing blocked access to sites. I do not use the default policy in any of my web filters.

Ian

I had this issue as well and couldn't seem to fix it - For me the fix ended up being to run "set http relay_invalid_http_traffic on" from the Console (not ssh).