[BUG] Web Filter blocking random categories

This is the web filter policy that I have applied to some firewall rules where I used the DPI engine

The web filter is blocking other random categories for no reason. This is the log

And example

2020-03-16 13:30:39Web filtermessageid="16002" log_type="Content Filtering" log_component="HTTP" log_subtype="Denied" status="" fw_rule_id="10" user="samsungtv" user_group="Clientless Open Group" web_policy_id="13" web_policy="" category="Content Delivery" category_type="Acceptable" url="d25xi40x97liuc.cloudfront.net/.../70a53108-c477-4e1e-be16-2b5f2cc987fe_320w.bif content_type="" override_token="" response_code="" src_ip="192.168.1.102" dst_ip="13.225.84.68" protocol="TCP" src_port="60076" dst_port="80" bytes_sent="661" bytes_received="0" domain="d25xi40x97liuc.cloudfront.net" exception="" activity_name="" reason="HTTP pipelined request encountered." user_agent="Ignition/1.0 (samsungtv, arm)" status_code="403" transaction_id="" referer="" download_file_name="" download_file_type="" upload_file_name="" upload_file_type="" con_id="1501110912" app_name="" app_is_cloud="0" override_name="" override_authorizer="" used_quota="0"

Why is this happening?

reason="HTTP pipelined request encountered." 

HTTP pipelining is not supported in DPI mode.  It is not commonly used but then we discovered that some netflix are using it.  We are working on what we can do.

See this post for more

https://community.sophos.com/products/xg-firewall/f/firewall-and-policies/118757/v18---exclusions-by-application

I am surprised, however, to see pipelining on a non-netflix.  I would be even more surprised to see it being used with microsoft and some of the other things in your log.  Can you look at a few of them and see if they all have the same reason= ?

This is the web filter policy that I have applied to some firewall rules where I used the DPI engine

The web filter is blocking other random categories for no reason. This is the log

And example

2020-03-16 13:30:39Web filtermessageid="16002" log_type="Content Filtering" log_component="HTTP" log_subtype="Denied" status="" fw_rule_id="10" user="samsungtv" user_group="Clientless Open Group" web_policy_id="13" web_policy="" category="Content Delivery" category_type="Acceptable" url="d25xi40x97liuc.cloudfront.net/.../70a53108-c477-4e1e-be16-2b5f2cc987fe_320w.bif content_type="" override_token="" response_code="" src_ip="192.168.1.102" dst_ip="13.225.84.68" protocol="TCP" src_port="60076" dst_port="80" bytes_sent="661" bytes_received="0" domain="d25xi40x97liuc.cloudfront.net" exception="" activity_name="" reason="HTTP pipelined request encountered." user_agent="Ignition/1.0 (samsungtv, arm)" status_code="403" transaction_id="" referer="" download_file_name="" download_file_type="" upload_file_name="" upload_file_type="" con_id="1501110912" app_name="" app_is_cloud="0" override_name="" override_authorizer="" used_quota="0"

Why is this happening?

reason="HTTP pipelined request encountered." 

HTTP pipelining is not supported in DPI mode.  It is not commonly used but then we discovered that some netflix are using it.  We are working on what we can do.

See this post for more

https://community.sophos.com/products/xg-firewall/f/firewall-and-policies/118757/v18---exclusions-by-application

I am surprised, however, to see pipelining on a non-netflix.  I would be even more surprised to see it being used with microsoft and some of the other things in your log.  Can you look at a few of them and see if they all have the same reason= ?

Hello Michael

Here you can donwload formated csv where you can see all the devices (windows 10 pc, android phones, alexa, SamsungTV, etc,) and urls, I have checked all of them and the reason is always "HTTP pipelined request encountered."

1drv.ms/.../s!Ar4JZUxFYoRDhNIxBeRWJN1FSF0W5A

Going back to the old web proxy will fix the issue?

Will the pipelined requests be supported? is there an estimation date? or is there any other workraound instead of going back to the old web proxy?

Please tell once you have downloaded the file becasue I plan to remove it, thanks.

l0rdraiden said:

Hello Michael

Here you can donwload formated csv where you can see all the devices (windows 10 pc, android phones, alexa, SamsungTV, etc,) and urls, I have checked all of them and the reason is always "HTTP pipelined request encountered."

1drv.ms/.../s!Ar4JZUxFYoRDhNIxBeRWJN1FSF0W5A

Going back to the old web proxy will fix the issue?

Will the pipelined requests be supported? is there an estimation date? or is there any other workraound instead of going back to the old web proxy?

Please tell once you have downloaded the file becasue I plan to remove it, thanks.

You can remove the file.  In your case I suspect you are not encountering real pipelining but that the DPI engine is logging it as a pipeline error even though the underlying cause is something else.  I have not heard of this problem before.  If you can, please raise this direct with Sophos Support so that it can be investigated.  If you don't have support, hope that someone else reports it or that it gets fixed via other work that we are doing and retest after every MR.

Switching to proxy should resolve this.

Thanks for you help, I would consider this a high importance bug I mean the firewall is bloking things that it should not block, a dev should take this feedback and create a bug report, no need to wait for a paid customer to complain.

PMParth Could you consider this bug? I think is quite important one.

Web filter is completely broken in V18, both DPI and old proxy.

Since with DPI doesn't work lets try with the old proxy

 I still get websites blocked for no reason at all. Reason:none category:none

I don't think I'm the only one having this problem since my configuration is quite clean and simple.