Can not apply QoS when enable Web Filtering?

Hi guys, I have the same problem on our UTM 550 cluster.

We configure QOS, in particoular "downlaod throttling" for youtube and it works like a charm, but when apply the webfiltering (full transparent mode) to just a computer to try  (a very simple policy with set a quota for streaming) suddenly to this computer is not anymore apply the QOS (I checked out the statistics from youtube..).

I tried to apply the same "downlaod throttling policy" not just to WAN interface but also to the LAN, because when a computer used the webproxy I could see it traffic on this interface and not on the WAN one but nothing,

I tried everything and every combination, could anyone help me?

I  will also contact my partner in order to open a ticket in Sophos becuase it's really strange and disappointing  behaviuor

Thank you all

Riccardo

I have a similar problem, and I'm using bandwidth pools.

I'm trying to throttle uploads to AWS which use port 443. After much fiddling, I have a bandwidth pool set up on the external interface and a traffic selector that selects for all traffic from a particular host. This doesn't apply the QoS until I explicitly put the host in the 'Skip Transparent Mode Source Host/Nets' in the Filtering Options/Misc page. Then the bandwidth pool gets applied.

It seems if you are using Web Filtering, and you want to use Bandwidth Pools, you need to explicitly exclude the traffic or host from Web Filtering. If you don't do this, the QoS doesn't get applied (for web traffic, i.e. ports 80, 443, etc...).

It's not a great solution, since I'd love to us QoS/Bandwidth Pools and Web Filtering for these hosts, but it just doesn't work.

FYI - I'm using UTM 9.355-1.

Hi, Paulo, and welcome to the UTM Community!

Please insert pictures of the 'Status' tab, your Traffic Selector and your Bandwidth Pools.

Cheers - Bob

Looks good.  Some suggestions:

1. Disable the Internal interface on the Status tab.
2. Edit the External interface on the Status tab:
   
   1. Deselect 'Download Equalizer'
   2. Do not select 'Limit uplink'

Now try - any luck?

Cheers - Bob

Nope. Same issue. The QoS Bandwidth Profile only gets applied if I specifically exclude the host in Web Protection / Filtering Options / Misc / Transparent Mode Skiplist / Skip Transparent Mode Source Hosts Nets.

It's possibly due to the fact that AWS uses port 443, but then again, the web filtering is set to Transparent and HTTPS is set to URL filtering only.

Like I said, I have a solution, it just doesn't make sense. I don't see why I need to exclude the host from Web Filtering for it to work, unless QoS and Web Filtering are mutually exclusive when it comes to web traffic on ports 80, 443 etc...

I haven't tried it with other protocols. I may see if I can do an scp from the host to see if the QoS profile gets applied without specifying the exclusion.

Confirmed. If it's not a web port, the QoS bandwidth pool gets applied without having to exclude the host from web filtering. I did an scp of a large file and the bandwidth pool restrictions were applied no problem.

So it seems web filtering seems to get in the way of Bandwidth Pools if the traffic you're trying to apply QoS to is web traffic, in which case, the pool is bypassed unless you specifically exclude the host from the web filtering.

Use application control for throttling. A lot easier and you don't have to worry about proxies. Only drawback is that they sometimes create a sub category like in the example below speedtest.net has its own category and you have to add it to throttle that particular website because http/s won't work (counter intuitive since http/s should cover all port 80/443 traffic but whatever). Create rules as below and you should be fine. 

I know its not as simple as ANY ANY throttle but it works with web proxy and most of the time all you need to throttle is http/s traffic anyway. Hope this helps.

Yep, I tried application control originally, and only tried any any throttle because I thought it wasn't selecting for the AWS profile. Didn't work. I even created rules by selecting 'Shape' traffic from the Flow Monitor screens, where the UTM automatically creates the rules. It just plainly doesn't work for the AWS port 443 traffic unless I specifically exclude it from Web Filtering. I have successfully created both inbound and outbound bandwidth pools before (even for speedtest.net)

Maybe it's a peculiarity of my setup which doesn't seem that strange. I really don't understand why it's not working as it should. I also tried mapping the host to it's own SNAT rule, thinking NAT could be getting in the way, it just doesn't seem to want to play. I can try every other type of traffic (by changing the selector), but 443 traffic, the profile just get's ignored.

I think it's a bug, but I have a workaround so it's ok for now. It may be a particularity of the specific host (it has a bonded interface which may be causing some strange issues?) - I will try with another host.

You're right, Paulo, I was thinking about Uplink Balancing - you can't do QoS on individual clients after the traffic has passed through the Web Filtering proxy.

Cheers - Bob

I know this is an old post, but I'd like to add my 2 cents.

This is not a bug in the firmware or anything, this is simply how the Sophos UTM process traffic. Web proxy has higher precedence over QoS. This means if web filtering is enabled, the UTM will ignore what you have in QoS. 

Now, why would Sophos behave like this? This makes web filtering useless when you want to apply QoS rules. It makes paying for web filtering subscription useless. 

Bob, do you have any comments on this?

I know this is an old post, but I'd like to add my 2 cents.

This is not a bug in the firmware or anything, this is simply how the Sophos UTM process traffic. Web proxy has higher precedence over QoS. This means if web filtering is enabled, the UTM will ignore what you have in QoS. 

Now, why would Sophos behave like this? This makes web filtering useless when you want to apply QoS rules. It makes paying for web filtering subscription useless. 

Bob, do you have any comments on this?

The trick here is to use "Application" selector rather than "Traffic" selector under Traffic Selectors (as per BillyBob's post above)

then add to the following:

Download Throttling = this will only affect the download speed. Upload will continue.

Bandwidth Pools = this has no effect on download or upload. It only had an effect on upload when you selected "Specify upper bandwidth limit"

Combination of both = download & upload is limited to what you set in QoS

I added the above to the WAN intereface. You may get more mileage by trying different interfaces for bandwidth pools etc

It's another one of those UTM Gotcha's (which I like to call them) where using one of the UTM proxies throws up exceptions to the normal logical rules that most people would apply.

If you select it this way, you will find the UTM has a very granular control for QoS in the fact that it can filter different types of applications using http rather than just blanket QoS all of http

I have this morning tried this again (on a non production rig) and it does work.

CAVEAT:

One strange thing I noted was that my upload seemed to drastically reduced when "download throttling" was enabled. No idea why.

I tried multiple times and the result was the same (50% decrease in upload speed (with download throttling enabled) with the download speed working as expected to the Qos limit)

Not using "Download throttling" and just using bandwidth pools with upper limit set, set the upload to the correct speed and my upload limit was not reduced by 50% as it was when download throttling was enabled.

Hi Louis,

Thank you for your inputs.

I tried the throttling with applications and it worked. However, in the traffic selector, you have to set Any to Any for it to work. You can't limit this to a certain network of your liking. I think this is because the UTM proxy does not retain the source ip address of the hosts. This is not good because I have a web server that I do not want to be affected by this throttling. The reason I wanted to throttle so that I can reserve whatever bandwidth left for my web server. Now, I have to decide whether web filtering or QoS is important and use just one. If I ended up selecting QoS, I will not continue with the web filtering subscription next year - it's a waste of money.

Quang, please be more specific about what traffic you want to prefer, whether it's request traffic or response traffic, where the web server is and where the users are relative to the UTM.

Cheers - Bob

BAlfson said:

Quang, please be more specific about what traffic you want to prefer, whether it's request traffic or response traffic, where the web server is and where the users are relative to the UTM.

Cheers - Bob

 

Hi Bob,

My web server is hosted behind the UTM with DNAT.

I have a user vlan and a server vlan. The inter-vlan routing are handled by my L3 switch, which has nothing to do with the UTM. However, I'd like to throttle "Any" download traffic to my user vlan to a certain bandwidth. In other words, I don't want users from the user vlan to saturate my total allowed bandwidth speed.

I created a traffic selector like this: Source=Any Service=Any Destination=user vlan

I then create a throttling rule on my external interface and throttle the speed for such traffic created above. This works fine with no issues ONLY if user vlan is not in the Allowed Networks under web filtering. 

Let me describe the situation in my own words to see if I understand.  You have a group of web servers that are used by people out on the Internet.  Your users sometimes fill your pipe with downloads and that interrupts the inbound web requests from the Internet.  You want to throttle the downloads requested by your users, but not the inbound requests from the Internet.  Correct?

Do the web servers also have outbound requests that result in large inbound responses?  If so, could these be configured to occur when your users are not present?

My idea is to use, for example "HTTP Response" = 80→1:65535 and "HTTP" = 1:65535 → 80.  Then, use a Download Throttling rule on the External interface to limit "HTTP Response" from "Internet" to "External (Address)" instead of to the server VLAN.  Also, on the External interface, place a Bandwidth Pool guaranteeing preference to "HTTP Response" traffic.

Cheers - Bob

BAlfson said:

Quang, please be more specific about what traffic you want to prefer, whether it's request traffic or response traffic, where the web server is and where the users are relative to the UTM.

Cheers - Bob

 

Hi Bob,

Sorry for the confusion and unclear description. Let's just take the web servers out of the equation here. Let me explain what I want to achieve:

I want to limit my internal users from all internet downloads activities to, let's say, 20mbps. 

Right now, I'm doing this by creating a traffic selector as such:

Source: Any
Service: Any
Destination: Internal users network group

I then create a throttle rule on the external (WAN) interface with the above traffic selector.

With this QoS rule, all internet download activities in my internal network is limit to 20mbps, and this is exactly what I want. However, this would only work without web filtering applied to the Internal users net work group.

I do understand that according to the rulz #2, proxy traffic (web filtering) gets processed first, hence the QoS rule I have above is ignored. 

Now, can I still achieve my goal above with web filtering also applied?

I'm just wondering if you are thinking about this in the opposite way ie you are trying to throttle the web users instead of guaranteeing the web server x amount of bandwidth?

https://community.sophos.com/kb/en-us/115020

I've not tried it but in there under bandwidth pools, it states that it should work like that.

Louis-M said:

I'm just wondering if you are thinking about this in the opposite way ie you are trying to throttle the web users instead of guaranteeing the web server x amount of bandwidth?

I've not tried it but in there under bandwidth pools, it states that it should work like that.

Hi Louis,

Bandwidth pool is actually for guaranteeing upload speed, not download.

As I suggested above, use 'Internet -> HTTP Response -> External (Address)' as the Traffic Selector in your Download Throttling rule instead of 'Any -> Any -> {Internal users network group}'.

Cheers - Bob

BAlfson said:

As I suggested above, use 'Internet -> HTTP Response -> External (Address)' as the Traffic Selector in your Download Throttling rule instead of 'Any -> Any -> {Internal users network group}'.

Cheers - Bob

 

Thank you, Bob. This works great. 

You also mentioned above to apply a QoS rule for bandwidth pool with https response. Is this to guarantee the upload or download speed?

Thanks!