QoS issues (again)

Hello,

What are the results when you perform the test without any users on the network and please share result? Also have you tried directly testing without the FW? PC->directly connected to router->Speedtest? could you share result if you're getting exactly 500 or at least near 500MB DL speed w/o the FW?

Also could you share your current SFOS version? Thank you

Regards,

I have replied to you, but my post was flagged as spam, possibly due to the fact that I pasted speedtest links to share the results (was running the windows app this time).

Anyway, since I don't know when/if my reply will be unflagged, I am replying again..

So at the moment there is not much bandwidth consumed at the house

Run a speedtest (from a linux machine now) without the firewall rule enabled:

Download: 422.91 Mbps (data used: 407.7 MB)
32.73 ms (jitter: 4.52ms, low: 3.74ms, high: 253.45ms)
Upload: 52.96 Mbps (data used: 25.2 MB)
44.28 ms (jitter: 9.33ms, low: 19.42ms, high: 292.46ms)
Packet Loss: 0.0%

I enable the firewall rule again:

Download: 313.15 Mbps (data used: 238.5 MB)
5.93 ms (jitter: 1.43ms, low: 2.98ms, high: 12.61ms)
Upload: 47.09 Mbps (data used: 22.4 MB)
2.84 ms (jitter: 0.41ms, low: 2.27ms, high: 12.94ms)
Packet Loss: 0.0%

I disabled the rule again and at the next speedtest I got 430/45

I enabled it then once more and got 310/47

The ISP's router is set to bridge mode, which effectively turns it to an ONT

I have seen many times the speed reaching at 500, but even if I didn't, the QoS rule (which has the correct numbers for KBps) should not limit me to 300, it should at least give me the full available speed since it is capped at a higher speed that the actual one. It should not contantly cap me at 300

I am running the latest sfos version SFVH (SFOS 20.0.2 MR-2-Build378)

Thanks for the screenshot. Never doubted that it works for you. It is just that for some reason it does not work for me..

And after a few tests, I believe there is a possibility that it would not work for you, either, if your speed was above the one you have.

What I mean:

I performed a first test using your numbers. (Total 32500, guaranteed 30000, limit 31000)

The results I get are similar to yours .. I got 204/44 - forgot to take a screenshot :)

I then cut the numbers in half (Total 16000, guaranteed 15000, limit 15500).  Run a speedtest and download speed indeed was cut in half

I then set the numbers at a quarter of yours (Total 8000, guaranteed 7500, limit 7800). Run a speedtest and again the speeds are cut at a quarter of your speeds.

So everything from about 250Mbits and below works as expected so far. I then set once again the speeds at about double your numbers. (Total 60000, guaranteed 58000, limit 59000)

And lo and behold, the speeds are not exceeding your numbers

I am starting to believe that there is a bug and it won't let speeds go above about 250Mbits.

And it is just a diabolical coincidence that your speeds don't exceed 250, so you have never experienced this issue.

But of course this is purely hypothetical. I hope there is someone with speeds above 400 that can run a test (??)

I previously had a 1000/50 link which maxed out the xg115w at around 600-700.

Don't know then...

Can't be sure. I can't understand why I am the only one having this issue.

Is there perhaps any bug introduced lately that is causing that behavior?

If there is anyone with the latest version kind enough to test with a connection faster than 400 Mbits, it would be great.

At least I will be sure that the problem is only on my side (although I can't comprehend why this would happen only to me...)

Do you have ips enabled on the testing rule p, if so what are the settings?
ian

I did not even have a rule in place during the above tests.

But just to make sure I enabled the rule now and got same exact results as the last speedtest screenshot (about 250down)

The rule has no ips configured. It is just a simple rule. Source zones all my internal networks, destination zone WAN and from the rest of the settings I have nothing configured, no app control, no web filter, no ips, just trafiic shaping:

The Home_QoS rule has the below settings:

Perhaps it is worth mentioning that if I change Upload bandwidth to half of the above (3300), the upload speed is limited to half, as it should

So once again the "low" numbers work as they should.

Numbers that offer a download limit above 250Mbits are the ones that don't work

OK, here is mine

Additionally let me share that for download speed in the traffic shaping rule, even if I put the max number 2560000, it still limits to 250Mbits 

It is as if it will discard any number above 30000-35000 for download limit

A suggestion you might like to consider, your hardware is being maxed out. You could try running two or three speed tests at the same time from different devices and look at the diagnostic graphs.

ian

It has crossed my mind and now that you mentioned it, too, I gave it a try

I run a windows app speedtest and a command line speedtest at the same time. During these tests I had the XG console running top.

The tests combined did not exceed  280Mbits down while CPU usage did not exceed 58%.

EDIT : Run again and CPU reached 72%. With only one test running it does not go beyond 60%

Are there any other commands I can issue from the console to possibly get more accurate readings?

The cpu is a 4 core so unless you go into console and run top or similar you will never see what is happening.

i did a deeper investigation into the cou, it is 11 years old model with in built nics, so probably you are seeing the best performance you will get.

ian

ian

The cpu is a 4 core so unless you go into console and run top or similar you will never see what is happening.

i did a deeper investigation into the cou, it is 11 years old model with in built nics, so probably you are seeing the best performance you will get.

ian

ian

Well, top is indeed what I run.

As I mentioned it does not exceed 72% and even that load was only momentarily

Does traffic shaping tax the CPU more?

I assume it does, but still, 72% is fairly low. It still has room to stretch its legs

For what it is worth, here is the CPU graph from the GUI (last 24hours)

Thank you for the graph. What it doesn't show is what the load on each core is? 72% would be the overall load for all cores. Probably means one or more cores is maxed.

Ian

I guess there is always this possibility

I run top from the console but top isn't clear, either.

htop would be a much better option since it distinguishes the CPU usage per core, but it is not installed

Do you happen to know if it is possible to install htop? Is there any possibility at all for sophos to install something additional using a package manager?

EDIT: Well, I found this page https://community.sophos.com/sophos-xg-firewall/f/recommended-reads/134787/sophos-firewall-understanding-top-and-atop-command-in-sophos-firewall-utm#mcetoc_1g4rqfs483

I tried running atop and top with Irix mode on and hit 1 to sort by CPU

While I CPU no CPU going to full utilization I must admit that don't  have a clear picture about it. 

EDIT2: I created a video while running 2 speedtests and showing top in the background. 

Don't know if you can understand more than I do, but if you can spare a minute, please take a look.

EDIT3: I took another video running speedtest more times. Will post it in 5

Here it is/cfs-file/__key/communityserver-discussions-components-files/126/Spedtestvideo.mp4

CPU1 runs to 99.7% and appears to carry the load which looks like thE CPU performance is the limiting factor.

Ian

I guess you mean cpu0, but it makes no difference anyway..

It appears that yes, this is the problem.

I need to consider a new appliance. Thanks Ian.

EDIT: I double checked the video and it seems that the high numbers are for id (which I assume means idle ??)

Are we sure that the CPU is being heavily loaded? If you have some time to explain to me what to observe, please do.

EDIT2: OK just observed in the video a 99% in si on CPU1. I assume this is what you were saying...

It is an old behavior, single connections are getting loaded to a single core. Therefore, most speed tests nowadays offers multi download speedtest per default. 

If you start another speedtest or another download of a file, it will be loaded to the second CPU etc. 

Thats the behavior for years (UTM and SFOS). 

Hi Chris,

if you watch the top of the display you will note that CPU1 idle drops to 0 a number of times in the video.

Ian

Thanks for the explanation, LuCar Toni !

The fact that a single speedtest, with traffic shaping enabled, overloads a CPU core is the problem, however. If a single test could go to 480-490 then I would be OK. However, even one speedtest taxes the CPu and can go up to about 300Mbits.

So I guess I am leaving shaping disabled for now and will look into upgrading my firewall appliance with something more powerful.

rfcat_vk : 

Yes I saw it.

Thanks again, Ian!

Just wanted to follow up on this issue.

So yesterday I tried setting up another appliance that I have, which has a Celeron N3350. The Celeron is newer and has better single core score than the Atom, but the Atom is faster overall. In fact, I replaced the Celeron with the Atom and since then, the GUI in not painfully slow anymore. However I needed to verify if the better single core performance of the Celeron would make any difference and how much.

And - what do you know - performing the same tests, I was still not getting the desired speeds from the Celeron (it was hitting 100%, too), but I got speeds of about 380Mbits.

So it is 100% certain that CPU is the limiting factor in my situation. I decommissioned the Celeron appliance again, though, because as mentioned, the GUI is painfully slow. The Atom loads the pages noticeably faster. 

I am in the process of ordering a mini PC with a bunch of Intel cards and an Intel N100 CPU. I am certain that N100 will be able to handle speeds even more than 1Gbit, so I will be OK for many years.

Thanks again for all that offered their help!

Make sure they are are not i219 or 225/6 series chips and there bios is NOT UEFI.

Ian