Video conference performance drops

Hi and welcome to the UTM Community!

You'll want to be aware of Rulz, especially #1 because we still need to know what's in the IPS log when two devices are on Zoom.

Also, you didn't say what device UTM is running on or what version you're using - 9.506?

Cheers - Bob

We are running 9.509-3 Below is a screenshot of a portion of the IPS Live Log while the video conference is happening. Eth0 is LAN & Eth1 is WAN

Thanks!

With the IPS Exceptions running I dont see anything in the Sophos IPS logs. My comment about the 90% packet loss is directed to Zoom's Settings and Statistics as seen below. 

What do you see if you do tcpdump on eth0?  And then on eth1?  Are packets lost inside your network or before they reach you?

Did you learn anything from trying Sachin's suggestion?

Cheers - Bob

Below is a screenshot of some of the tcpdump output. One thing that stood out to me was the amount of packets without a listed length, could this be part of the problem?

I also ran an ifconfig before and after these tests, there was no jump in collisions or drops. There is only 1 collision shown by ifconfig which seems acceptable.

I also ran a iftop command on both interfaces to monitor bandwidth usage during the video conference. The second last column on the right (avg bandwidth per 20 seconds) showed 600Kbp~ DL and 65Kb~ UL to and from Zoom servers. Upload is noticeably bad here.

During all these tests I had the IPS Exceptions enabled

The usual approach is to put things in the /home directory unless they're large.  In this case, you could test with a 100MB file and then delete it from /home when you're done.

Strange that there are those zero-length packets without ports.  What do folks in the Zoom online community say about your packet capture?

Cheers - Bob

I haven't engaged Zoom support but certainly can reach out to them.

I also just downloaded a 120Mb file to the Sophos which downloaded at 7.68M/s (15 Seconds)
I feel like I should be seeing faster downloads speeds.  I checked the Sophos Web Admin to see how much activity there was at the time of downloading the file, we were using less than 2Mbps. Speed tests show I can get up to 50Mbps+ DL.

You said your connection is 65 Mbps - 7.68 MB/s is faster than that...

Cheers - Bob

Ah ok, I wasnt sure if the M stood for Megabits or Megabytes. Either way ill check in with Zoom support

So ive talked with Zoom support and they dont see any issues on their end. During my testing with them I put one of the 3 computers onto a completely separate network and WAN connection, the issue still occurred. Zoom recommended I contact our ISP but I dont think thats really the problem as we arnt pinning our available download bandwidth. I dont expect that they would be throttling Zoom and certainly not UDP traffic. Ill start running some WireShark captures to see if I can notice any other oddities.

Ok so I've made some interesting discoveries but still no solution to this.

In my testing I have put two devices on a completely separate WAN\Network and then left one on our main network while running some test meetings. 

Once the 3rd device joins the meeting, Zoom clients shift the meeting from each other's WAN IP's to Zoom servers. Its at this point that I notice a large amount of OUTBOUND Packet fragmentation as seen in the screenshot. It gets to the point where every second OUTBOUND UDP packet from our network is fragmented and reassembled. Inbound packets are not fragmented nor is there a drop in external video quality.

I adjusted our WAN MTU as I noticed that it was set to 576 for some reason, its now 1500 but still no luck.

Before the 3rd device joins the meeting there is little to no packet fragmentation and packet lengths can be 1100~ (bigger than some packets sent during the 3 person meeting).

WireShark Packet captures in a completely separate network show none of this fragmentation.

Any other thoughts?

Thanks!

576?  Shame on your ISP!  Still, you should check with them to see if 1500 is correct for your connection.

Then, if the problem is with outbound packets, try setting the MTU on the internal interface to a lower number and then move up until you see what begins to cause fragmentation.  Any luck with that?

Cheers - Bob

576?  Shame on your ISP!  Still, you should check with them to see if 1500 is correct for your connection.

Then, if the problem is with outbound packets, try setting the MTU on the internal interface to a lower number and then move up until you see what begins to cause fragmentation.  Any luck with that?

Cheers - Bob

So I was unable to change the MTU as the Interface would automatically change back to 576. However after changing our ISP a couple days ago I've been able to adjust it. Ive changed the WAN MTU to 800 after testing streaming and other types of web browsing, this seemed to suit our needs as 1500 would create some buffering problems for some sites. I also found that speed tests wouldnt go as high as they should so I created a Test LAN Interface and set its MTU to 768. 768 was the largest frame size I could use to ping Zoom servers and with Dont Fragment option set.

Speed tests are now showing 300 DL and 300 UL and using wget confirms these speeds. Yet despite all this and jacking directly into this Test LAN interface (bypassing our LAN's HP Switch) Outbound Zoom traffic is still fragmenting and meeting quality is awful. 

Ill continue to test but wanted to provide an update.

Thanks!

Hello Winter7undra,

Im curious if you came to a conclusion on this problem, I'm having the exact same issue, except its concerning conferences in MS Teams.

One-on-one works flawlessly, but 3 or more participants leads to massive packet drops (some Teams report says up to 98%) at our end.

Hey Paul,

The issue was two fold. The WAN MTU being too small and something on our ISP's end kept changing it back to a lower value. Because of other reasons we switched ISPs and the MTU remained at 1500 which fixed the issue. Hope that helps :)

Hi Paul,

Did you get anywhere with this?

I have exactly same issue, I have 2 WAN connections on the UTM and if I NAT teams UDP traffic it out via one line it is fine, but not the other or leaving it load balancing.

Awrite!  Welcome to the UTM Community, Alistair!

Please show pictures of the Edits of any relevant Multipathing rules.

Cheers - Bob

Hi,

In my case it was the UDP Flood Protection that caused the issue (under Intrusion Protection \ Anti D-dos/Flooding). This is disabled by default, but enabling it with values roughly 2000 pps (Source) / 3000 pps (Dest.) should make conferences bearable. I have not done any further experimentation, no more complaints so far.

I also enabled "TCP window scaling" under Firewall - "Advanced" tab.

This helped us. Our TCP window scaling and UDP flood protection were already turned on, but UDP flood protection was set to 200/300 pps source/destination, respectively. We then set it to 2000/3000 and this helped a bit, but we still couldn't get more than 2 people at a time without both video and audio becoming unlivable. We then turned UDP flood protection off completely and we can now add as many as we need and there is no drop in performance. 

Forgive the noob question, but what are the ramifications of leaving UDP Flood Protection off? 

I added a UDP Flood exception under Network--Intrusion Prevention--Exceptions and will report back if it helps or not.

Here are the IP's used by ZOOM as well, however, I only used the DNS Host *.zoom.us to start

https://support.zoom.us/hc/en-us/articles/201362683-Network-Firewall-or-Proxy-Server-Settings-for-Zoom