This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

Outgoing Google Hangouts video no longer works

Howdie,

I have a problem that's cropped up with Sophos UTM in recent months and was hoping to get some input from others on it. I've been using UTM since the astaro days (Astaro 6 maybe? Don't recall when I first set it up) when the home user IP limit was 10. I still think it's the best solution out there for firewall, IPS, etc. Which is why this is frustrating. I don't want to switch. :) I'm currently running UTM on a quad port Qotom PC (specifically, this guy: https://smile.amazon.com/gp/product/B01AAKGQQG/ref=oh_aui_detailpage_o05_s00?ie=UTF8&psc=1) and before that an older fit-PC (a celeron dual NIC version). I use UTM as my gateway to my ISP (Comcast using the "Blast" speeds which are ~12 up and 150 down).

Up until about a month or so ago I was able to use google hangouts without a problem (i.e., teleconferencing for work). But, recently my outgoing video won't work for hangouts. I can see my video locally, but it won't show up for anyone remotely. This only happens on my network and on any computer / phone / device I use on my network to access a hangout. I assumed I had changed something that broke it.

After much finagling, I couldn't find any setting in UTM that would fix it so I made a backup and reset UTM to factory settings. It still didn't work. So...I put in the most basic of firewall rules (Internal (network) -> Any protocol -> Any) thinking maybe something outgoing was being blocked. Nope. I rechecked to make sure all proxying, filtering, etc was off. Yep, all off. In desperation, I SSH'd to the box (after enabling SSH) and manually cleared all iptables rules (left the nat table untouched however so I still had internet access). Nope, still doesn't work.

To rule out some kind of ISP issue, I took UTM out of the loop entirely and set my Linksys AC1200 router up as my internet gateway. THAT worked. Others could then see my video just fine. Seeing as how I did not have this problem previously with UTM, my only conclusion is that some update has done something to muck up google hangouts outgoing video (I try to apply UTM updates pretty regularly). I'm not clear where this would happen however. I don't know exactly how hangouts video works, but I understand it uses STUN for NAT traversal. Although I believe it has several other protocols it falls back on. The linksys router is quite likely not linux based (probably vxworks) so probably handles NAT a little differently. But...seeing as how this used to work with UTM I'm assuming it's from a UTM update. Unless Google changed something...which would make me very sad.

So, anyone have any idea what might be going on here? 

Also, could some kind soul please try using a hangout on a network behind an up to date UTM (I'm currently using 9-405.5...I haven't yet applied the 9-406.3 update that just came out today) and see if you get video. You can easily replicate the problem by using two browsers (e.g., Chrome and Safari) on the same hangout (it won't work with two tabs in the same browser...hangouts prevents it).

NOTE: Though I never had these rules before, as a troubleshooting exercise I tried opening up ports referenced in here: https://support.google.com/a/answer/1279090?hl=en. That also didn't help. Which isn't surprising since that one rule I added in allowed all outgoing internal traffic anyway.



This thread was automatically locked due to age.
Parents
  • Also,

    One update I have. While looking into this more I discovered that chrome has a built in utility for debugging webrtc stuff (which is what hangouts uses). You just go to chrome://webrtc-internals/. It's pretty cool. There are lots of graphs in there and of particular interest was a graph labeled packetsLost for the "Sending video" stream which gives # of packets lost over time for video sending. When I use hangouts behind UTM that graph is forever increasing. The stats shown for the video stream indicate ~60% of all video packets are lost over time. When I use hangouts from work (different network without UTM) or behind my linksys router, that packetsLost graph is mainly flat with only occasional lost packets. So, something somewhere is definitely causing the packets to not get send (either dropped, rejected, not routed...something).

    Cheers,

    John

Reply
  • Also,

    One update I have. While looking into this more I discovered that chrome has a built in utility for debugging webrtc stuff (which is what hangouts uses). You just go to chrome://webrtc-internals/. It's pretty cool. There are lots of graphs in there and of particular interest was a graph labeled packetsLost for the "Sending video" stream which gives # of packets lost over time for video sending. When I use hangouts behind UTM that graph is forever increasing. The stats shown for the video stream indicate ~60% of all video packets are lost over time. When I use hangouts from work (different network without UTM) or behind my linksys router, that packetsLost graph is mainly flat with only occasional lost packets. So, something somewhere is definitely causing the packets to not get send (either dropped, rejected, not routed...something).

    Cheers,

    John

Children
  • John, if you didn't look at the Intrusion Prevention log, you can't know if Anti-UDP Flooding activity is causing your problem.

    Cheers - Bob

     
    Sophos UTM Community Moderator
    Sophos Certified Architect - UTM
    Sophos Certified Engineer - XG
    Gold Solution Partner since 2005
    MediaSoft, Inc. USA
  • I was not aware udp flood protection would still happen with IPS off. All three DoS protections are actually unchecked in the IPS config as well. But, just the same, I just double checked all four logs while replicating the problem. IPS is completely empty as is the Advanced Threat Protection log (also off anyway). The Application Control log has a few entries of the form:

    2016:09:24-16:57:37 astaro afcd[11313]: _afc_conn_get_age(): The timestamp of connection 0x00043600 is in the future; correcting ...

    But, they're all from much earlier today. NOTE: I do have application control on right now as I've since restored my normal configuration after resetting to factory defaults since resetting address the problem.

    The firewall log does have blocked items in my current configuration, but I don't think any of them are related to this problem. Mostly the items I already mentioned in the original post with one additional guy that's popping up a lot (whether replicating this problem or not) from a btsync box I have running. That's a thing to look into for another day.

    -John

  • Out of curiosity, would you be willing to try a hangout to test if you're seeing this problem (assuming you're behind a UTM box)? It'd be nice to verify I'm not crazy. :) That would at least narrow down if there really is something specific to my setup I've missed with a factory reset or if there's something different from a couple months ago. This definitely used to work...I've been using hangouts while working from home since 2010 and I've been using Astaro for a lot longer than that. :)

    I've found using the chrome://webrtc-internals page in chrome that you don't even have to connect to another person. You can see the packet loss directly in the tool in the graphs at the bottom if you expand the send (video) graph. You just need to be connected to a hangout (no one else has to be on it). On a non-UTM network the packet loss graph is mostly flat and a very low number. It's monotonically increasing on my network showing a lot of packet loss. E.g., here's an example with being on a hangout for only about 2 minutes. The packetLoss graph shows a steady positive slope and is already getting up near the thousands.

  • John, I'm beginning to suspect a hardware issue or an MTU setting.  Does #7 in Rulz help?

    Cheers - Bob

     
    Sophos UTM Community Moderator
    Sophos Certified Architect - UTM
    Sophos Certified Engineer - XG
    Gold Solution Partner since 2005
    MediaSoft, Inc. USA
  • I actually tried almost all of the suggestions in #7 before I even found the Rulz post. :) Specifically, #1 doesn't apply (Intel NICs on both machines I used), #2 is ok (settings already off after reset), #4 done, #5 done, #6 done (I suspected the cable modem couldn't handle throughput...I was wrong, but now I have a backup!), and #8 (tried a different port).

    The only two I didn't try are MTU settings (#3) and link settings (#7). I can try those later today when not working. Though I feel like it is a very specific form of packet loss (i.e., basically all hangouts video packets). I would expect all of sorts of connectivity issues in such an instance of a hardware problem rather than only hangouts video.

    By any chance, do you have the ability to try this for yourself behind a UTM router? I'd really love to see if someone else can replicate this issue. Starting a google hangout is very easy (assuming you have a google account :)) E.g., this is one I just created from https://hangouts.google.com/hangouts which I think anyone can access: https://hangouts.google.com/hangouts/_/7zydsosg4vgcnjw4hlrgi2mfnae?authuser=0&hl=en. As I mentioned in a past reply, you can verify the problem by joining that with two different browsers (e.g., firefox and chrome) and observing the lack of video for the caller from the opposite browser in each browser (you can always see your own video).

     

    -John

  • Bob,

    I've finally fixed it. It was an MTU setting. I finally discovered it by running tcpdump on both the internal and external IFs in two terminals with a hangouts session going. What I found was something like this:

    What I found was that on the local network I'd have packets like this:

    12:22:06.565503 IP 172.<OBSCURED FOR SECURITY>.61021 > 74.125.134.127.19305: UDP, length 855

    and on the external IF I'd see packets like this:

    12:22:06.565605 IP 66.<OBSCURED FOR SECURITY>.61021 > 74.125.134.127.19305: UDP, length 855
    12:22:06.565617 IP 66.<OBSCURED FOR SECURITY> > 74.125.134.127: ip-proto-17

    The ip-proto-17 caught my attention. With a little research that lead me to believe the packets from the internal interface were getting fragmented when going to the external IF. Looking in sophos I found the MTU of the external IF was set to 576. I attempted to override it to 1500, but it would just revert again.

    Which lead me to this post: https://community.sophos.com/products/unified-threat-management/f/hardware-installation-up2date-licensing/79288/disable-bad-bugfix-in-9-405-5-fix-nutm-2840-aws-utm-ignores-mtu-sent-by-dhcp-server

    with a discussion about auto-mtu-discovery being added to UTM. Which seemed to basically force the MTU reported by the DHCP provider. Which lead to this later post: https://community.sophos.com/products/unified-threat-management/f/hardware-installation-up2date-licensing/80641/sophos-utm-9-407-3-released talking about a flag to disable that now being exposed in CC.

    And, specifically, this comment with instructions on how to flip that flag off on an interface: https://community.sophos.com/products/unified-threat-management/f/hardware-installation-up2date-licensing/80641/sophos-utm-9-407-3-released#306734

    It would be REALLY nice if Sophos could add a UI option to flip that flag in the advanced interface configuration section as it's fairly complex to flip and a very easy step to forget about.

    I appreciate your help in looking into this and hope it helps for anyone else running into a similar issue.

    NOTE: In case that link with the steps to flip off the auto-mtu-discovery flags is ever lost, these are the steps from the linked comment:

    >Thanks Bulirich, I tried it and it works hooray :-)

    >

    >The fix:

    >Login as loginuser then root in ssh shell:

    >cc 
    >RAW 
    >lock_override 
    >OBJS 
    >interface 
    >ethernet (or cable, or other type) 
    >REF_ (Tap TAB two times - then you can see the interface list. Mine is called "REF_IntCabExternaWan[WAN,interface,ethernet]"
    >(You will get a look like this:)

    >'additional_addresses' => [],
    >'bandwidth' => 0,
    >'comment' => 'Added by installation wizard',
    >'inbandwidth' => 100000000,
    >'itfhw' => 'REF_ItfEthEth1',
    >'link' => 1,
    >'mtu' => 576,
    >'mtu_auto_discovery' => 1,
    >'name' => 'WAN',
    >'outbandwidth' => 20000000,
    >'primary_address' => 'REF_ItfPri000024',
    >'proxyarp' => 0,
    >'proxyndp' => 0,
    >'status' => 1
    >}

    >Then write:

    >mtu_auto_discovery=0 
    >w  (write the changes) 

    >Now go into Webadmin and find the WAN link, change the MTU under Advanced to 1500 and voila! :-)

    >----

    >Best regards Martin ;-)

  • As ancient as this is I just got caught up in this bug again. This must be the third time. Low and behold, MTU was 576. Kids couldn't do their google meet after upgrading to 702. Pulled my hair out all day trying to figure this out and it was this bug again. The fix is the same and still works.

  • For the last few years, the 'Advanced' section of Interface definition allows you to specify a fixed MTU, but it gets set back to the false number.  See my post from almost two years ago to see a cleaner way to prevent that.

    Cheers - Bob

     
    Sophos UTM Community Moderator
    Sophos Certified Architect - UTM
    Sophos Certified Engineer - XG
    Gold Solution Partner since 2005
    MediaSoft, Inc. USA
  • That the thing, I did that fix when you posted it and the setting got reset! Again, I can only assume 702 brings in a new version of that file perhaps but that's only speculation and I don't currently have time to investigate.

  • If you followed the interactive cc approach above, then the single command line my post suggests would work.

    Cheers - Bob

     
    Sophos UTM Community Moderator
    Sophos Certified Architect - UTM
    Sophos Certified Engineer - XG
    Gold Solution Partner since 2005
    MediaSoft, Inc. USA