This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

Google & "UDP flood" action

Hello

So to get straight to the point, I'm running Sophos UTM (FW Ver.: 9.203-3, Virtual) Home License and, as the thread title shows, browser-based Google products are affected by the IPS and some of its traffic are being tagged by the IPS as "UDP flood" firewall rule 60013, which is to Drop UDP_FLOOD attempts

As a result, some Google products will be capped at 2mbps download speeds. Strangely enough, it only happens to one wired client and not any virtualised clients nor wireless clients (so this means neither the LG Smart TV running YouTube nor the Android smartphone running Google Maps experience this issue.) When this plays out:

  • YouTube will load videos at 2mbps, causing buffer to 1080p videos and less often to 720p videos; and
  • Google Maps will load its chunks of map and image data slowly


Now, I can definitely turn off UDP flood protection, but that leaves a gigantic gap on my network. It's probably not the best practice when the UTM is responsible as the gateway between the Internet and my network at home. You now understand why this is probably something I would consider to avoid. I had disabled it for a minute and it definitely increased the loading speeds to what my ISP provides, which is 30x more than what it was throttling me to. As of right now, it's enabled.

Has anyone else experienced this issue? Has anyone found a fix for this? This started happening probably around the time where the OpenSSL Heartbleed vulnerability was discovered, if not a month or two before it.

UPDATE: Alright, so here's what's getting hit by IPS so that we all have that general idea...

IP of Google is the source IP. 
WAN IP is the destination.
Action is UDP Flood
Source Port: 443
Destination Port: Some random port on the 50000~60000s. 

Last time I checked, 443 isn't exactly UDP for the nature of what's being transported and a corporation like Google would keep atop for any such UDP floods to prevent it from happening.

Second Update
:
Please don't tell me this is too difficult. It has to be some simple explanation.
2014:07:14-16:40:20 core ulogd[4786]: id="2105" severity="info" sys="SecureNet" sub="ips" name="UDP flood detected" action="UDP flood" fwrule="60013" initf="eth1" srcmac="[Source MAC]" dstmac="[Destination MAC]" srcip="206.111.13.173" dstip="[My WAN IP]" proto="17" length="1228" tos="0x00" prec="0x00" ttl="57" srcport="443" dstport="55971"


Thanks


This thread was automatically locked due to age.
Parents
  • Hello

    I have found the solution for this, just add these google addresses as an exception to udpflood

    • ip4:64.18.0.0/20
    • ip4:64.233.160.0/19
    • ip4:66.102.0.0/20
    • ip4:66.249.80.0/20
    • ip4:72.14.192.0/18
    • ip4:74.125.0.0/16
    • ip4:108.177.8.0/21
    • ip4:173.194.0.0/16
    • ip4:207.126.144.0/20
    • ip4:209.85.128.0/17
    • ip4:216.58.192.0/19
    • ip4:216.239.32.0/19
    • ip6:2001:4860:4000::/36
    • ip6:2404:6800:4000::/36
    • ip6:2607:f8b0:4000::/36
    • ip6:2800:3f0:4000::/36
    • ip6:2a00:1450:4000::/36
    • ip6:2c0f:fb50:4000::/36

     

     

     

    Regards

     

     

  • Hi, Moises, and welcome to the UTM Community!

    Your solution will work perfectly, but I prefer to use the UDP 443 Exception as that makes the Exception more specific.

    Cheers - Bob

     
    Sophos UTM Community Moderator
    Sophos Certified Architect - UTM
    Sophos Certified Engineer - XG
    Gold Solution Partner since 2005
    MediaSoft, Inc. USA
  • Sorry to resurrect an old thread.  Seems there's two schools of thought to dealing with this issue.  Either exempt port 443 udp in either direction, or add a bunch of hosts to the exemption along with the service port.  Neither is an ideal solution because the first opens that port up to potential flooding from any external ip (?), while the latter has constantly changing hosts.

    I suppose the former (exempt the service port only) is the better of the two in that this exemption is valid for requests initiated behind the firewall.  Uninitiated external port 443 requests will still be blocked.  Bob am I correct in this understanding?

  • Hi Jay,

    that is Google’s QUIC protocol. Close outgoing udp port 443. This protocol bypass the proxy, don't allow it in a productive environment. When closing the udp port chrome is going to use tcp.

    Google’s QUIC protocol:

    https://ma.ttias.be/googles-quic-protocol-moving-web-tcp-udp/

    Regards

    mod

  • Agreed with the MoD, but I also prefer to add UDP 443 to the 'Allowed Target Services' in Web Filtering. That allows browsers using the explicit proxy to benefit from the increased throughput of https over UDP.   Using this means that you have the block as mod prescribes and the Anti-UDP Flooding Exception as described above.  In any case, there's no reason to open UDP 443 inbound.

    Cheers - Bob

     
    Sophos UTM Community Moderator
    Sophos Certified Architect - UTM
    Sophos Certified Engineer - XG
    Gold Solution Partner since 2005
    MediaSoft, Inc. USA
Reply
  • Agreed with the MoD, but I also prefer to add UDP 443 to the 'Allowed Target Services' in Web Filtering. That allows browsers using the explicit proxy to benefit from the increased throughput of https over UDP.   Using this means that you have the block as mod prescribes and the Anti-UDP Flooding Exception as described above.  In any case, there's no reason to open UDP 443 inbound.

    Cheers - Bob

     
    Sophos UTM Community Moderator
    Sophos Certified Architect - UTM
    Sophos Certified Engineer - XG
    Gold Solution Partner since 2005
    MediaSoft, Inc. USA
Children
  • Hi Bob,

    I've never though about that. Do you have experience with UDP 443 over the proxy? I've never tested that.

    regards

    mod

  • Bob, so to clarify, what does this rule look like in "Allowed Target Services" ?

    Also, by having it listed as an exception in intrusion prevention, doesn't the firewall still block inbound 443/udp requests?  Or because it's listed as an allowed service, it never makes it to the firewall which has no inbound 443/udp rule defined?

  • Intrusion Prevention only happens to traffic allowed in.  It's unlikely that you have a web server using UDP 443, so no such traffic will even be seen by the Exception.

    As #2 in Rulz explains, the implicit Allow rule in Web Filtering is used before the manual firewall Block rule is considered.

    Cheers - Bob

     
    Sophos UTM Community Moderator
    Sophos Certified Architect - UTM
    Sophos Certified Engineer - XG
    Gold Solution Partner since 2005
    MediaSoft, Inc. USA
  • Bob, with the rule set up as above I'm still getting massive IPS flood detection in the ips log on downloads with resulting speeds ~2-3 mbps.  For uploads there are no flood entries in the ips log and speeds are good, ~400-700 mbps.  Test is done using google drive in windows chrome browser to upload and dl large files.

    So it seems at least outbound the rule is properly respected but not inbound. 

    It almost seems like there needs to be connection tracking helper for this to work right to establish an inbound udp connection.

    Of course at this point, the best solution seems to be to just block udp/443 outbound altogether so chrome falls back on tcp.

    Also, I do want to point out, the issue isn't only with chrome browser.  For some time now i've had really slow google photo uploads (300-400 KBps).  This is related to this topic because once udp/443 out is blocked, uploads are happening at 7-10 MBps (megabytes/sec).

    Google drive on android seems to be stuck to around 2-3 MB/s in either direction regardless of how 443/udp is configured.

  • "still getting massive IPS flood detection "

    UDP 443 requests have source ports in 1:65535.  UDP 443 responses have 443 as the source port and destination ports in 1:65535.  You can help future visitors here by showing a picture of the Edits of your Exception and of the UDP service you defined.

    If Web Filtering is in Transparent mode, I would block UDP 443 in a firewall rule.

    Cheers - Bob

     
    Sophos UTM Community Moderator
    Sophos Certified Architect - UTM
    Sophos Certified Engineer - XG
    Gold Solution Partner since 2005
    MediaSoft, Inc. USA
  • Bob,

    Web filtering is in transparent mode.

    Here's the rules I have defined for allowed target service.

    This works fine for uploads but floods IPS with UDP flood notices when downloading.

    To eliminate use of QUIC entirely I have this firewall rule.

    Had to create separate service objects because allowed target service would not accept a UDP only definition. TCP/UDP was accepted.

    Are you saying QUIC would work correctly with the allowed target services entries if web filtering was off or not in transparent mode?

  • The article mentioned earlier was certainly helpful.

    The most startling point was the observation that QUIC  provides a replacement for TLS 1.2.    Very smart people have performed tremendous research into TLS in recent years, producing a long strain of bad news that has forced us to migrate to TLS 1.2 from everything older, to discard MAC algorithms, and to discard older ciphers.    Who is vetting the QUIC encryption layer -- just Google?   Do business users really want to assume that Google has figured all of this out?

    Also based on the article, it appear that QUIC will always bypass https inspection (decrypt-and-scan) in the UTM web proxy, and probably most other brands as well.

    For file transfer applications like Google Drive, I wonder if the performance benefit is noticeable, other than on cell phones. File transfers imply long session, and TCP provides larger packet sizes.   At some point it would seem that the larger packet size would dominate the initial setup.   TCP might lose if the connection delivers a lot of packets out of order.

    The comments about UDP flood alarms suggest that by its very nature, QUIC makes connection tracking at the firewall into a very difficult task.   Since the plan is to update the protocol frequently, firewalls will have this problem on a continuing basis.

    I think I will block UDP 443 and live with the less optimal performance.

  • It's 2000 hours here.  I'd say google drive is doing pretty well with tcp.

    Left waveform is upload, right is download.

    What's interesting, on a modern android cell phone (snapdragon 820, 4gb ram, ac wifi capable of 300 mbps sustained speeds), I can only upload/download at 2-3 MB/s (16-25 mbps).  Google photos does better at about 80 mbps.

  • This whole discussion raises a larger security issue.

    Google has demonstrated that UDP can be useful for talented developers to roll a custom protocol that bypasses normal security checks.    It brings back a recollection of reading that some malware has also been seen using UDP to implement a hidden protocol that bypasses normal firewall security.

    Google's research, referenced in the QUIC article, indicates that most organizations do not block outbound UDP, and most firewalls allow UDP responses.   Organizations are not blocking outbound UDP because it is so hard to build the expertise about"normal" outbound traffic to know what must be allowed before implementing an outbound block.

    It is probably time for the industry to start blocking UDP by default.   If you are forwarding all external DNS traffic to Quad9, Google, Norton DNS, or your ISP, then you don't even need to allow UDP 53 globally, just to and from those servers.

    But I will have to collect some logs to know what other UDP traffic lurks on my network.   Fortunately, UTM has the tools to do so.

  • The QUIC Response doesn't need to be in 'Allowed Target Services' or in your Firewall Block rule.  It does need to be in an Exception for Anti-UDP Flooding for "Internet IPv4 -> QUIC Response -> Any" traffic.

    Cheers - Bob

     
    Sophos UTM Community Moderator
    Sophos Certified Architect - UTM
    Sophos Certified Engineer - XG
    Gold Solution Partner since 2005
    MediaSoft, Inc. USA