This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

High bandwidth usage

Hi all,

Yesterday we started getting high bandwidth usage showing on the external interface, almost all bandwidth taken.  It's coming from cloudfront.net servers and all I can see is the external interface is the client and its unclassified traffic on tcp/80.

I've throttled it based on the cloudfront IP ranges to stop the impact.

I can find a way to see if it's a user doing something as the logs show no high activity from specific internal clients.

Any idea what can be done to track this down?

Thanks


This thread was automatically locked due to age.
Parents
  • We've had this same issue, twice in under a week. The external WAN interface maxes out our pipe ~100mbit, but we don't see anything close to the same amount of corresponding traffic/usage on any of the internal interfaces. It usually lasts about 30 minutes and makes any internet related activity basically unusable.

    The previous ideas about proxy/cache make sense, but has anybody figured out what specific application or process is causing it?

    SG310

    Firmware 9.408-4

  • Steve, did you see my post on 8 Apr 2015 in this thread?

    Cheers - Bob

     
    Sophos UTM Community Moderator
    Sophos Certified Architect - UTM
    Sophos Certified Engineer - XG
    Gold Solution Partner since 2005
    MediaSoft, Inc. USA
  • I did, but there is no corresponding traffic on the internal interfaces, which leads me to believe that the client (A/V) isn't receiving the data. If the proxy doesn't have anywhere to send the traffic why/what does it keep downloading? The flow monitor doesn't give you a specific IP for where its connecting to/from, just the IP for our WAN interface and AWS.

    If this was just one connection that gets "stuck" downloading something, it shouldn't be noticeable to users. It would just use up whatever pipe is available, but when this happens everything else slows to a crawl, similar to if somebody was using BitTorrent and making thousands of connections that are all using as much available bandwidth as possible.

    Steve

     

  • "I did, but there is no corresponding traffic on the internal interfaces, which leads me to believe that the client (A/V) isn't receiving the data." Exactly.  That's the phenomenon I was describing.  The problem is that the sending server times out so the Proxy restarts the download.

    Cheers - Bob

     
    Sophos UTM Community Moderator
    Sophos Certified Architect - UTM
    Sophos Certified Engineer - XG
    Gold Solution Partner since 2005
    MediaSoft, Inc. USA
  • Do you have several hits for 'deferred download status refresh timeout, removing' when you search your Websecurity logs?

    I had one customer looping a download last week until the data partition was nearly full of broken downloads. After getting the hint to search for that phrase in websecurity logs I was able to identify WHAT was tried to be downloaded and was never successful I could create an exception to stop that bevaviour.

    I could only get rid of the nearly 15G 'trash downloads' by clearing the proxy cache (also caching is not even active on that UTM).

    Gruß / Regards,

    Kevin
    Sophos CE/CA (XG+UTM), Gold Partner

Reply
  • Do you have several hits for 'deferred download status refresh timeout, removing' when you search your Websecurity logs?

    I had one customer looping a download last week until the data partition was nearly full of broken downloads. After getting the hint to search for that phrase in websecurity logs I was able to identify WHAT was tried to be downloaded and was never successful I could create an exception to stop that bevaviour.

    I could only get rid of the nearly 15G 'trash downloads' by clearing the proxy cache (also caching is not even active on that UTM).

    Gruß / Regards,

    Kevin
    Sophos CE/CA (XG+UTM), Gold Partner

Children
  • Thanks, Kevin - youdaman!

    Here's a command that searches all of the http logs in 2017 on your UTM to get the info you want.  It gives you every candidate for such an Exception:

    zgrep 'deferred download status refresh timeout, removing' /var/log/http/2017/*/* |grep -oP 'url="^https?://.*?/'|sort -n|uniq -c|sort -n

    The result at one client's box was:

          1 url="http://www.xxxxxxxxxxx.net/
          1 url="http://www.xxxxxxxx.org/
          2 url="http://xxxxxxxxxxx.yyyyyy.com/
         22 url="http://xxxxxxxxxx.zzzzzz.com/

    Cheers - Bob

    EDIT 2017-05-05: modified grep to look only at the FQDN

     
    Sophos UTM Community Moderator
    Sophos Certified Architect - UTM
    Sophos Certified Engineer - XG
    Gold Solution Partner since 2005
    MediaSoft, Inc. USA
  • We do have several entries that contain that message and they appear around the same time as when the incident was happening and one entry is for an AWS IP. I tried going to the same link again and it didn't reproduce the issue, so I'm going to have to wait until it happens again to confirm. It says blocked in the log entry, but it let me go to it without issue.

    THANK YOU for giving us something to look for and hopefully this ends up being the cause.

    Possible offending log entry:

    2017:03:22-10:02:13 asg-1 httpproxy[4783]: id="0002" severity="info" sys="SecureWeb" sub="http" name="web request blocked" action="block" method="GET" srcip="x.x.x.x" dstip="54.243.187.x" user="" group="" ad_domain="" statuscode="500" cached="0" profile="REF_DefaultHTTPProfile (Default Web Filter Profile)" filteraction="REF_DefaultHTTPCFFAction (Default content filter action)" size="0" request="0xb82ff800" url="X.com/.../FINAL_Corporate Responsibility Program Overview_2017 Refresh 02.pdf" referer="X.com/corporate-responsibility" error="deferred download status refresh timeout, removing" authtime="0" dnstime="570" cattime="1201" avscantime="0" fullreqtime="171762345" device="0" auth="0" ua="Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; Touch; rv:11.0) like Gecko" exceptions="" category="181" reputation="neutral" categoryname="Marketing/Merchandising" country="United States" content-type="application/pdf"

  • Our distributor is entitled to the thanks, the fast growing data partition was too much for me to analyze, so I opened a case if they know a possible reason for that behaviour;-)

    The high bandwidth usage was annoying but not the main problem that day, our icinga monitoring lost ping connections permanently but fortunately the customer uses mainly local ressources and email :-)

    Here is a screenshot of the partition growing constantly that day (SG135), reason was an automatic update of a program, they were currently testing on 2(!) PCs only.

    Gruß / Regards,

    Kevin
    Sophos CE/CA (XG+UTM), Gold Partner

  • We were experiencing this too. It does make you scratch your head and it's a little hard to trace. Wouldn't it be great is there was some sort of log and cleanup for items like this?

    I'm wondering if this expression could be sent to syslog and then our syslog alert us to it?

     

    Also, if you do experience these symptoms and do find the regex specified in there, what actions do you take to rectify?

  • I recently experienced very similar symptoms (as per the original post) using a Sophos XG (home edition) with a newly purchased Amazon Fire HD 8 (2017) tablet.  Sophos XG reports indicated the Fire 8 HD's DHCP ip address, with video as the file type.  I talked with Amazon tech support but they were unable to figure this out on their end.  Per this thread, I created a web exception for cloudfront.net and that specific ip address (only), which appears to have solved the problem for now.  Without the exception, the only resolution was to turn off the Fire HD 8's wifi and to reboot the modem, as the traffic persisted until the modem was rebooted.

  • This isn't the right place to get help for XG.

    Cheers - Bob

     
    Sophos UTM Community Moderator
    Sophos Certified Architect - UTM
    Sophos Certified Engineer - XG
    Gold Solution Partner since 2005
    MediaSoft, Inc. USA
  • Agree with sublime.  The culprit is the Amazon Firestick.  Playing a video from Amazon Firestick causing a huge bandwidth hog and CPU runs up to 90% and everything comes to a halt.  Using Sophos UTM (home license)

     

    Thanks