High bandwidth usage

Thanks, Kevin - youdaman!

Here's a command that searches all of the http logs in 2017 on your UTM to get the info you want.  It gives you every candidate for such an Exception:

zgrep 'deferred download status refresh timeout, removing' /var/log/http/2017/*/* |grep -oP 'url="^https?://.*?/'|sort -n|uniq -c|sort -n

The result at one client's box was:

      1 url="http://www.xxxxxxxxxxx.net/
      1 url="http://www.xxxxxxxx.org/
      2 url="http://xxxxxxxxxxx.yyyyyy.com/
     22 url="http://xxxxxxxxxx.zzzzzz.com/

Cheers - Bob

EDIT 2017-05-05: modified grep to look only at the FQDN

We do have several entries that contain that message and they appear around the same time as when the incident was happening and one entry is for an AWS IP. I tried going to the same link again and it didn't reproduce the issue, so I'm going to have to wait until it happens again to confirm. It says blocked in the log entry, but it let me go to it without issue.

THANK YOU for giving us something to look for and hopefully this ends up being the cause.

Possible offending log entry:

2017:03:22-10:02:13 asg-1 httpproxy[4783]: id="0002" severity="info" sys="SecureWeb" sub="http" name="web request blocked" action="block" method="GET" srcip="x.x.x.x" dstip="54.243.187.x" user="" group="" ad_domain="" statuscode="500" cached="0" profile="REF_DefaultHTTPProfile (Default Web Filter Profile)" filteraction="REF_DefaultHTTPCFFAction (Default content filter action)" size="0" request="0xb82ff800" url="X.com/.../FINAL_Corporate Responsibility Program Overview_2017 Refresh 02.pdf" referer="X.com/corporate-responsibility" error="deferred download status refresh timeout, removing" authtime="0" dnstime="570" cattime="1201" avscantime="0" fullreqtime="171762345" device="0" auth="0" ua="Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; Touch; rv:11.0) like Gecko" exceptions="" category="181" reputation="neutral" categoryname="Marketing/Merchandising" country="United States" content-type="application/pdf"

Our distributor is entitled to the thanks, the fast growing data partition was too much for me to analyze, so I opened a case if they know a possible reason for that behaviour;-)

The high bandwidth usage was annoying but not the main problem that day, our icinga monitoring lost ping connections permanently but fortunately the customer uses mainly local ressources and email :-)

Here is a screenshot of the partition growing constantly that day (SG135), reason was an automatic update of a program, they were currently testing on 2(!) PCs only.

We were experiencing this too. It does make you scratch your head and it's a little hard to trace. Wouldn't it be great is there was some sort of log and cleanup for items like this?

I'm wondering if this expression could be sent to syslog and then our syslog alert us to it?

Also, if you do experience these symptoms and do find the regex specified in there, what actions do you take to rectify?

I recently experienced very similar symptoms (as per the original post) using a Sophos XG (home edition) with a newly purchased Amazon Fire HD 8 (2017) tablet.  Sophos XG reports indicated the Fire 8 HD's DHCP ip address, with video as the file type.  I talked with Amazon tech support but they were unable to figure this out on their end.  Per this thread, I created a web exception for cloudfront.net and that specific ip address (only), which appears to have solved the problem for now.  Without the exception, the only resolution was to turn off the Fire HD 8's wifi and to reboot the modem, as the traffic persisted until the modem was rebooted.

This isn't the right place to get help for XG.

Cheers - Bob

Agree with sublime.  The culprit is the Amazon Firestick.  Playing a video from Amazon Firestick causing a huge bandwidth hog and CPU runs up to 90% and everything comes to a halt.  Using Sophos UTM (home license)

Thanks