High bandwidth usage

Question

Hi all,

Yesterday we started getting high bandwidth usage showing on the external interface, almost all bandwidth taken. It's coming from cloudfront.net servers and all I can see is the external interface is the client and its unclassified traffic on tcp/80.

I've throttled it based on the cloudfront IP ranges to stop the impact.

I can find a way to see if it's a user doing something as the logs show no high activity from specific internal clients.

Any idea what can be done to track this down?

Thanks

This thread was automatically locked due to age.

kerobra_retired · Accepted Answer

Do you have several hits for 'deferred download status refresh timeout, removing' when you search your Websecurity logs? 
 I had one customer looping a download last week until the data partition was nearly full of broken downloads. After getting the hint to search for that phrase in websecurity logs I was able to identify WHAT was tried to be downloaded and was never successful I could create an exception to stop that bevaviour. 
 I could only get rid of the nearly 15G 'trash downloads' by clearing the proxy cache (also caching is not even active on that UTM).

BAlfson · Answer

Thanks, Kevin - youdaman! 
 Here's a command that searches all of the http logs in 2017 on your UTM to get the info you want. It gives you every candidate for such an Exception: 
 zgrep 'deferred download status refresh timeout, removing' /var/log/http/2017/*/* |grep -oP 'url="^https?://.*?/'|sort -n|uniq -c|sort -n 
 The result at one client's box was: 
 1 url=" http://www.xxxxxxxxxxx.net/ 1 url=" http://www.xxxxxxxx.org/ 2 url=" http://xxxxxxxxxxx.yyyyyy.com/ 22 url=" http://xxxxxxxxxx.zzzzzz.com/ 
 Cheers - Bob 
 EDIT 2017-05-05: modified grep to look only at the FQDN