This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

Partial content (Range) rejected by proxy after upgrading to 9.6

After upgrade from 9.510-5 to 9.600-5, proxy started to respond with status "416 Requested range not satisfiable" to any request that has to be scanned with AV and includes Range: header. In /var/log/http.log, the denied request has reason="range". Sniff from the outside wire shows that the request from proxy to webserver is processed correctly.

 

Workaround: After putting problematic URL patterns to web filtering exception rule with AV scanning turned off, the requests are processed as before upgrade to 9.6.

 

Is this intended behavioral change in 9.6? I know, the partial content serving can be abused to bypass AV scanning, so generally it is good idea to handle requests with partial content differently and not to pass their responses without scanning, but IMHO there should be more sophisticated algorithm at the proxy to deal with this situation:

1) send HEAD to determine the content size, whether it is within size limit for scanning;
2) if it is over the size limit, process request without scanning;
3) if it is within the size limit, request whole content (and possibly cache it for subsequent range requests), scan it and serve partial content to the client.

 

OH

 



This thread was automatically locked due to age.
  • Hi!

     

    I've the same problem. I usually only is a real problem when streaming content. To get rid of "reason=range" in web proxy I've to disable nearly all the checks instead of only deactivating "AV Bypass".

    @Sophos: could please someone fix this? I installed the latest version (ftp.astaro.com/.../u2d-sys-9.600005-601005.tgz.gpg) of UTM today and the problem is still not fixed.

    I really don't like to change all my proxy settings just because of this bug, because when it gets fixed somewhen it's likely to happen unnoticed. So, using all the available security features in future for the workaround'ed devices will never be enabled again.

    And now think about companies using UTM instead of private customers ...

  • Hallo and welcome to the UTM Community!

    Have you tried selecting 'Bypass content scanning for streaming content' on the 'Misc' tab?

    Cheers - Bob

     
    Sophos UTM Community Moderator
    Sophos Certified Architect - UTM
    Sophos Certified Engineer - XG
    Gold Solution Partner since 2005
    MediaSoft, Inc. USA
  • Yes this is intended.
     
    The way that 9.5 behaved was to strip the range header and pass it along.  So if you requests from 1MB to 2MB of a 20MB file, in 9.5 it served back the 20MB file.  This caused some things to work (albeit poorly) and other things to not work (and often saturate the connection trying to download again and again).
     
    The two most common uses of range requests are:
    1) Streaming media.  In this case, you can use the option 'Bypass content scanning for streaming content'.
    2) Background downloaders.  This include microsoft updates using BITS as well as other things that try and download things in small chunks.  This is best resolved by, if you trust the source company, creating an exception for the destination to turn of AV scanning.
     
  • "This is best resolved by, if you trust the source company, creating an exception for the destination to turn of AV scanning."

    We are getting this problem and users are having problems browsing to PDF files. We don't want to turn off AV scanning.

  • In some scenarios (pdf.js or Chrome or some browser extensions) the browser may try to make multiple requests, which can include range requests.  They are trying to display the first page quickly and then download the rest of the PDF in the background.

    With UTM 9.6 AFAIK we correctly will fail the request with 416 and set Accept-Ranges: None when we do not allow it.  The browser/plugin/page should then fall back to full downloads.  The end user experience should be success.  There may be a logged failure, which is normal and expected.

    What UTM version are you running, what browser are you using, what website are you visiting, and what is the full end user experience?

     

    References:

    https://stackoverflow.com/questions/32725608/chrome-sends-two-requests-when-downloading-a-pdf-and-cancels-one-of-them

    https://stackoverflow.com/questions/1817750/do-most-browsers-make-multiple-http-requests-when-displaying-a-pdf-from-within-t

  • Hello Michael.

    We are using 9.601-5. I can't find any entries in the Web Filtering logs for reason="range" before the upgrade to that.

    We use Microsoft Edge 44.17763.1.0 and Internet Explorer 11.316.17763.0.

    Various websites produce the problem. If I put the website in Web Protection - Web Filter Profiles - Filter Actions - Allowed Sites, the user gets the PDF more often but not always.

    When clicking on the PDF link, either nothing happens, a blank page appears or the PDF loads. For most people, most of the time, the PDF will load.

  • The quick answer is that we tested pdfs when we implemented this in UTM, however there are many combinations so it is possible that some combination that is not right.  But 9.6 has been out for a while and the Dev team has not heard about this from anywhere else.

    If there is a product issue, this needs to be raised through a support ticket.

  • Hi Nigel,

    How about showing us a line from the Web Filtering log where a PDF did not load?

    Cheers - Bob

     
    Sophos UTM Community Moderator
    Sophos Certified Architect - UTM
    Sophos Certified Engineer - XG
    Gold Solution Partner since 2005
    MediaSoft, Inc. USA
  • 2019:03:20-16:11:17 proxy-1 httpproxy[5548]: id="0002" severity="info" sys="SecureWeb" sub="http" name="web request blocked" action="block" method="GET" srcip="xxx.xxx.xxx.xxx" dstip="185.217.40.162" user="xxxxxxxx" group="Active Directory Users" ad_domain="xxxxxxxx" statuscode="416" cached="0" profile="REF_HttProContaInterScc (Corporate)" filteraction="REF_HttCffActivDirecUsers (Standard Active Directory Users)" size="0" request="0xccd16a00" url="www.ellenmacarthurfoundation.org/.../GC-Spring-Report-Summary.pdf" referer="www.ellenmacarthurfoundation.org/.../GC-Spring-Report-Summary.pdf" error="" authtime="966" dnstime="9" aptptime="0" cattime="86" avscantime="0" fullreqtime="80895" device="0" auth="2" ua="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36 Edge/15.15063" exceptions="" category="105" reputation="neutral" categoryname="Business" content-type="application/pdf" reason="range"

    These happen hundreds of times per day for PDF files since the UTM upgrade. It is thousands of times per day including other files. It will retry and usually get the PDF, but not always.