Sophos Central Endpoint and SEC: Computers fail/hang on boot after the Microsoft Windows April 9, 2019 update. Please follow knowledge base article 133945
Learn about the Benefits of Multi-Factor Authentication (MFA). Turn your MFA on now!
This article describes the changes in Range requests from Sophos UTM v9.5 to v9.6. Range requests are used to handle partial content download. The following sections are covered:
Applies to the following Sophos products and versions Sophos UTM v9.6
Most HTTP transactions involve a user agent (e.g. a browser) requesting a complete file or a resource from a server, which the server passes back in its response.
When downloading large items, such as streaming video content or software installers, downloading an entire file at once can be inefficient, especially on a mobile device that can lose its connection if a user goes out of network range. Imagine that you had successfully downloaded 295 MB of a 300 MB file when your connection dropped. Wouldn't it be great if you could just reconnect and grab the final 5 MB remaining instead of starting all over again.
The HTTP protocol provides the Range (header) for this purpose, it allows a user agent to request the server pass just a limited range of bytes instead of the whole file. So in the above case, the client could include the following header: Range: 295000000-
The server then, if it can, would pass a chunk of the file starting at byte 295,000,000 and going until the end.
This is very handy for doing restartable downloads, but it has implications for security. To provide proper protection against downloading malware, it is necessary to scan an entire file from beginning to end and sometimes back the other way again. Range requests allow files to be downloaded in separate chunks, which are pieced together at the end to reform the whole file. But if a virus is downloaded in 10 KB chunks, scanning each chunk is not going to reveal the malicious content within the file.
For this reason, the SG UTM prevents byte Range downloads unless they do not need to be scanned for malware. This means:
Range requests are defined as a standard by IETF in RFC7233. The standard makes it clear that clients agents should not assume that Range requests will be successfully fulfilled. It describes responses from servers that cannot do so, and how clients should react to those responses. But like many web standards, implementations vary, especially between the huge number of software updaters and download mangers that make most use of Range requests.
In version 9.5, when a client requested a partial file download with a Range: header, the UTM would remove the range header before forwarding the request to the web server. This would result in the whole file being downloaded (and scanned) before being passed back to the client.
Some client software would handle this situation well, but we came across many situations where the client would reject the full content response. In some situations, the client would then re-issue the partial GET request, causing the full file to be downloaded again. If this continued, customers would therefore experience significant bandwidth usage increases.
In version 9.6, we improved handling of range requests by increasing the number of situations in which range requests were considered safe (e.g. for streaming media file content). But we also addressed the issue of bandwidth wastage by changing how we handle unsafe Range requests. The UTM now responds to all unsafe Range requests by sending a 416 Range not satisfiable error code. This should cause the downloading agent to specifically send a new request for the full file. For a compliant agent, the end result is the same as under version 9.5, because it will result in the whole file being requested and downloaded. However, there are some clients that appear to fail completely when they receive a 416 Range not satisfiable error code.
416 Range not satisfiable
Transactions that are being blocked in this way can be identified in the web protection logs by looking for action="block", statuscode='416' and reason="range", for example:
2019:06:06-11:14:47 utm httpproxy: id="0002" severity="info" sys="SecureWeb" sub="http" name="web request blocked" action="block" method="GET" srcip="192.168.xx.xx" dstip="xx.xx.xx.xx" user="" group="" ad_domain="" statuscode="416" cached="0" profile="REF_DefaultHTTPProfile (Default Web Filter Profile)" filteraction="REF_DefaultHTTPCFFAction (Default content filter action)" size="0" request="0x95a71c00" url="http://www.example.com/download/file.exe" referer="" error="" authtime="41" dnstime="1533" aptptime="268" cattime="149" avscantime="0" fullreqtime="33689" device="1" auth="2" ua="curl/7.59.0" exceptions="" category="9998" reputation="unverified" categoryname="Uncategorized" country="United States" content-type="text/html" reason="range"
The solution for this is to create an exception for the problematic traffic by going to Web Protection > Filtering Options and selecting the Exceptions tab. To prevent blocking of Range: requests you will need to create an exception that skips the Antivirus check, and then set some selection criteria. You can base the selection criteria for your exception on the domain or URL that the content is coming from, or on the User-agent string sent by the downloading agent. You can also create an exception that uses website tags, that can be applied to domains or URLs via the Websites tab. For example, if you want to allow byte-range downloads from 'updates.example.com':
Now that you've created an exception with this tag, you can exclude further sites just by adding them to the Websites page with the same tag.
Sign up to the Sophos Support SMS Notification Service to get the latest product release information and critical issues.
If you've spotted an error or would like to provide feedback on this article, please use the section below to rate and comment on the article. This is invaluable to us to ensure that we continually strive to give our customers the best information possible.
Every comment submitted here is read (by a human) but we do not reply to specific technical questions. For technical support post a question to the community. Or click here for new feature/product improvements. Alternatively for paid/licensed products open a support ticket.