BUG: File extension cause some URLs to get blocked.

This is a correct block.  Blocking by file extension does just that - it does not look to see whether that dll call returns HTML content.  If you want to block by the type of content you actually receive then use MIME type.

Thanks for the reply Michael but I respectfully disagree. This most definitely is not the desired behavior and if XG can't differentiate between a URL and file extension then I don't even need to use it at home for casual surfing. I wasn't even testing for extension blocking and the original bug was caused by wrong categorization message displayed  https://community.sophos.com/products/xg-firewall/v16beta/f/177/t/80088 

In UTM9 for example, extension blocking only blocks file downloads otherwise what is the point of having that option [:(] Lets try using dll block in UTM9... Ok, it seems like we are blocking dll extensions successfully.

and the log for the above block...

2016:09:08-09:08:05 gatekeeper httpproxy[5161]: id="0064" severity="info" sys="SecureWeb" sub="http" name="web request blocked, forbidden file extension detected" action="block" method="GET" srcip="192.168.0.3" dstip="212.27.63.113" user="" group="" ad_domain="" statuscode="403" cached="0" profile="REF_DefaultHTTPProfile (Default Web Filter Profile)" filteraction="REF_DefaultHTTPCFFAction (Default content filter action)" size="2673" request="0xdf405600" url="dllfiles.free.fr/.../6to4svc.dll" referer="www.search-dll.com/.../6to4svc.dll.html" error="" authtime="0" dnstime="84051" cattime="65105" avscantime="0" fullreqtime="355332" device="0" auth="0" ua="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.101 Safari/537.36" exceptions="" category="141" reputation="neutral" categoryname="Portal Sites" country="France" application="freefr" app-id="735" reason="extension" extension="dll" filename="6to4svc.dll",

Now lets go to ebay and see if it gets blocked

2016:09:08-09:12:29 gatekeeper httpproxy[5161]: id="0001" severity="info" sys="SecureWeb" sub="http" name="http access" action="pass" method="GET" srcip="192.168.0.3" dstip="66.171.224.104" user="" group="" ad_domain="" statuscode="200" cached="0" profile="REF_DefaultHTTPProfile (Default Web Filter Profile)" filteraction="REF_DefaultHTTPCFFAction (Default content filter action)" size="703" request="0xabcc000" url="vi.vipr.ebaydesc.com/.../eBayISAPI.dllViewItemDescV4&item=252524153308&t=0&tid=10&category=58297&seller=rlco2469&excSoj=1&excTrk=1&lsite=0&ittenable=false&domain=ebay.com&descgauge=1" referer="www.ebay.com/.../252524153308 error="" authtime="0" dnstime="34099" cattime="310" avscantime="21767" fullreqtime="198730" device="0" auth="0" ua="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.101 Safari/537.36" exceptions="" category="158" reputation="neutral" categoryname="Auctions/Classifieds" country="United States" content-type="text/html" sandbox="-"

I guess UTM9 knows stuff XG doesn't... This is what is so frustrating about XG, I don't know why Sophos doesn't use the things that have already been established in their flagship UTM9. Both XG and UTM9 should behave exactly alike in functionality that has already been agreed upon. Alas, XG is still reinventing stuff. 

Regards

Bill

My old reply got lost I guess. Anyway, are you suggesting that since .dll will block a url with .dll in it, if I use .com as file extension, it will block all .com TLDs [:O] [:D] Surely not...

I should have said it works the way it was designed, or that the behavior is explainable - which is a horrible way of saying it is a feature not a bug.

Very roughly, in the UTM we do:
During request, don't look at the URL.
During response, look at the content-disposition header to find the filename.  If there is no filename then use the URL.

In SFOS we do:
During request, look at the URL.
During response, look at the content-disposition header to find the filename.

For most cases this is an improvement.  It is more efficient, blocks earlier, and saves bandwidth.  For example it means if you block zip files the system won't go and download 100MB of zip from the server before deciding to block.

So it is not that "the UTM knows stuff the XG doesn't".  Its that we deliberately are doing blocking earlier in the phase. It is because Microsoft decided it was smart to use a DLL to return html content, so that in this one case there is an unintended consequence.


The file extension block only checks the last part of the path.  Therefore if you block .com the following will be blocked by URL:
download.net/file.com
download.net/file.com?foobar

The following will not be blocked by URL
download.com
download.com/foobar
download.com/file.com/foobar
download.com/file.comfoobar
download.com/foobar?file.com


In any case, the issue blocking .dll extensions causes problems with websites that use ISAPI.dll has been forwarded.  However the more customer impact there is, the higher priority there is in getting this changed.

Hi Michael,

I am not sure I understand your example because download.com is the last part of of the path.

ian

Michael Dunn said:

Very roughly, in the UTM we do:

During request, don't look at the URL.
During response, look at the content-disposition header to find the filename.  If there is no filename then use the URL.

In SFOS we do:
During request, look at the URL.
During response, look at the content-disposition header to find the filename.

During UTM9 migration phase, you guys will have one hell of a time dissecting everyone's backups created on UTM9 if the default behavior of both firewalls is not exactly the same. No matter how careful you guys are, minor things like URL parsing etc will slip through the cracks since both firewalls don't behave the same way when using the exact same functions.

In any case, I understand what XG is doing under the hood, why it feels quicker and why it should technically load most pages faster!

1) The url http://www.download.com/foobar/file.dll?realfilename=evil.exe
At various times during the URL is broken into scheme, host, domain, path, and query parameters.  The file extension logic is only applied to the end of the path.  It is not applied to the host/domain or to the query.  Therefore http://download.com will not match file extension .com because the .com is not part of the path.

2) Sophos could take the approach that UTM 9.x -> SFOS .v16 is exactly 1:1.  Then later on we change/improve things when SFOS v16 -> v17.  But then people will complain "you changed things!" during the v16 to v17 upgrade.  The only way to make the people who cannot accept change happy is to never ever change.  Just stick with the version you like and stay there.  However if you want improvements you will have to accept that things change.  The best place to make that change is in the UTM -> SFOS upgrade, that is where people will expect it and where they will pay most attention to their policy.

All changes between we are making UTM and SFOS v16 are either "different, but better" or "an issue we will fix" or  "an issue we will not fix".  For example changes to security that disallow access to insecure sites is a change - but it is better.  Yes - someone will complain that a site that used to work with the old version no longer works with the new version, but the change we made is actually a good one.  Part of our job is (in my opinion) to identify and reduce the number of "an issue that we will fix" especially in Beta.  For the "an issue we will not fix" we should identify and explain, maybe put in release notes.  But for the "different, but better"...  That is just the price of improvements.  When you buy a new car with anti-lock brakes the car will behave different than your old one.