XG Blocking large Downloads

Hey guys,

 

Need some advice here.

Trying to download some larger files 3Gb+ and the download starts and I can see it ticking over - then I get a Network Failure error in Chrome or Edge.

I have created a FW Rule at the top of XG and added my PC to it - No App Policy, No scanning, No Web Filters etc. I can see I am going out of that rule fine.

 

But when downloading large files it just stops. Tested from various sources and no matter where it comes from (diff files) it gets stopped after a few seconds.

 Web Surfing, Email etc is fine.

 Testing the same download on another network works fine.

 

The log files are not helping as they simply report all is Allowed and nothing blocked. I have no IPS rules etc. The Policy checker reports all ALLOWED for Firewall and Web Policies.

Any clue where to look next - it was working prior to MR-6

  • Hello ,

     

    This is a known issue when you are trying to download any file over HTTPS even if no policy (web, app, IPS) to it. This behaviour is only observer over HTTPS no matter what the file size and HTTP request will work. Only work around is to disable HTTPS scanning or add website in Web exception or keeping the service awarrenhttp in debug. Currently the above behaviour has been marked with two bug details as bellow.

     

    • NC-47824 : File downloading stopped on enabling HTTPs scanning
    • NC-45724 : Full file download retry failure after 416 (Range Not Satisfiable) being returned by proxy (Fix is available with GES and is expected to b released in MR7)

     

    Most unusual part of NC-47824 is if you add the service awarrenhttp in debug file get downloaded successfully and no irregularity is observed in the logs.

     

    Regards, Ronak.

  • In reply to Ronak Sheth:

    Hey Ronak

     

    The rule I created at the top of my rules for my PC / User only had no HTTP / HTTPS / FTP scanning and no App / Proxy / IPS applied.

     

    EDIT: I see what you are saying. I did have it in an exception and also in the FW bypass but made no difference.

     

    But it dies very soon after starting.

  • In reply to M8ey:

    Hey ,

     

    As I said this is a known issue in XG. Even if you create a network rule at the top of all rules without any policy but still downloading will fail over HTTPS. Only work around is to disable HTTPS scanning or add website in Web exception or keeping the service awarrenhttp in debug

     

    I will suggest you to put the awarrenhttp service in debug mode and reproduce the issue. If the file is downloaded successfully it is the bug NC-47824

     

    Regards, Ronak.

  • In reply to Ronak Sheth:

    Hi Ronak,

    M8ey has stated that he had no HTTPS scanning enabled for at least his second test and i would assume he also had it disabled for the first as he stated he had no policies enabled so therefore no real reason to enable HTTPS interception.

    M8ey, i would recommend doing a dual tcpdump to a file using: tcpdump -b -neXXs0 -i WANPORT host <download target> -w /tmp/httpstestWAN.pcap

    tcpdump -b -neXXs0 -i LANPORT host <download target> -w /tmp/httpstestLAN.pcap

    Then discovering whether it is the XG or the client that sends the FIN/RST.

    I would also enable awarrenhttp debug but if you're in a high throughput environment i would actually tail the awarrenhttp to the cli output and set putty to record the output to a local file because those log files can roll over quickly. If we track the timestamp of the connection close we may be able to relate it to awarrenhttp.

    However, as you did not have any web policy enabled i suspect this may not be tracked in the log and this could be a connection closed for other reasons. In which cas i would extract the entire log directory and track the timestamps through log files pertaining to the fw and snort.

    The command i use is: tar -zcvf /tmp/logdump.tar.gz /log/*.*

    You can then extract all the logs as one file and we can see what's happening.

    Also, what's the download source, is it a google service?

    Emile

  • In reply to EmileBelcourt:

    Thanks for that - I will see if I can make time today to do some logging.

    I have 400+ Users hanging off the XG so I am careful about breaking it in performance during the day :-)

     

    The website was simply:

     

    https://www.itechtics.com/?dl_id=70   This is Windows 10 1903 x64 Offline Installer ISO at around 4.6Gb

     

    I tried a few other sources as well as one of the staff was having trouble getting some movie assets off a supplier website and had the same issue. That site was also in the Web Exceptions HTTPS bypass.

    Did this all happen with MR-6?

     

  • In reply to M8ey:

    That site fails to connect due to security issues, not by the XG, but by the browsers.

    Ian

  • In reply to rfcat_vk:

    Hey Ian,

     

    Not quite sure I follow - my browsers do not block it - I can start the download in Chrome, Edge or IE and it starts downloading fine then craps out within about 15 sec

     

    What are you seeing?

  • In reply to M8ey:

    Hi,

    both safari and FF on the MBP indicate the connection cannot be established because the connection is insecure.

    Ian

     

    and W10 can't find the site.

  • In reply to rfcat_vk:

    Website is available for me (Firefox) and download starts. Stopped it after 1.1 GB as there was no interruption.

    Currently 17.5.4 is installed and I keep my hands of the newer versions as there seem to be too many issues with them.

  • In reply to Jelle:

    Hi Jelle,

    I see the problem with the site from my end, it redirects to an advertisement site which I have blocked in the Sophos home anti-xxxx

     

    Ian