Netflix not working despite - Knowledge base is wrong?

I have followed this  https://community.sophos.com/kb/en-us/125061

and still I can't get netflix to work, why? only works when I disable web scanning in the rule number 6

  • In reply to TheBalmasque:

    I've been doing a little more investigation with my Sony TV, and an Apple TV device.

    Firstly, the specific issue with Web scanning and Netflix video playback revolves around malware scanning. Netflix video streaming happens over HTTP. It makes extensive use of byte range requests via the Range: HTTP header. These are not compatible with HTTP malware scanning. XG does try to automatically make exceptions to this when we know it's streaming media traffic, but unfortunately Netflix traffic is very hard to distinguish from regular traffic. This is made harder because the Smart TV Netflix video streaming app uses IP addresses in the URL instead of a hostname, and provides no other HTTP headers to go on.

    If malware scanning is disabled, Netflix works fine.

    We can't disable malware scanning for all traffic of course. One way to do this for Netflix traffic alone by specifying traffic in a firewall rule, using FQDN objects to try and pick up any IP addresses associated with the various Netflix domains, and disabling HTTP scanning. But as you've found, it doesn't always work.

    For FQDN objects to work, there has to be a DNS lookup for the corresponding domain that returns the IP address. I spent some time capturing traffic between my TV and my XG firewall today, and found that much of the time, there were NO DNS responses corresponding to the IP addresses that Netflix used to get video content. In some cases, those IPs were already recorded against the FQDN object on my XG, but I can only assume that's because some other Netflix client (a web browser, maybe, or a mobile app) accessed the content in a different way.

    So I think the reason why you're finding it doesn't always work, is that the TV must be getting the IP addresses provided directly from other Netflix transactions and not DNS. If you watch the traffic generated by the Netflix app, there is also a lot of HTTPS traffic that is presumably powering the menu system and controlling access to the content.

    What I did notice was that URLs used to access the video playback were pretty consistent in format. They look like this:

    hxxp://205.250.87.205/?o=AQHs5r9i3IqU22PaiJ-aVneFNYYH-WKkoC1KT1kYinKL88Ok1GAa0D3_OSHJCzPOHuGeZVqwcqPn2HYXvcBG33-nXwabGNfsBtIy-eO2T6_ks6exhLKmM2SHC37MCbMxvqQ9JlZUPK5yEkc3AxCdUy9El4-L7DwIgV9i4kOVsr88H9boYWCsCijkjSFqrY71nVHJfsuKug_96k-kIFuXF94&v=3&e=1547017850&t=nqVlvSUm-FIT_altCqugFB6hyb0

    Another way to skip malware scanning is to use a Web Exception. Web exceptions can use regular expressions to match URLs. When I created an exception that excludes scanning for the following regular expression, I was able to watch Netflix videos again, regardless of whether the IP address was known or not:

    ^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}/\?o=[a-zA-Z0-9_-]*&v=[0-9]*&e=[0-9]*

    This expression matches any URL with four 1-3 digit numbers in the hostname part, and also matches the first three parameters in the URL.

    Perhaps this will help in your case?

     

  • In reply to RichBaldry:

    Rich (my boss) beat me to it with the reply.  And he has more test toys when he works from home - though he cannot reproduce the problem.  :)
    While his RegEx may provide another solution (which I want to investigate), it doesn't explain why the current solution does not work.

    Test 1:
    On the XG command line, run the following:
    service -ds nosync fqdnd:restart
    This will restart the service collecting IPs and will clear the list of known IPs.
    Take a look at the FQDN objects.  It should be "no subdomains found"
    Power off (eg unplug) your TV.
    Note: It is important to fully force the TV (or set top box, or whatever is doing netflix) to clear its cache with a complete shutdown.
    Power on your TV.
    Start Netflix.  Watch a show.
    Take a look at the FQDN objects.  It should have a bunch of IPs.
    Are there now IPs for nfxvideo.net ?
    Just in case the full power-off actually fixed the problem, turn off the exception and test with just the Netflix destination.
    Any better?
    Make sure you are not doing any netflix on any other device during this test.

    Test 2:
    Can you set up the following:
    FW rule 1 : Source: TV  Destination: Netflix  Web: None
    FW rule 2 : Source: TV  Destination: Any  Web: None
    FW rule 3 : Source: LAN  Destination: WAN  Web: Whatever your normal is
    Now watch some videos on Netflix.
    If you go to the Firewall list of rules you will see In and Out bytes.  Confirm that Rule 2 has traffic that is hitting it.
    This will prove that there is a traffic that is not being caught by Rule 1.

    Edit Rule 2.  Set Web Policy to Allow All.  This will cause the web traffic to go through the web proxy, but not AV, so it should work and give you web logging.
    Now watch some videos on Netflix.
    Go to Log Viewer, Detailed View (icon), Web Filter.  If you need to, you can use filter or search to narrow to the ip address of your TV.
    You should be getting a bunch of URLs that look like IP addresses.
    Are you getting anything that is not an IP address?
    Can you post a few lines from the log here?
    Take the IP address and go back to the FQDN host objects.  Is the IP listed anywhere in any of the host object?
    If the IP is not listed, that means the there is a problem in the capturing of IPs.
    If the IP is listed, that means the there is a problem in the destination rule matching.
     
  • In reply to Michael Dunn:

    @ : Thank you very much for your investigation. I didn`t know the FQDN objects are "learing" new ip adresses when an other device access Netfilx, very interesing. Cool

    @Michael Dunn: Hello, sorry i dont understand the following phrase: On the XG command line, run the following.  

    What command should i run to to restart the IP collection service?

     

    Tomorrow i will try to follow your steps. Thank you very much in advance. Cool

  • In reply to TheBalmasque:

    Sorry, must have forgotten to paste in the command.  I edited the above post. 

    service -ds nosync fqdnd:restart

     

    This is a service that watches all DNS requests, learning new IP addresses on an FQDN and putting it into a memory cache.  This is cleared if you restart the service.  This will watch all DNS requests, so if your TV, your phone, and your computer all do DNS requests to Netflix, it learns them all.  In testing this, in order to be pure, try to only do one device at a time.  This service does not know anything about DNS requests that occur before it started.  Therefore if you start Netflix on your TV, which does DNS requests, then reboot your XG, the XG will not know about any of the DNS that the TV did.  In THEORY the TV should be using the TTL (time to live) to re-do the DNS requests periodically, so that over time the XG learns all the IPs, even ones that are from before it booted.  One GUESS is that the TV is doing DNS requests and caching the results a really long time so that the XG never learns them.  That is why I want a full power cycle of the TV, to try to force it to do the DNS requests while the XG is monitoring.

  • In reply to Michael Dunn:

    Thank you for the command line. It worked for me. All adresses are gone and now they are collecting again.

    Under *.nflxvideo.net now there is 1 IP adress:

    ipv4-c008-fra002-ix.1.oca.nflxvideo.net 45.57.74.137

     

    I started with Test 1:

    With my range exceptions it worked. All videos go through without any errors. When i turn of my exeptions the videos didnt start anymore with an error message from netflix.

    Here are some logs:

     

    There are some IP-adresses from my range exception like 45.57.78.132. This ip-adress is not coverd by the netflix fqdn-objects.

     

    Test 2:

    As you see there is a lot traffic in FW 2.

    When i go to the logs i see the following:

     

    There are several hits for the firewall ID3 which is my http-scanning rule. I dont see the FW 2 logs even there is traffic encountered.

  • In reply to TheBalmasque:

    I am having the same problem connecting to Netflix from Roku devices. 

    If I add the Sophos XG v16 web exceptions, it works fine. The second I turn them off, to test the v17 fix...it kills Netflix (only my Roku devices).

  • In reply to Joshua Mallow:

    I think the main problem is, that there are some IP-addresses (like 45.57.78.132) not associated with the netflix fqdn-objects.

    But i'm excited to hear some new infos from the Sophos team. Cool

  • In reply to TheBalmasque:

    The problem does seem to be on the collection of IP addresses into FQDN objects.  I'm trying to get an expert in this area to take a deeper look.