Netflix not working despite - Knowledge base is wrong?

Question

I have followed this https://community.sophos.com/kb/en-us/125061 
 and still I can't get netflix to work, why? only works when I disable web scanning in the rule number 6

Michael Dunn · Accepted Answer

In 17.1 MR2 some new logic around cache expiry was brought into FQDN objects. There was unconfirmed reports that this negatively impacted NetFlix, however since we had very few people complaining and could not reproduce ourselves we could not be sure. This new cache expiry logic was disabled by default in 17.5. 
 
 Therefore if you are running 17.1 MR2, MR3, or MR4 please upgrade to 17.5 to see if that resolves the problem. 
 Please note that NetFlix is actually quite hard for us to test, and it does work when we use it. Netflix has complications because different access methods (Web, Android, iOS, Roku, etc) use different mechanisms and becasue some ISP override the NetFlix server DNS entries with their own so as to point to local regional caches to reduce network load - meaning that customers in different countries have different experiences. The majority of our paying corporate customers do not use NetFlix on their networks, though we know it is common with people using the free home license. 
 
 NetFlix also... I won't say they deliberately obscure their traffic, but they don't make it easy. You may notice that no one complains about Hulu or other streaming services. Only Netflix. Because they don't play nice. 
 
 Let me briefly explain the two methods listed in the KB: 
 
 One method has a series of regex that include all the IP addresses owned by netflix. This does not include IP addresses owned by ISPs where the ISP is overriding the netflix server with their own local server. 
 If this method does not work, it might work better if you make sure that the DNS used by the streaming device ultimately points to 8.8.8.8 (or other non-ISP DNS server) and not to an ISP's server. 
 
 The other method uses a dynamic learning method to determine the IPs by passively watching DNS traffic. Because of this it should be more robust on whether the streaming device is going to Netflix or ISP servers. It is easier to configure and it is more efficient (less CPU resources when watching Netflix). However because it is dynamically learning, some people have found it more problematic. Also, as stated there is a reported (but not fully confirmed) issue in this dynamic learning method in 17.1 MR2-MR4 that is resolved in 17.5. 
 
 The KB is not wrong. However I will have it updated to change the emphasis to there being two methods, rather than an v16 method and a v17 method. Both methods work in 17.x. 
 Anyone who: is running 17.5 is using the FQDN Host method is having problems is willing to work with us to resolve it for the betterment of everyone (especially if you are a paying customer and therefore get paid support, but even if you are not) Please let us know in this thread and we will try to get someone working with you to diagnose more.

Michael Dunn · Answer

Here are all the IP ranges that Netflix owns. This does not include IPs that ISPs may be using as local caches. 
 https://ipinfo.io/AS2906 
 
 This has carefully been converted into RegEx for the exception. Please don't modify these regex, they are specially crafted to meet those IP ranges. 
 ^23\.246\.([0-9]|[1-5][0-9]|6[0-3])\.[0-9] ^37\.77\.(1(8[4-9]|9[0-1]))\.[0-9] ^45\.57\.([0-9]|[1-9][0-9]|1([0-1][0-9]|2[0-7]))\.[0-9] ^64\.120\.(1(2[8-9]|[3-9][0-9])|2([0-4][0-9]|5[0-5]))\.[0-9] ^66\.197\.(1(2[8-9]|[3-9][0-9])|2([0-4][0-9]|5[0-5]))\.[0-9] ^192\.173\.(6[4-9]|[7-9][0-9]|1([0-1][0-9]|2[0-7]))\.[0-9] ^69\.53\.(2(2[4-9]|[3-4][0-9]|5[0-5]))\.[0-9] ^108\.175\.(3[2-9]|4[0-7])\.[0-9] ^185\.2\.(2(2[0-3]))\.[0-9] ^185\.9\.(1(8[8-9]|9[0-1]))\.[0-9] ^198\.38\.(9[6-9]|1([0-1][0-9]|2[0-7]))\.[0-9] ^198\.45\.(4[8-9]|5[0-9]|6[0-3])\.[0-9] ^208\.75\.(7[6-9])\.[0-9] 
 
 In order to get Netflix working for you, you added the IP range "45.57.74.100-200" which is in those subset. Someone adventurous could try to convert all the ranges into FQDN hosts. Or if want to use the ranges, it is easier to just use the RegEx exception. 
 
 Can you go into the XG Device Console (not an ssh command line) and tell me the results of 
 show fqdn-host

RichBaldry · Answer

I've been doing a little more investigation with my Sony TV, and an Apple TV device. 
 Firstly, the specific issue with Web scanning and Netflix video playback revolves around malware scanning. Netflix video streaming happens over HTTP. It makes extensive use of byte range requests via the Range: HTTP header. These are not compatible with HTTP malware scanning. XG does try to automatically make exceptions to this when we know it's streaming media traffic, but unfortunately Netflix traffic is very hard to distinguish from regular traffic. This is made harder because the Smart TV Netflix video streaming app uses IP addresses in the URL instead of a hostname, and provides no other HTTP headers to go on. 
 If malware scanning is disabled, Netflix works fine. 
 We can't disable malware scanning for all traffic of course. One way to do this for Netflix traffic alone by specifying traffic in a firewall rule, using FQDN objects to try and pick up any IP addresses associated with the various Netflix domains, and disabling HTTP scanning. But as you've found, it doesn't always work. 
 For FQDN objects to work, there has to be a DNS lookup for the corresponding domain that returns the IP address. I spent some time capturing traffic between my TV and my XG firewall today, and found that much of the time, there were NO DNS responses corresponding to the IP addresses that Netflix used to get video content. In some cases, those IPs were already recorded against the FQDN object on my XG, but I can only assume that's because some other Netflix client (a web browser, maybe, or a mobile app) accessed the content in a different way. 
 So I think the reason why you're finding it doesn't always work, is that the TV must be getting the IP addresses provided directly from other Netflix transactions and not DNS. If you watch the traffic generated by the Netflix app, there is also a lot of HTTPS traffic that is presumably powering the menu system and controlling access to the content. 
 What I did notice was that URLs used to access the video playback were pretty consistent in format. They look like this: 
 hxxp://205.250.87.205/?o=AQHs5r9i3IqU22PaiJ-aVneFNYYH-WKkoC1KT1kYinKL88Ok1GAa0D3_OSHJCzPOHuGeZVqwcqPn2HYXvcBG33-nXwabGNfsBtIy-eO2T6_ks6exhLKmM2SHC37MCbMxvqQ9JlZUPK5yEkc3AxCdUy9El4-L7DwIgV9i4kOVsr88H9boYWCsCijkjSFqrY71nVHJfsuKug_96k-kIFuXF94&v=3&e=1547017850&t=nqVlvSUm-FIT_altCqugFB6hyb0 
 Another way to skip malware scanning is to use a Web Exception. Web exceptions can use regular expressions to match URLs. When I created an exception that excludes scanning for the following regular expression, I was able to watch Netflix videos again, regardless of whether the IP address was known or not: 
 ^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}/\?o=[a-zA-Z0-9_-]*&v=[0-9]*&e=[0-9]* 
 This expression matches any URL with four 1-3 digit numbers in the hostname part, and also matches the first three parameters in the URL. 
 Perhaps this will help in your case?

Michael Dunn · Answer

Badrobot said: 
 Just throwing a thought out there about the difference between netflix and others such as hulu or hbo now. 
 
 Their combat of VPNs and cross-border content is not the problem. We know exactly what the problem is. 
 The problem is that when transferring the actual video content, NetFlix does the following: - uses IPs rather than hostnames (eg GET http://1.2.3.4/) - uses HTTP range requests - declares the mime contenttype the generic application/octet-stream instead of video/ 
 If they changed any one of these three things, we would be fine. If they used hostnames, we can build an easy exception for them. If they declared the mime type we can apply the skip scanning for streaming. If they did not use range requests we could scan the whole file. 
 Hulu and other streaming services do not do all three. I have not done deep examinations, but I suspect they either do not do HTTP range requests or they declare their mime type correctly. 
 
 Here is a completely fake example. 
 
 Netflix 
 GET 1.2.3.4/content?video=aaaaa Range: 20000-30000 content-type application/octet-stream 
 Hulu 
 GET video.hulu.net/content?video=aaaaa;range=20000-30000 content-type video/mpeg 
 You see the latter using a hostname, using their own custom range handling in the GET parameters (instead of HTTP standard), and actually declaring the content-type? Much better, and why other services work out of the box.

Netflix could fix this today on all systems everywhere (with changes to any proxy or any client) if they changed their servers to use the correct mimetype.

TheBalmasque · Answer

Problem solved:

I defined 3 IP-Range objects "Netflix Range 74,75,78" and put it in the destiantion network list. (For all ranges from 45.57.*.100 - 45.57.*.200) 
 The network rule is only for my Panasonic TV and no other devices. I hope this works for you too. [H]