Guest User!

You are not Sophos Staff.

Thread Info
  • State Verified Answer
  • Locked Locked
  • Replies 10 replies
  • Subscribers 27 subscribers
  • Views 6622 views
  • Users 0 members are here
Options
Suggested
This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

Web Blocking ignored if URL is not accessed directly

saspus

Hello, 

 

SFOS 16.05.2 MR-2

 

So I've created a firewall rule and applied web policy to  to block URL category "Spyware & Malware"

Next, I've typed a url (see screenshot, did not want to put it in plain text) and confirmed in the logs that it is blocked.

However when I visit another - permitted web site -- e.g. http://elderscrolls.wikia.com which apparently indirectly accesses that blocked web site - it is then allowed. See screenshot. 

Note, same source IP, same rule.

 

Any ideas?

 

 

 

Edit1.

I've narrowed it down to the length of URL:

This is correctly blocked:

www.decenthat.com

This also is correctly blocked:

www.decenthat.com/bcn?fe=14 

 

However adding more stuff to the request - apparently makes it allowed:

www.decenthat.com/bcn?fe=149  <---- allowed

www.decenthat.com/bcn?fe=1490168440455&vv=1490167133770 <--- also allowed

 

 

This is just silly.



This thread was automatically locked due to age.
  • +1

    Can you go to all URL Groups, Custom Categories, and Exceptions and make sure that you are not hitting one of those?

    For example, if you tried to do an exception based on an ip of 149.* it would apply.

  • 0 in reply to Michael Dunn

    Hi Michael - thank you for the hint - I should have thought of that...

    It was indeed one of the exception rules.

     

    In fact it was the bypass rule for Netflix made according to this KB article:

    https://community.sophos.com/kb/en-us/125061

    Apparently one of the regexes there is too loose. I'll try to find which one and update later.

  • 0 in reply to saspus

    Oh dear that was a badly written list of regex in that KB.  It looks like the source was a post from the forum.

    I don't know where the list of IPs came from, but it isn't supported by anything I can find online.  They way they were written made almost any number match.

    The KB will be updated soon to remove all the IPs and fix up some of the others regex.

     

    That being said, it may be that the whole KB is now irrelevant.  NetFlix has changed how they deliver content since that KB was written.

    I know that Netflix over the computer browser no longer uses range requests - which was the thing causing the problem in the first place.  I don't know if Netflix over the XYZ device (eg TVs) are also updated.

    If you don't mind, can you try turning off the exception entirely and then test NetFlix on all devices you normally watch on.  If they all work, then great.

    If NetFlix doesn't work, try using the exception as it is now written in the KB.

  • 0 in reply to Michael Dunn

    Yes, those regexps are rather ugly. 

    As far as I remember the desktop browser worked fine; however the Netflix iOS app didn't.

    I'll retest later tonight and post update. 

  • 0 in reply to saspus

    So, the problem is indeed with iOS Netflix app. 

    I have tried the new URL list from the updated KB and it did not work. The reason being the requests netflix app makes are in the form 

    http://45.57.27.143/?o=AQH67uu6-bp5k.....

    - with IP address belonging to the Netflix CDN. No symbolic URL is used - so URL filtering won't help. 

    What I ended up doing and what worked for me was:

    1. find out Netflix org name - starting with that IP from the log

    alex$ curl ipinfo.io/.../org
    AS2906 Netflix Streaming Services Inc.

    2. Find associated IPv4 ranges:

    alex$ whois -h whois.radb.net -- '-i origin AS2906' | grep ^route:

    route:      69.53.224.0/19
    route:      208.75.76.0/22
    .....
    route:      23.246.29.0/24
    route:      23.246.28.0/24 

    3. Convert it into ip ranges using nice tool here http://www.techzoom.net/Tools/IPAddressCalculator (this is complete set of ranges)

    23.246.0.0-23.246.63.255
    37.77.184.0-37.77.191.255
    45.57.0.0-45.57.127.255
    64.120.128.0-64.120.255.255
    66.197.128.0-66.197.255.255
    69.53.224.0-69.53.255.255
    108.175.32.0-108.175.47.255
    185.2.220.0-185.2.223.255
    185.9.188.0-185.9.191.255
    192.173.64.0-192.173.127.255
    198.38.96.0-198.38.127.255
    198.45.48.0-198.45.63.255
    208.75.76.0-208.75.79.255

    4. Add each range as IP host into XG (tedious, should have written a script)

    5. Add all those IP Hosts into IP Host Group

    6. Create a firewall rule to match that group as destination and not set HTTP/HTTPS scanning.

    Probably since the format of Netflix requests accesses is just http://<IP>/something it would be possible to just wrap those ranges into regexp to put into Web Exclusions and avoid creating extra firewall rules (I'm not very good with regexps though, it will take some time)

    Edit: Regexp approach won't not work since XG limits length of URL regexp.

  • 0 in reply to saspus

    Thanks for the test of NetFlix on IOS.  It appears that that the IP address regex are still needed. 

    Here is the corrected regex for each of those ranges.  Note this tool: www.analyticsmarket.com/.../ipregex

    As an aside, we are trying to get this working natively in v17 without this trick.  I'll make sure we test IOS NetFlix as well.  :)

     

    23\.246\.([0-9]|[1-5][0-9]|6[0-3])\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))

    37\.77\.(1(8[4-9]|9[0-1]))\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))

    45\.57\.([0-9]|[1-9][0-9]|1([0-1][0-9]|2[0-7]))\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))

    64\.120\.(1(2[8-9]|[3-9][0-9])|2([0-4][0-9]|5[0-5]))\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))

    66\.197\.(1(2[8-9]|[3-9][0-9])|2([0-4][0-9]|5[0-5]))\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))

    69\.53\.(2(2[4-9]|[3-4][0-9]|5[0-5]))\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))

    108\.175\.(3[2-9]|4[0-7])\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))

    185\.2\.(2(2[0-3]))\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))

    185\.9\.(1(8[8-9]|9[0-1]))\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))

    192\.173\.(6[4-9]|[7-9][0-9]|1([0-1][0-9]|2[0-7]))\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))

    198\.38\.(9[6-9]|1([0-1][0-9]|2[0-7]))\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))

    198\.45\.(4[8-9]|5[0-9]|6[0-3])\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))

    208\.75\.(7[6-9])\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))

  • 0 in reply to Michael Dunn

    Hi Michael, 

    Thank you for the link and heads-up about v17 - that is good to know. 

    I've also found that tool yesterday and tried to insert generated ranges - unfortunately they would not fit into the allowable length of 100 characters (which seems rather arbitrary limit): 

    As a side note, at least there was validation and a message popup -- unlike with the editor of contents of IP Host object - it allows to stick a lot of addresses in IP List but then silently truncates all past 60 characters. (60 character limit is mentioned in API documentation which I found after trying to add those IPs via API and received an error for long lists) which is another  arbitrary limit). I'm almost certain its a known bug -- but just in case thought to mention it in case it is not.

    --Alex.

  • 0 in reply to saspus

    *sigh*  Thanks.  I'll look into the limit, which is arbitrary.

    All of the ranges are .0 to .255 so the final octet doesn't actually need to be verified as a valid IP.  That shortens all of them, the following regex just checks there is at least one character instead of making sure it is a valid ip.  Can you try:

    23\.246\.([0-9]|[1-5][0-9]|6[0-3])\.[0-9]
    37\.77\.(1(8[4-9]|9[0-1]))\.[0-9]
    45\.57\.([0-9]|[1-9][0-9]|1([0-1][0-9]|2[0-7]))\.[0-9]
    64\.120\.(1(2[8-9]|[3-9][0-9])|2([0-4][0-9]|5[0-5]))\.[0-9]
    66\.197\.(1(2[8-9]|[3-9][0-9])|2([0-4][0-9]|5[0-5]))\.[0-9]
    192\.173\.(6[4-9]|[7-9][0-9]|1([0-1][0-9]|2[0-7]))\.[0-9]
    69\.53\.(2(2[4-9]|[3-4][0-9]|5[0-5]))\.[0-9]
    108\.175\.(3[2-9]|4[0-7])\.[0-9]
    185\.2\.(2(2[0-3]))\.[0-9]
    185\.9\.(1(8[8-9]|9[0-1]))\.[0-9]
    198\.38\.(9[6-9]|1([0-1][0-9]|2[0-7]))\.[0-9]
    198\.45\.(4[8-9]|5[0-9]|6[0-3])\.[0-9]
    208\.75\.(7[6-9])\.[0-9]

  • 0 in reply to Michael Dunn

    Yep, good catch about last octet!

     

    I've also prepended each expression with "^" just to harden it a bit and removed redundant URL based regexes from the KB, as well as *llnwd\.net which I don't think is needed.

    This is the final Exception filter which seem to work (and yes, I did disable my IP group based firewall rule):

    ^([A-Za-z0-9.-]*\.)?ne?t?fli?x(img|ext|video)?\.(com|net)/
    ^23\.246\.([0-9]|[1-5][0-9]|6[0-3])\.[0-9]
    ^37\.77\.(1(8[4-9]|9[0-1]))\.[0-9]
    ^45\.57\.([0-9]|[1-9][0-9]|1([0-1][0-9]|2[0-7]))\.[0-9]
    ^64\.120\.(1(2[8-9]|[3-9][0-9])|2([0-4][0-9]|5[0-5]))\.[0-9]
    ^66\.197\.(1(2[8-9]|[3-9][0-9])|2([0-4][0-9]|5[0-5]))\.[0-9]
    ^192\.173\.(6[4-9]|[7-9][0-9]|1([0-1][0-9]|2[0-7]))\.[0-9]
    ^69\.53\.(2(2[4-9]|[3-4][0-9]|5[0-5]))\.[0-9]
    ^108\.175\.(3[2-9]|4[0-7])\.[0-9]
    ^185\.2\.(2(2[0-3]))\.[0-9]
    ^185\.9\.(1(8[8-9]|9[0-1]))\.[0-9]
    ^198\.38\.(9[6-9]|1([0-1][0-9]|2[0-7]))\.[0-9]
    ^198\.45\.(4[8-9]|5[0-9]|6[0-3])\.[0-9]
    ^208\.75\.(7[6-9])\.[0-9]

    Thank you for help with this!

    Alex.

  • 0 in reply to saspus

    Thanks, looks good.  I'll get the KB updated.  The llwnd one is a limelight content delivery network.  I don't know if it is still used and if it is whether we need to do an exception to make it work.  Lets assume it is enough to leave it out of the exception.

    When the v17 beta comes out, I'd love people's help in testing NetFlix with no exception at all.  :)