I really want to like this product, but...

I'll combine your first comment with your #5 first - if you had bought this for your workplace, you'd likely be entitled to support. Support wouldn't be non-existent, and they'd work with you through these issues. I'm working with their support engineers now on a few bugs, and they're excellent. How they should be supporting other folks - like forum folks - that aren't paying XG customers yet - that's a different discussion. I think their involvement here could (should?) be better... especially since they seem to want people to use the free home version... and I think it will get better... Support just got handed a new product, too, so they are working to understand this new XG product as well. Give it a little time.

Now - 1) webfilter issues. Just about any webfilter with HTTPS filtering is going to break something like netflix and amazon fire. I enabled Watchguard's version of https filtering at my house - had very similar issues. I came from and am certified in the watchguard product, so I had a spare of their smaller boxes lying around to test. Anyway - anything that tried to validate SSL certificates - and for which installing new certificates was nearly impossible - generally had major issues. Amazon Kindle e-readers would freak as well. This is a known issue with just about any deep packet inspection https filter, and the only way resolve that is to install the ca certificate from the firewall onto the end devices - or except the traffic from the https filter. The XG device actually makes it fairly easy to except certain traffic from HTTPS inspection. Entire categories can be exempt via the policy, and when ssl certificates for those sites are detected, the traffic is not intercepted and monkeyed with, or individual IP addresses can be exempt as well. I've found it works acceptably well. I have had to add in a few *special* exemptions - logmein rescue, for example, certain sites using websockets in an odd way, that sort of thing - but overall, it is working like an absolute champ.

2) can the gateway be pinged? The xg relies on pings - not whether traffic is passing through the interface - to determine the status of the gateway, I believe. I think I remember that from the XG certification course.... Maybe that was only multi-wan was setup...

3) What's the load average look like on the performance screen?

4) Do a packet capture on the smtp traffic to see what the SMTP conversation looks like. Could be any NUMBER of things - including user error (hey, it happens... even on a stupid smtp connection. )

I would like to thank you for taking the time try and explain things to us. I do understand this is a new product in fact its very much v1 as has been said many times in the beta program when I was a part of it that said communication from Sophos even in the beta program was none existent and when a company is asking for are help in testing and asking for our feedback and it falls a dead ears with no response that's never a good thing.

As for Netflix not working on mobile devices this is something going back to UTM9.x at least in UTM9 we could fix it with some Regex entries and move on with our lives without completely disabling web filter and HTTP scanning, Its not just when HTTPS scanning is enabled its when plane HTTP scanning us used. No matter what we do with different combinations if you have ANY of the following 3 things active it breaks Netflix on mobile devices

web filter
HTTP Scanning
HTTPS Scanning

That's a better one than the one I found. Mine was an old forum post here: www.astaro.org/.../50903-exception-rule-netflix-streaming.html

Drashna's is much more thorough. I'll look through his when I get ready to do this.

My pleasure. The old Sophos forums are a bit of a mess when it comes to all the suggestions regarding Netflix not working. Chris the owner of Drashna blog belongs to one of the forums that i am pretty active in called
homeservershow.com and I must say he does great work, You can trust that everything you read on that blog I sent you has been tested and works. If you get this working your going to make a lot of peaple very happy this holliday season, myself included.

If you get a chance have a look at the frustration about the new XG Firewall regarding Netflix not working as well as other things. See the link below to a thread I started on the HSS, My forum name is itGeeks
homeservershow.com/.../index.php

Ok, so I won't say "challenge completed" yet - because I cheated to get it working... still working on the webfilter side, but here is one SURE-FIRE way to get it to work while keeping filtering on for the majority of users...

So we'll say... CHALLENGE PENDING for now - and use this to get it working!

1) Create a clientless users group
Objects > Identity > Groups
Add.
Enter a group name.
Group Type: Clientless
Quarantine Digest: Disable
Save.

2) Create Clientless users for each exempt device.
Objects > Identity > Clientless Users
Add (or add range, if they are in a specific range)
Enter a username - something descriptive for the device (ex: ccamp-iphone)
IP Address: the internal ip address
Group: The clientless group you created in step 1
Name: Some name. Descriptive. "My Iphone"
Email: fake an email address. Next version won't require this.
Description: More useless description info. Not required.

Click the plus sign if you need to add additional devices.

Click save.

3) Create security policy
Security Policies
Click on your HTTPS filter rule, click the plus sign, click "Above (User/Network Rule)"
About this rule---
Name: Allow Clientless to Bypass Filter
Identity---
Match Rule based on user identity: On
User or Groups: Clientless Group created in step 1
Source---
Zone: Lan
Networks: Any
Services: HTTP,HTTPS, others if you need them, but these suffice for this walkthrough
Destination---
Zone: WAN
Networks: Any
Malware Scanning---
Scan FTP OFF
Scan HTTP OFF
Decrypt & Scan HTTPS OFF

Save.

This bypasses the specific clientless devices you created from the webfilter entirely. This is actually a reasonably good solution - and may be the "best" solution for roku/appletv/chromecast and other fixed devices that do not regularly leave your network.

This is NOT the best solution for mobile devices, but it will work. The best solution for mobile devices would be to either FIX the damned web filter (still working on it) or create a separate wireless network for them.

First I would like to say I am grateful for all your hard word on this, I further more agree that Sophos just needs to fix this very old streaming problem, This was a problem in UTM9 as well but at least we could fix it with regex entries and NOT have to completely bypass devices like we do now to get it working. Does none of the Sophos employees have mobile devices and use services such as Netflix? That would be mind blowing if they did not. It seems to me that Sophos does not want anyone using such services if using there product.

No I mean no harm in what I am about to say & again I am very great full for your hard work but is what you posted not the long-hand way of solving the Netflix problem vs what we are already doing in a short-hand way? The way I solved the Netflix problem is easy, I created a policy for all my mobile devices called streaming devices and I just turned off HTTP and HTTPS scanning and I disabled the web filter and Netflix works. Am I missing something with your way of doing this? Just seems like a lot of extra work with no benefit unless I am missing something?

The difference is in the reports you get. Your way works - and is the quickest way. My way lets you easily identify the ip addresses with user names/descriptions if/when you run traffic reports. Either works.

And before we throw Sophos under the bus for their HTTP/S filtering - lets ask ourselves - is any vendor ACTUALLY PRODUCING a true HTTP and HTTPS filter that works *well* with highly advanced streaming services like netflix? The answer is likely no. Netflix in particular is a very odd bird due to the way it streams using direct IP addresses instead of host names that can easily have regex or string comparisons done against them.

I may open a support ticket on this and see what support says. I may also put my challenge on hold until the new version is released in January/February... see what they fix/change. I don't expect them to FIX it outright, but they very may well change something that either breaks whatever I come up with *OR* fix something else (like regex) that allows me to make things actually work.

Thank you for the detailed explanation. That would be great if you could enter a support ticket. I think the more we hound Sophos to get this fixed once and for all the better. I look forward to any updates you may have.

Thanks again for taking the time...

Hi ChavousCamp,
Unrelated to Netflix, Do you know to fix I am having trouble with Windows downloading new Insider builds when the device is behind XG, The only way I can download the new builds is to disable HTTP and HTTPS scanning. When I go into the logs XG does not show anything blocked. What do I have to do to fix this so I don't have to DISABLE HTTP scanning?

Thanks

Until they allow multiple regex expressions per single rule this will be a continuing problem. It was brought up multiple times during beta testing and I have already addressed my feelings on that subject so I won't hijack this thread.

Regarding no blocks showing in logging, refer to this thread https://community.sophos.com/products/xg-firewall/f/46/t/15732 by  

billybob, Thanks for the info. I agree with you 100% but now the million dollar question, Will they fix it? It was said so many times on the beta forum that this product was not ready for prime time but ready or not they role it out anyway. I guess that's the price you pay for early adoption and I am more then willing to hang in there and keep using it as long as they fix things in a timely manor and don't make us wait years. I can only hope that the next update due out in Jan-Feb addresses some of this stuff.

how is this going? I have heard only one regex is supported, so I expect it will be necessary to use the 'or' symbol so mush the 'dreashna' items into a single regex....

how is this going? I have heard only one regex is supported, so I expect it will be necessary to use the 'or' symbol so mush the 'dreashna' items into a single regex....

Neal as you can see not very well. We need better support in the Regex department or we going no place.

Regex is needed, definitely