I really want to like this product, but...

I'll combine your first comment with your #5 first - if you had bought this for your workplace, you'd likely be entitled to support. Support wouldn't be non-existent, and they'd work with you through these issues. I'm working with their support engineers now on a few bugs, and they're excellent. How they should be supporting other folks - like forum folks - that aren't paying XG customers yet - that's a different discussion. I think their involvement here could (should?) be better... especially since they seem to want people to use the free home version... and I think it will get better... Support just got handed a new product, too, so they are working to understand this new XG product as well. Give it a little time.

Now - 1) webfilter issues. Just about any webfilter with HTTPS filtering is going to break something like netflix and amazon fire. I enabled Watchguard's version of https filtering at my house - had very similar issues. I came from and am certified in the watchguard product, so I had a spare of their smaller boxes lying around to test. Anyway - anything that tried to validate SSL certificates - and for which installing new certificates was nearly impossible - generally had major issues. Amazon Kindle e-readers would freak as well. This is a known issue with just about any deep packet inspection https filter, and the only way resolve that is to install the ca certificate from the firewall onto the end devices - or except the traffic from the https filter. The XG device actually makes it fairly easy to except certain traffic from HTTPS inspection. Entire categories can be exempt via the policy, and when ssl certificates for those sites are detected, the traffic is not intercepted and monkeyed with, or individual IP addresses can be exempt as well. I've found it works acceptably well. I have had to add in a few *special* exemptions - logmein rescue, for example, certain sites using websockets in an odd way, that sort of thing - but overall, it is working like an absolute champ.

2) can the gateway be pinged? The xg relies on pings - not whether traffic is passing through the interface - to determine the status of the gateway, I believe. I think I remember that from the XG certification course.... Maybe that was only multi-wan was setup...

3) What's the load average look like on the performance screen?

4) Do a packet capture on the smtp traffic to see what the SMTP conversation looks like. Could be any NUMBER of things - including user error (hey, it happens... even on a stupid smtp connection. )

I'll combine your first comment with your #5 first - if you had bought this for your workplace, you'd likely be entitled to support. Support wouldn't be non-existent, and they'd work with you through these issues. I'm working with their support engineers now on a few bugs, and they're excellent. How they should be supporting other folks - like forum folks - that aren't paying XG customers yet - that's a different discussion. I think their involvement here could (should?) be better... especially since they seem to want people to use the free home version... and I think it will get better... Support just got handed a new product, too, so they are working to understand this new XG product as well. Give it a little time.

Now - 1) webfilter issues. Just about any webfilter with HTTPS filtering is going to break something like netflix and amazon fire. I enabled Watchguard's version of https filtering at my house - had very similar issues. I came from and am certified in the watchguard product, so I had a spare of their smaller boxes lying around to test. Anyway - anything that tried to validate SSL certificates - and for which installing new certificates was nearly impossible - generally had major issues. Amazon Kindle e-readers would freak as well. This is a known issue with just about any deep packet inspection https filter, and the only way resolve that is to install the ca certificate from the firewall onto the end devices - or except the traffic from the https filter. The XG device actually makes it fairly easy to except certain traffic from HTTPS inspection. Entire categories can be exempt via the policy, and when ssl certificates for those sites are detected, the traffic is not intercepted and monkeyed with, or individual IP addresses can be exempt as well. I've found it works acceptably well. I have had to add in a few *special* exemptions - logmein rescue, for example, certain sites using websockets in an odd way, that sort of thing - but overall, it is working like an absolute champ.

2) can the gateway be pinged? The xg relies on pings - not whether traffic is passing through the interface - to determine the status of the gateway, I believe. I think I remember that from the XG certification course.... Maybe that was only multi-wan was setup...

3) What's the load average look like on the performance screen?

4) Do a packet capture on the smtp traffic to see what the SMTP conversation looks like. Could be any NUMBER of things - including user error (hey, it happens... even on a stupid smtp connection. )

Yes, that's what I was getting at - their involvement on this very forum.

1) Maybe I misundertand the issue with the webfilter then - I don't want to have to mess with certificates. I do want web browsing to be scanned for malware/virus and I have tested with the webfilter switched on and it does indeed detect those things. However, as stated above Netflix etc break with the webfilter on. That means that Video Streaming functionality and malware/virus detection are mutually exclusive at the moment with this product.

2) I have no idea if it can be pinged - this is the WAN and is not on my LAN - I'll try the below recommendation from Wayne!

3) Load average has gone above 2 which is what triggers it - however I cannot see why it goes above 2 when cpu usage and RAM usage are negligible. There appears to be no logical correlation.

4) It seems I am not the only one with this issue now - I'll have to dig further, but the lack of any proper realtime log makes troubleshooting more difficult. Unless I am missing that feature somehow?

I tried setting the interface to static and back to DHCP and it refreshed the interface but still shows as down.

I'd check the modem for whether pings are allowed, except it is a Huawei HG612 and it is locked down - don't really want to have to go putting the hacked firmware on it really.

Yeah their involvement should be better here - but again - they have to balance HOW MUCH support they provide to folks using their product for free against supporting paid customers. It will likely get better, given some time. I've seen SOME involvement by their guys here... We will also be able to judge their commitment to a free product by the support response. It may be that they decide they really don't care about the free product. I'd hate to see that - as it can provide valuable feedback and exposure - but we will ultimately "just have to see."

RE web filtering - Back in the day, everything went over HTTP except for specific bits that *needed* encryption. Now, HTTPS now is on for a lot of websites *by default* for all traffic. HTTPS filtering is something that is not to be done half-way, as it REALLY can, as you've seen, screw with services. It IS complex, but NOT as complex as it sounds. Unfortunately, in order to do more than basic category blocking of HTTPS traffic that doesn't always work thanks to wildcard certs, CDNs & SSL accelerators like Cloudflare, etc, you have to effectively perform a "man in the middle" attack on your own traffic. That means having the XG presenting your clients with a "fake" version of the legitimate website's certificate, and then the XG communicates with the secure website. In order to do that, the XG has created its own CA certificate. You just need to trust that certificate. Some devices can't do that. Most devices can.

Go here: Protection > Web Protection > Web Content Filter, find HTTPS Scanning CA. See which one is listed. (I can't remember which is the default, and we use a custom CA here)

Then go here: Objects > Identity > Certificate Authority, find the CA listed above and click the download button on the far right.

Then, install that onto each computer in the trusted root store. See KB article here: www.sophos.com/.../42153.aspx for instructions. Those instructions are for the web appliance, but starting at the end of #1, it becomes generic.

To exempt devices from policy, you can either create rules for them based on their IP, or create clientless users here: Objects > Identity > Clientless Users for your other devices and then create one rule exempting the clientless users group from from https filtering. You can also create a separate wireless network on a separate network segment and exempt all of that traffic from scanning - almost like a guest wifi or something. There are options.

In case you can't tell, I am a *BIG* proponent of HTTPS filtering, and doing anything I can to improve the understanding of HTTPS filtering. It CAN be a pain... but the benefits are excellent and well worth a little bit of setup pain.

Also - I will say this - XG's HTTPS filtering is about as good as I've found. It beats watchguard's, and a few others.

Unfortunately, HTTPS filtering for home use is just the other side of "pain in the ass." I still haven't implemented it at home properly yet - for the same reasons you mentioned. It just broke too much stuff "out of the box" and I haven't had time to do it properly yet.

Good luck!

I would like to thank you for taking the time try and explain things to us. I do understand this is a new product in fact its very much v1 as has been said many times in the beta program when I was a part of it that said communication from Sophos even in the beta program was none existent and when a company is asking for are help in testing and asking for our feedback and it falls a dead ears with no response that's never a good thing.

As for Netflix not working on mobile devices this is something going back to UTM9.x at least in UTM9 we could fix it with some Regex entries and move on with our lives without completely disabling web filter and HTTP scanning, Its not just when HTTPS scanning is enabled its when plane HTTP scanning us used. No matter what we do with different combinations if you have ANY of the following 3 things active it breaks Netflix on mobile devices

web filter
HTTP Scanning
HTTPS Scanning

You go ChavousCamp, We are counting on you to get Netflix working. I am not sure what thread you where talking about for the Regex exemptions but have a look at Drashnas blog for a great list regex entries for many things that work great in UTM9-
drashna.net/.../

That's a better one than the one I found. Mine was an old forum post here: www.astaro.org/.../50903-exception-rule-netflix-streaming.html

Drashna's is much more thorough. I'll look through his when I get ready to do this.

My pleasure. The old Sophos forums are a bit of a mess when it comes to all the suggestions regarding Netflix not working. Chris the owner of Drashna blog belongs to one of the forums that i am pretty active in called
homeservershow.com and I must say he does great work, You can trust that everything you read on that blog I sent you has been tested and works. If you get this working your going to make a lot of peaple very happy this holliday season, myself included.

If you get a chance have a look at the frustration about the new XG Firewall regarding Netflix not working as well as other things. See the link below to a thread I started on the HSS, My forum name is itGeeks
homeservershow.com/.../index.php

Ok, so I won't say "challenge completed" yet - because I cheated to get it working... still working on the webfilter side, but here is one SURE-FIRE way to get it to work while keeping filtering on for the majority of users...

So we'll say... CHALLENGE PENDING for now - and use this to get it working!

1) Create a clientless users group
Objects > Identity > Groups
Add.
Enter a group name.
Group Type: Clientless
Quarantine Digest: Disable
Save.

2) Create Clientless users for each exempt device.
Objects > Identity > Clientless Users
Add (or add range, if they are in a specific range)
Enter a username - something descriptive for the device (ex: ccamp-iphone)
IP Address: the internal ip address
Group: The clientless group you created in step 1
Name: Some name. Descriptive. "My Iphone"
Email: fake an email address. Next version won't require this.
Description: More useless description info. Not required.

Click the plus sign if you need to add additional devices.

Click save.

3) Create security policy
Security Policies
Click on your HTTPS filter rule, click the plus sign, click "Above (User/Network Rule)"
About this rule---
Name: Allow Clientless to Bypass Filter
Identity---
Match Rule based on user identity: On
User or Groups: Clientless Group created in step 1
Source---
Zone: Lan
Networks: Any
Services: HTTP,HTTPS, others if you need them, but these suffice for this walkthrough
Destination---
Zone: WAN
Networks: Any
Malware Scanning---
Scan FTP OFF
Scan HTTP OFF
Decrypt & Scan HTTPS OFF

Save.

This bypasses the specific clientless devices you created from the webfilter entirely. This is actually a reasonably good solution - and may be the "best" solution for roku/appletv/chromecast and other fixed devices that do not regularly leave your network.

This is NOT the best solution for mobile devices, but it will work. The best solution for mobile devices would be to either FIX the damned web filter (still working on it) or create a separate wireless network for them.

First I would like to say I am grateful for all your hard word on this, I further more agree that Sophos just needs to fix this very old streaming problem, This was a problem in UTM9 as well but at least we could fix it with regex entries and NOT have to completely bypass devices like we do now to get it working. Does none of the Sophos employees have mobile devices and use services such as Netflix? That would be mind blowing if they did not. It seems to me that Sophos does not want anyone using such services if using there product.

No I mean no harm in what I am about to say & again I am very great full for your hard work but is what you posted not the long-hand way of solving the Netflix problem vs what we are already doing in a short-hand way? The way I solved the Netflix problem is easy, I created a policy for all my mobile devices called streaming devices and I just turned off HTTP and HTTPS scanning and I disabled the web filter and Netflix works. Am I missing something with your way of doing this? Just seems like a lot of extra work with no benefit unless I am missing something?