Web filtering https option in UTM console possibly mislabeled.

Hi Richard,

It is not a typo, this option does indeed provide you the ability to disable web filtering for all HTTPS traffic when on Transparent Mode. 

Could you confirm what you mean when you say that "with the box checked https traffic is being proxied by the UTM in transparent mode and with the box unchecked https traffic is NOT being proxied in transparent mode"

If you mean that with this option enabled, you are able to reach https sites with no problems and with it disabled, you are unable to. That is simply due to the proxy being disabled, so all https traffic is just allowed to flow. To transparently filter HTTPS traffic, you must enable Decrypt & Scan which requires deploying the Proxy CA: Sophos UTM: How to Deploy the Web Protection Proxy CA

Please let me know if you have any questions.

Thanks,
Karlos

Hi Karlos,

Thanks for your explanation, but I still think that check box is mislabeled and your last paragraph helps me confirm it.  Here's what I have going on, I am using a commercial application that uses HTTPS and it has been working for years with that check box checked, i.e. if we assume the box is currently labeled correctly, having the check box checked means "Do not proxy HTTPS traffic in transparent mode" implying that the proxy will NOT be involved in HTTPS.  I know the application's destination server is AWS (Amazon Web Services).  The application broke on it's own on approximately 1/18/2018 which I believe coincided with Amazon removing support for flawed security protocol TLS 1.1 (the server at Amazon is currently ONLY supporting TLS 1.2).

I believe there are some community requests and articles indicating that the Sophos Web proxy is NOT yet supporting TLS 1.2.  So, what I know is I can un-check this box, which to me means disable this option (but, the option is described with the word 'not' in it, so un-checking this box would mean to enable the proxy for HTTPS.  As of 1/18/2018, un-checking this box makes the application work, not the other way around.   I can simply check the box and the application will NOT work, un-check the box and the application works again.  If anyone is having trouble following this paragraph it's likely because the check box was labeled with the word 'not' in it which I think is not the best practice.  Contrast to if the check box was labeled 'Proxy HTTPS traffic in transparent mode' versus how it is currently labeled, checking the box would clearly mean the proxy would be involved in HTTPS, un-checking, i.e. disabling the option, would mean the proxy would NOT be involved in HTTPS traffic.  I am confident the application works with TLS 1.2 and confirmed this by temporarily putting a very old OTS router temporarily in place (a router that would not have provided a Web proxy) and the application worked.

In summary, I have an application that has worked for years with this box checked, now I have to un-check the box for the application to work.  Just the opposite case of what you described in your last paragraph.

thanks,

Richard

Bob,

Exactly and me either.  But, unchecking the box fixes the issue.  The camera is partially functional, it has a live view capability which works with or without the box checked.  However, it has two upload capabilities, take a photo and upload it, or take a video clip and upload it.  Upload to AWS does not work at all with the box checked.  I can pick any camera's IP, and watch/capture the traffic flow from the UTM.  There is definitely attempted traffic from the camera through the UTM to AWS, but when the box is checked I can tell that the communication aborts before finishing.  The way I can tell is that I can find IP address of the AWS server in the packetfilter.log where it is dropped with fwrule=60001 (i think is the correct number).  Anyhow, it's the fwrule that indicates this is attempted unsolicited input, i.e. the firewall has nowhere to direct the last packet received to since the connection has been torn down.

Since we're on the topic, I made a post a while pack about this stealthiness of the UTM.  I could do a 'netstat -an' on the UTM and there would be no connections for a specific internal IP address even though I know the internal IP had connections through the router.  I don't know if something similar is going on here, but it is very strange to me, box checked or not, the internal IP address of the camera does not appear in any of the logs, much less http.log.

I don't know whether this may help you pinpoint it or not, but I have a more complex environment than most.  I put my IOT devices off on their on VLAN (interface eth0.3).  Wonder does that have anything to do with the IPs on that VLAN not showing in the logs?  You are exactly correct though, in a perfect world unchecking or checking that box shouldn't make any difference for this application.  But, I am telling you as of like 1/18/2018 it most certainly does.  I almost never found the workaround for this issue because strangely, going to the first tab and turning off the green switch on the top right had no bearing on the this problem. No matter whether the switch on the first tab is on or off, if that box on the second tab is checked the uploads fail. That was misleading at best.

thanks,

Richard

"60001" means that's it's a default drop out of the INPUT chain, Richard.  Read #1 in Rulz and try looking at the Intrusion Prevention log.  Also, show us the related "60001" line from the full Firewall log file.

Cheers - Bob

Bob,

I appreciate that and I know what 60001 means.  With the box unchecked the 60001 does not occur.  With the box checked the 60001 does occur somewhere in the middle of the communication; I conjecture connection has broken down, i.e. the UTM sees it as unsolicited traffic.  It was just something I noticed when I was debugging this problem.

Back on topic, uncheck the box the application works, check the box, it does not.  Ideally, the application should work with or without proxy, unless of course it is purposely configured to be filtered by the proxy.  Since I am still convinced that the box labeled 'Do not proxy https in transparent mode' is labeled backwards.  Can you point me to an easy test/example to convince me otherwise?  I assume I should be able to see something from a 'tail -f http.log'.

thanks,

Richard

Yes, tail -f /var/log/http.log would indeed do the trick, Richard.

I still think there' something else going on.

Cheers - Bob

Like I said, Richard, I think there's something else going on with the camera.  You can see if a PC browser's HTTPS traffic is handled by the Proxy depending on whether you check that box.  Give WebAdmin and the configuration daemon a few minutes to complete rewriting the code that runs the UTM before you test a change - I would expect this particular change to take place quickly, but I haven't tested it.

Cheers - Bob

Bob, Karlos,

Thanks for your replies and patience.  After further research I have concluded that the box is indeed labeled the way it is currently working and it is the proxy that is now allowing blinkforhome to work for me.  I was misled by at least three items: 1. I determined that AWS is now only supporting TLS 1.2 so it was difficult for me to reconcile why/how a very old Engenius wireless router allowed the cameras to work again (I can only conjecture that Amazon still has a common cipher active with the old router, and oh BTW, with the older cipher settings on the UTM).  2.  I had accidentally typed the wrong subnet prefix for the camera, so the camera's IP was appearing in the http.log after all.  3. I was not able to speak with a support engineer at Blink until late in the game and I'm not sure he was confident one way or the other about TLS 1.2 support.

I apologize, you were correct all along.  When the box is NOT checked, then proxy gets involved in https.  I do stand by the box should be relabeled to "Proxy HTTPS in transparent mode" and the code should be changed to match, but that is an interface opinion and at least the setting currently operates according to the current wording.

One strange anomaly, I did see a stray 'url=https://.....' a couple of minutes or so after I had checked the box and hit apply.  But, maybe that was just during transition to the other mode.  What is weird about that is checking the box and hitting the apply pretty much instantaneously breaks the cameras.  So, the problem is still a mystery since I had this box checked for at least a couple of years.  I am open to any ideas and will continue to talk with blink; I would think with the size of their customer base there would be a lot of complaints, but I would say by far most of their customers are using OTS cheap routers.  What settings are in play when the box is checked, is that just straight NATing?

thanks,

Richard

When the box is checked, it's the firewall rules that determine if the traffic passes and then, yes, you do need a Masquerading rule for the traffic.

Cheers - Bob

Bob,

Masquerading for that VLAN (3) is and has been in place all along.  So, there is some other reason why with the box checked the camera can no longer upload to AWS.  Is there a log snippet I can get you that may help to debug?

thanks,

Richard

Hey Richard,

Thanks for the update. As Bob mentioned, checking that box then passes the responsibility over to the Firewall to either allow or block the traffic. Have you filtered your firewall logs by the Camera IP/subnet to see whether packets are being dropped? You could also look into your IPS logs, if it's enabled. 

Cheers,
Karlos

Hey Richard,

Thanks for the update. As Bob mentioned, checking that box then passes the responsibility over to the Firewall to either allow or block the traffic. Have you filtered your firewall logs by the Camera IP/subnet to see whether packets are being dropped? You could also look into your IPS logs, if it's enabled. 

Cheers,
Karlos

Hi Karlos,

When the box is checked, the only logs where the camera's IP address appears are as follows:

1. confd-debug.log

2. confd.log

3. dhcpd.log

4. http.log

5. system.log

I don't think any of the logs will help us; number 1, 2, and 5 mention the IP only for license tracking; 3 has to do with the dhcp lease of the camera's IP; 4 has no log lines after I checked the box for this test.  I can use tcpdump to capture a trace from the camera, but I cannot determine why the connection terminates prematurely.  All of this worked prior to 1/18/2018.

From the Linux command line or the Web interface is there a way I can tell if any threat patterns were updated around that date?  I do not have automatic firmware update enabled and I don't recall updating the firmware around 1/18.

thanks,

Richard

As root:

 zgrep '"Successfully installed' /var/log/up2date/2018/01/up2date-2018-01-17.log.gz
 zgrep '"Successfully installed' /var/log/up2date/2018/01/up2date-2018-01-18.log.gz

Cheers - Bob

2018:01:17-00:00:59 gopierce auisys[20280]: id="371Z" severity="info" sys="system" sub="up2date" name="Successfully installed Up2Date package" status="success" action="install" package_version="9.27137" package="aptp"
2018:01:17-01:15:44 gopierce auisys[26944]: id="371Z" severity="info" sys="system" sub="up2date" name="Successfully installed Up2Date package" status="success" action="install" package_version="9.8974" package="avira-xvdf"
2018:01:17-01:15:59 gopierce auisys[26944]: id="371Z" severity="info" sys="system" sub="up2date" name="Successfully installed Up2Date package" status="success" action="install" package_version="9.12071" package="savi"
2018:01:17-02:00:43 gopierce auisys[31005]: id="371Z" severity="info" sys="system" sub="up2date" name="Successfully installed Up2Date package" status="success" action="install" package_version="9.27138" package="aptp"
2018:01:17-02:30:36 gopierce auisys[1305]: id="371Z" severity="info" sys="system" sub="up2date" name="Successfully installed Up2Date package" status="success" action="install" package_version="9.8975" package="avira-xvdf"
2018:01:17-03:01:00 gopierce auisys[4090]: id="371Z" severity="info" sys="system" sub="up2date" name="Successfully installed Up2Date package" status="success" action="install" package_version="9.27139" package="aptp"
2018:01:17-04:15:36 gopierce auisys[10866]: id="371Z" severity="info" sys="system" sub="up2date" name="Successfully installed Up2Date package" status="success" action="install" package_version="9.8976" package="avira-xvdf"
2018:01:17-04:30:39 gopierce auisys[12237]: id="371Z" severity="info" sys="system" sub="up2date" name="Successfully installed Up2Date package" status="success" action="install" package_version="9.8977" package="avira-xvdf"
2018:01:17-05:00:42 gopierce auisys[14970]: id="371Z" severity="info" sys="system" sub="up2date" name="Successfully installed Up2Date package" status="success" action="install" package_version="9.27140" package="aptp"
2018:01:17-06:30:37 gopierce auisys[22849]: id="371Z" severity="info" sys="system" sub="up2date" name="Successfully installed Up2Date package" status="success" action="install" package_version="9.8978" package="avira-xvdf"
2018:01:17-06:30:52 gopierce auisys[22849]: id="371Z" severity="info" sys="system" sub="up2date" name="Successfully installed Up2Date package" status="success" action="install" package_version="9.12072" package="savi"
2018:01:17-07:00:39 gopierce auisys[25595]: id="371Z" severity="info" sys="system" sub="up2date" name="Successfully installed Up2Date package" status="success" action="install" package_version="9.27141" package="aptp"
2018:01:17-08:00:38 gopierce auisys[30835]: id="371Z" severity="info" sys="system" sub="up2date" name="Successfully installed Up2Date package" status="success" action="install" package_version="9.27142" package="aptp"
2018:01:17-08:00:55 gopierce auisys[30835]: id="371Z" severity="info" sys="system" sub="up2date" name="Successfully installed Up2Date package" status="success" action="install" package_version="9.8979" package="avira-xvdf"
2018:01:17-08:30:36 gopierce auisys[1115]: id="371Z" severity="info" sys="system" sub="up2date" name="Successfully installed Up2Date package" status="success" action="install" package_version="9.8980" package="avira-xvdf"
2018:01:17-10:00:38 gopierce auisys[9191]: id="371Z" severity="info" sys="system" sub="up2date" name="Successfully installed Up2Date package" status="success" action="install" package_version="9.27143" package="aptp"
2018:01:17-10:30:35 gopierce auisys[11932]: id="371Z" severity="info" sys="system" sub="up2date" name="Successfully installed Up2Date package" status="success" action="install" package_version="9.282" package="aws"
2018:01:17-11:00:38 gopierce auisys[14576]: id="371Z" severity="info" sys="system" sub="up2date" name="Successfully installed Up2Date package" status="success" action="install" package_version="9.27144" package="aptp"
2018:01:17-11:15:36 gopierce auisys[16007]: id="371Z" severity="info" sys="system" sub="up2date" name="Successfully installed Up2Date package" status="success" action="install" package_version="9.8981" package="avira-xvdf"
2018:01:17-12:00:49 gopierce auisys[20209]: id="371Z" severity="info" sys="system" sub="up2date" name="Successfully installed Up2Date package" status="success" action="install" package_version="9.8982" package="avira-xvdf"
2018:01:17-13:00:38 gopierce auisys[25466]: id="371Z" severity="info" sys="system" sub="up2date" name="Successfully installed Up2Date package" status="success" action="install" package_version="9.27145" package="aptp"
2018:01:17-13:30:36 gopierce auisys[28247]: id="371Z" severity="info" sys="system" sub="up2date" name="Successfully installed Up2Date package" status="success" action="install" package_version="9.12073" package="savi"
2018:01:17-14:00:42 gopierce auisys[30910]: id="371Z" severity="info" sys="system" sub="up2date" name="Successfully installed Up2Date package" status="success" action="install" package_version="9.27146" package="aptp"
2018:01:17-14:30:40 gopierce auisys[1118]: id="371Z" severity="info" sys="system" sub="up2date" name="Successfully installed Up2Date package" status="success" action="install" package_version="9.8983" package="avira-xvdf"
2018:01:17-15:30:36 gopierce auisys[6112]: id="371Z" severity="info" sys="system" sub="up2date" name="Successfully installed Up2Date package" status="success" action="install" package_version="9.8984" package="avira-xvdf"
2018:01:17-16:00:46 gopierce auisys[9160]: id="371Z" severity="info" sys="system" sub="up2date" name="Successfully installed Up2Date package" status="success" action="install" package_version="9.27147" package="aptp"
2018:01:17-16:01:03 gopierce auisys[9160]: id="371Z" severity="info" sys="system" sub="up2date" name="Successfully installed Up2Date package" status="success" action="install" package_version="9.8985" package="avira-xvdf"
2018:01:17-16:30:36 gopierce auisys[11560]: id="371Z" severity="info" sys="system" sub="up2date" name="Successfully installed Up2Date package" status="success" action="install" package_version="9.8986" package="avira-xvdf"
2018:01:17-17:00:36 gopierce auisys[14302]: id="371Z" severity="info" sys="system" sub="up2date" name="Successfully installed Up2Date package" status="success" action="install" package_version="9.27148" package="aptp"
2018:01:17-20:00:36 gopierce auisys[29793]: id="371Z" severity="info" sys="system" sub="up2date" name="Successfully installed Up2Date package" status="success" action="install" package_version="9.12074" package="savi"
2018:01:17-20:15:40 gopierce auisys[31135]: id="371Z" severity="info" sys="system" sub="up2date" name="Successfully installed Up2Date package" status="success" action="install" package_version="9.27149" package="aptp"
2018:01:17-21:00:44 gopierce auisys[2716]: id="371Z" severity="info" sys="system" sub="up2date" name="Successfully installed Up2Date package" status="success" action="install" package_version="9.27150" package="aptp"
2018:01:17-22:15:34 gopierce auisys[9643]: id="371Z" severity="info" sys="system" sub="up2date" name="Successfully installed Up2Date package" status="success" action="install" package_version="9.12075" package="savi"
2018:01:17-23:00:32 gopierce auisys[13548]: id="371Z" severity="info" sys="system" sub="up2date" name="Successfully installed Up2Date package" status="success" action="install" package_version="9.27151" package="aptp"

2018:01:18-00:00:59 gopierce auisys[19130]: id="371Z" severity="info" sys="system" sub="up2date" name="Successfully installed Up2Date package" status="success" action="install" package_version="9.27152" package="aptp"
2018:01:18-02:00:43 gopierce auisys[29721]: id="371Z" severity="info" sys="system" sub="up2date" name="Successfully installed Up2Date package" status="success" action="install" package_version="9.27153" package="aptp"
2018:01:18-02:01:00 gopierce auisys[29721]: id="371Z" severity="info" sys="system" sub="up2date" name="Successfully installed Up2Date package" status="success" action="install" package_version="9.8987" package="avira-xvdf"
2018:01:18-03:00:39 gopierce auisys[2655]: id="371Z" severity="info" sys="system" sub="up2date" name="Successfully installed Up2Date package" status="success" action="install" package_version="9.27154" package="aptp"
2018:01:18-04:15:36 gopierce auisys[9494]: id="371Z" severity="info" sys="system" sub="up2date" name="Successfully installed Up2Date package" status="success" action="install" package_version="9.12076" package="savi"
2018:01:18-05:00:43 gopierce auisys[13485]: id="371Z" severity="info" sys="system" sub="up2date" name="Successfully installed Up2Date package" status="success" action="install" package_version="9.27155" package="aptp"
2018:01:18-07:00:39 gopierce auisys[23832]: id="371Z" severity="info" sys="system" sub="up2date" name="Successfully installed Up2Date package" status="success" action="install" package_version="9.27156" package="aptp"
2018:01:18-07:00:56 gopierce auisys[23832]: id="371Z" severity="info" sys="system" sub="up2date" name="Successfully installed Up2Date package" status="success" action="install" package_version="9.8988" package="avira-xvdf"
2018:01:18-08:00:39 gopierce auisys[29105]: id="371Z" severity="info" sys="system" sub="up2date" name="Successfully installed Up2Date package" status="success" action="install" package_version="9.27157" package="aptp"
2018:01:18-08:45:37 gopierce auisys[746]: id="371Z" severity="info" sys="system" sub="up2date" name="Successfully installed Up2Date package" status="success" action="install" package_version="9.12077" package="savi"
2018:01:18-10:15:38 gopierce auisys[8999]: id="371Z" severity="info" sys="system" sub="up2date" name="Successfully installed Up2Date package" status="success" action="install" package_version="9.27158" package="aptp"
2018:01:18-12:00:51 gopierce auisys[18255]: id="371Z" severity="info" sys="system" sub="up2date" name="Successfully installed Up2Date package" status="success" action="install" package_version="9.27159" package="aptp"
2018:01:18-13:15:36 gopierce auisys[24901]: id="371Z" severity="info" sys="system" sub="up2date" name="Successfully installed Up2Date package" status="success" action="install" package_version="9.8989" package="avira-xvdf"
2018:01:18-13:30:39 gopierce auisys[26283]: id="371Z" severity="info" sys="system" sub="up2date" name="Successfully installed Up2Date package" status="success" action="install" package_version="9.8990" package="avira-xvdf"
2018:01:18-14:00:38 gopierce auisys[29036]: id="371Z" severity="info" sys="system" sub="up2date" name="Successfully installed Up2Date package" status="success" action="install" package_version="9.27160" package="aptp"
2018:01:18-14:30:42 gopierce auisys[31849]: id="371Z" severity="info" sys="system" sub="up2date" name="Successfully installed Up2Date package" status="success" action="install" package_version="9.8991" package="avira-xvdf"
2018:01:18-15:00:36 gopierce auisys[2069]: id="371Z" severity="info" sys="system" sub="up2date" name="Successfully installed Up2Date package" status="success" action="install" package_version="9.8992" package="avira-xvdf"
2018:01:18-16:00:43 gopierce auisys[7567]: id="371Z" severity="info" sys="system" sub="up2date" name="Successfully installed Up2Date package" status="success" action="install" package_version="9.27161" package="aptp"
2018:01:18-17:00:35 gopierce auisys[13024]: id="371Z" severity="info" sys="system" sub="up2date" name="Successfully installed Up2Date package" status="success" action="install" package_version="9.27162" package="aptp"
2018:01:18-17:30:36 gopierce auisys[15804]: id="371Z" severity="info" sys="system" sub="up2date" name="Successfully installed Up2Date package" status="success" action="install" package_version="9.12078" package="savi"
2018:01:18-21:00:36 gopierce auisys[1479]: id="371Z" severity="info" sys="system" sub="up2date" name="Successfully installed Up2Date package" status="success" action="install" package_version="9.12079" package="savi"

thanks,

Richard

So, we don't see any Snort signature updates and since the accesses weren't going through the Proxy before, no anti-virus was being applied.  That just leaves aptp - Application Control.  Are there any differences in what's in that log on 1/16 and 1/19?

Cheers - Bob

Bob,

Both of those logs are zero length.

thanks,

Richard

If Sophos Support can't help, it may be time to get some backups off the box and reload from ISO.  Before doing that, try restoring a backup from before the problem began - don't forget to make a new backup first!

Cheers - Bob

Thanks Bob.  I had already tried a restore some time ago, didn't help.  I do not have a backup of the entire filesystem and wish I did.  Maybe if I get some time I will save off an image and try from the ISO,  but I really don't want to do that if at all possible.  Maybe Sophos support will have some ideas.  Checking or not checking that box shouldn't impact whether the https communication works or not (unless, of course the site is explicitly and purposely filtered).  Who knows what changed?  Potentially something on AWS which is a complex environment where regional servers are scattered all of the world.  But, it's not great for Sophos since cheap OTS routers are not having problems.

Richard

Any updates on this mystery, Richard?

Cheers - Bob

Hi Bob,

Thanks for checking back, but no updates yet.  I was hoping Sophos support would join in at some point.

I'm confident there is an issue, but I'm at a loss as to what it is.  Applications should work with or without proxy. Blinkforhome worked for at least two years without proxy, now (as of approximately 1/18/2018) https proxy is required to work.  Check the 'Do not proxy https...' box it will break Blinkforhome, i.e. uploads of photos and video to AWS.  My workaround for now is to uncheck the box.

thanks,

Richard