Web filtering https option in UTM console possibly mislabeled.

Hi Richard,

It is not a typo, this option does indeed provide you the ability to disable web filtering for all HTTPS traffic when on Transparent Mode. 

Could you confirm what you mean when you say that "with the box checked https traffic is being proxied by the UTM in transparent mode and with the box unchecked https traffic is NOT being proxied in transparent mode"

If you mean that with this option enabled, you are able to reach https sites with no problems and with it disabled, you are unable to. That is simply due to the proxy being disabled, so all https traffic is just allowed to flow. To transparently filter HTTPS traffic, you must enable Decrypt & Scan which requires deploying the Proxy CA: Sophos UTM: How to Deploy the Web Protection Proxy CA

Please let me know if you have any questions.

Thanks,
Karlos

Hi Karlos,

Thanks for your explanation, but I still think that check box is mislabeled and your last paragraph helps me confirm it.  Here's what I have going on, I am using a commercial application that uses HTTPS and it has been working for years with that check box checked, i.e. if we assume the box is currently labeled correctly, having the check box checked means "Do not proxy HTTPS traffic in transparent mode" implying that the proxy will NOT be involved in HTTPS.  I know the application's destination server is AWS (Amazon Web Services).  The application broke on it's own on approximately 1/18/2018 which I believe coincided with Amazon removing support for flawed security protocol TLS 1.1 (the server at Amazon is currently ONLY supporting TLS 1.2).

I believe there are some community requests and articles indicating that the Sophos Web proxy is NOT yet supporting TLS 1.2.  So, what I know is I can un-check this box, which to me means disable this option (but, the option is described with the word 'not' in it, so un-checking this box would mean to enable the proxy for HTTPS.  As of 1/18/2018, un-checking this box makes the application work, not the other way around.   I can simply check the box and the application will NOT work, un-check the box and the application works again.  If anyone is having trouble following this paragraph it's likely because the check box was labeled with the word 'not' in it which I think is not the best practice.  Contrast to if the check box was labeled 'Proxy HTTPS traffic in transparent mode' versus how it is currently labeled, checking the box would clearly mean the proxy would be involved in HTTPS, un-checking, i.e. disabling the option, would mean the proxy would NOT be involved in HTTPS traffic.  I am confident the application works with TLS 1.2 and confirmed this by temporarily putting a very old OTS router temporarily in place (a router that would not have provided a Web proxy) and the application worked.

In summary, I have an application that has worked for years with this box checked, now I have to un-check the box for the application to work.  Just the opposite case of what you described in your last paragraph.

thanks,

Richard

Hi Richard,

Thanks for providing the details. UTM 9.5 does offer TLS 1.2 support. This KB is dated, and RC4 should now be removed from the cipher list but follow for commands to display cipher list and to manually set TLS 1.2: HTTPS Encryption for HTTP Proxy 

P.S. note that a "!" before a cipher, means it's not being used.

Also, do you have HTTPS scanning enabled or just URL filtering? If the latter, then it will only check the domain against blocked categories or blacklist. If it is not found in either, it should let your application through, hence why it works with the option not checked. As per why the application stops working when the option is checked, maybe you can look through your firewall logs using the destination IP for any blocks. You can also look through your Application Control logs if it is enabled.

Thanks,

Karlos

Hi Karlos, 

Thank you for the link to the updated Ciphers tip to support TLS 1.2.  But the plot thickens, I have the radio button 'URL filtering only' selected and I updated the ciphers as follows:

# cc get http tlsciphers_server
TLSv1.2:HIGH:!MD5:!aNULL:!EDH

When the check box labeled 'Do not proxy HTTPS traffic in transparent mode' is checked then the application still does not work.  I'm kinda at a loss.

Richard

What application doesn't work, Richard, what do you see when it doesn't work and what do you see in the Web Filtering log?

Cheers - Bob

Hi Bob,

The application is BlinkForHome.  Each camera has it's own internal IP address and the camera communicates with via https to an AWS (Amazon Web Services) local region server.  I have used Sophos UTM since it was Astaro and I have had the camera system for at least three years.  All have worked very well together, but something broke the app on approximately 1/18.  I conjecture, but do not know for sure that Amazon improved security by dropping a cipher that the UTM proxy was using.  I have set both of the following confd settings:

# cc get http | grep -i cipher

          'tlsciphers_client' => 'TLSv1.2:HIGH:!MD5:!aNULL:!EDH',

          'tlsciphers_server' => 'TLSv1.2:HIGH:!MD5:!aNULL:!EDH',

And, while I think this cipher list is improved over the outdated one, it has not done anything for resolving this issue.  I'm not even sure these settings have anything to do with the proxy.  There are reverseproxy settings, but I didn't see any config item under there that contained ciphers.

In summary, for years I have had the box labeled 'Do not proxy https in transparent mode' checked and the cameras were fully functional.  Now, I have to uncheck that box to make them work.  I assumed the box was labeled backwards.  To test my theory, I removed the Sophos UTM and my normal wireless Access Points and temporarily replaced my entire network with a very old Engenius router with wireless that would have not provided a Web proxy, i.e. straight NAT and the cameras worked.  That tells me the cameras have no problem uploading to AWS directly without a proxy.  Back to the UTM, uncheck the box, cameras work.  Check the box, they do not.  However, if we take the check box as it is currently labeled, unchecking would mean that the proxy is involved, while checking it would mean that the proxy is NOT involved.  Seems to me to be too much evidence that this is completely backwards.

Also, I would be glad to check logs if I knew what to look for.

thanks,

Richard

Hi Bob,

The application is BlinkForHome.  Each camera has it's own internal IP address and the camera communicates with via https to an AWS (Amazon Web Services) local region server.  I have used Sophos UTM since it was Astaro and I have had the camera system for at least three years.  All have worked very well together, but something broke the app on approximately 1/18.  I conjecture, but do not know for sure that Amazon improved security by dropping a cipher that the UTM proxy was using.  I have set both of the following confd settings:

# cc get http | grep -i cipher

          'tlsciphers_client' => 'TLSv1.2:HIGH:!MD5:!aNULL:!EDH',

          'tlsciphers_server' => 'TLSv1.2:HIGH:!MD5:!aNULL:!EDH',

And, while I think this cipher list is improved over the outdated one, it has not done anything for resolving this issue.  I'm not even sure these settings have anything to do with the proxy.  There are reverseproxy settings, but I didn't see any config item under there that contained ciphers.

In summary, for years I have had the box labeled 'Do not proxy https in transparent mode' checked and the cameras were fully functional.  Now, I have to uncheck that box to make them work.  I assumed the box was labeled backwards.  To test my theory, I removed the Sophos UTM and my normal wireless Access Points and temporarily replaced my entire network with a very old Engenius router with wireless that would have not provided a Web proxy, i.e. straight NAT and the cameras worked.  That tells me the cameras have no problem uploading to AWS directly without a proxy.  Back to the UTM, uncheck the box, cameras work.  Check the box, they do not.  However, if we take the check box as it is currently labeled, unchecking would mean that the proxy is involved, while checking it would mean that the proxy is NOT involved.  Seems to me to be too much evidence that this is completely backwards.

Also, I would be glad to check logs if I knew what to look for.

thanks,

Richard

Bob,

Wanted to add one more interesting detail, the LAN (internal) IP address of the camera doesn't appear in any of the logs in /var/log; that's why I haven't been that interested in searching http.log.  I can of course glean a few things from tcpdump.

thanks,

Richard

What are you seeing that leads you to conclude the cameras aren't functional?  Does anything work with them?

If there's no evidence of the traffic in the Web Filtering log, I don't understand why unchecking the box made any difference.

Cheers - Bob

Bob,

Exactly and me either.  But, unchecking the box fixes the issue.  The camera is partially functional, it has a live view capability which works with or without the box checked.  However, it has two upload capabilities, take a photo and upload it, or take a video clip and upload it.  Upload to AWS does not work at all with the box checked.  I can pick any camera's IP, and watch/capture the traffic flow from the UTM.  There is definitely attempted traffic from the camera through the UTM to AWS, but when the box is checked I can tell that the communication aborts before finishing.  The way I can tell is that I can find IP address of the AWS server in the packetfilter.log where it is dropped with fwrule=60001 (i think is the correct number).  Anyhow, it's the fwrule that indicates this is attempted unsolicited input, i.e. the firewall has nowhere to direct the last packet received to since the connection has been torn down.

Since we're on the topic, I made a post a while pack about this stealthiness of the UTM.  I could do a 'netstat -an' on the UTM and there would be no connections for a specific internal IP address even though I know the internal IP had connections through the router.  I don't know if something similar is going on here, but it is very strange to me, box checked or not, the internal IP address of the camera does not appear in any of the logs, much less http.log.

I don't know whether this may help you pinpoint it or not, but I have a more complex environment than most.  I put my IOT devices off on their on VLAN (interface eth0.3).  Wonder does that have anything to do with the IPs on that VLAN not showing in the logs?  You are exactly correct though, in a perfect world unchecking or checking that box shouldn't make any difference for this application.  But, I am telling you as of like 1/18/2018 it most certainly does.  I almost never found the workaround for this issue because strangely, going to the first tab and turning off the green switch on the top right had no bearing on the this problem. No matter whether the switch on the first tab is on or off, if that box on the second tab is checked the uploads fail. That was misleading at best.

thanks,

Richard

"60001" means that's it's a default drop out of the INPUT chain, Richard.  Read #1 in Rulz and try looking at the Intrusion Prevention log.  Also, show us the related "60001" line from the full Firewall log file.

Cheers - Bob

Bob,

I appreciate that and I know what 60001 means.  With the box unchecked the 60001 does not occur.  With the box checked the 60001 does occur somewhere in the middle of the communication; I conjecture connection has broken down, i.e. the UTM sees it as unsolicited traffic.  It was just something I noticed when I was debugging this problem.

Back on topic, uncheck the box the application works, check the box, it does not.  Ideally, the application should work with or without proxy, unless of course it is purposely configured to be filtered by the proxy.  Since I am still convinced that the box labeled 'Do not proxy https in transparent mode' is labeled backwards.  Can you point me to an easy test/example to convince me otherwise?  I assume I should be able to see something from a 'tail -f http.log'.

thanks,

Richard

Yes, tail -f /var/log/http.log would indeed do the trick, Richard.

I still think there' something else going on.

Cheers - Bob

Like I said, Richard, I think there's something else going on with the camera.  You can see if a PC browser's HTTPS traffic is handled by the Proxy depending on whether you check that box.  Give WebAdmin and the configuration daemon a few minutes to complete rewriting the code that runs the UTM before you test a change - I would expect this particular change to take place quickly, but I haven't tested it.

Cheers - Bob

Bob, Karlos,

Thanks for your replies and patience.  After further research I have concluded that the box is indeed labeled the way it is currently working and it is the proxy that is now allowing blinkforhome to work for me.  I was misled by at least three items: 1. I determined that AWS is now only supporting TLS 1.2 so it was difficult for me to reconcile why/how a very old Engenius wireless router allowed the cameras to work again (I can only conjecture that Amazon still has a common cipher active with the old router, and oh BTW, with the older cipher settings on the UTM).  2.  I had accidentally typed the wrong subnet prefix for the camera, so the camera's IP was appearing in the http.log after all.  3. I was not able to speak with a support engineer at Blink until late in the game and I'm not sure he was confident one way or the other about TLS 1.2 support.

I apologize, you were correct all along.  When the box is NOT checked, then proxy gets involved in https.  I do stand by the box should be relabeled to "Proxy HTTPS in transparent mode" and the code should be changed to match, but that is an interface opinion and at least the setting currently operates according to the current wording.

One strange anomaly, I did see a stray 'url=https://.....' a couple of minutes or so after I had checked the box and hit apply.  But, maybe that was just during transition to the other mode.  What is weird about that is checking the box and hitting the apply pretty much instantaneously breaks the cameras.  So, the problem is still a mystery since I had this box checked for at least a couple of years.  I am open to any ideas and will continue to talk with blink; I would think with the size of their customer base there would be a lot of complaints, but I would say by far most of their customers are using OTS cheap routers.  What settings are in play when the box is checked, is that just straight NATing?

thanks,

Richard

When the box is checked, it's the firewall rules that determine if the traffic passes and then, yes, you do need a Masquerading rule for the traffic.

Cheers - Bob