An idea for reporting on HTTPS traffic that is exempted from scanning.

I am not entirely sure what you are asking for.  Are you talking about Web traffic that you have going through the proxy with an exception in place to turn off HTTPS decryption?  If so, you should still get categorization performed.

v18 has a bunch of new features and modes for HTTPS.  You could wait for that to see if it gives you what you are after, or more clearly state what it is you are looking for.

Hi Michael,

(super excited for the HTTPS stuff in v18, would love to have a catch up in email but will probably have to wait till beta/nda lift)

I think i can see what is being asked, you bilaterally scan the traffic by duplicating the traffic as it goes through, decrypting that so you can see what is going on without actually interfering with the client-server communications.

However, this won't work without brute forcing the connection (or doing naughty things) because the encryption negotiation with HTTPS is a secure end to end encryption and the key handshake is not visible unless you forcibly intercept it (like classic proxy).

It's a common idea but the performance requirements would also be as bad as just decrypting and scanning it in the first place frankly.

Emile

Hi Michael,

(super excited for the HTTPS stuff in v18, would love to have a catch up in email but will probably have to wait till beta/nda lift)

I think i can see what is being asked, you bilaterally scan the traffic by duplicating the traffic as it goes through, decrypting that so you can see what is going on without actually interfering with the client-server communications.

However, this won't work without brute forcing the connection (or doing naughty things) because the encryption negotiation with HTTPS is a secure end to end encryption and the key handshake is not visible unless you forcibly intercept it (like classic proxy).

It's a common idea but the performance requirements would also be as bad as just decrypting and scanning it in the first place frankly.

Emile

Thank you both for your replies. I see a lot of unidentified tcp 443 traffic (over 300Mb) a day go through and was trying to suggest away of identifying it. I have fine tuned my exceptions and reduced the amount but there is still a lot.

I am looking forward to trying v18.

Ian

I do know what some of the uncategorised data is and that is this site, sophos updates, sophosxi, but not the unclassified stuff.

When you say unidentified tcp 443 traffic what do you mean?
Is this traffic that is going through the transparent web proxy?
If it is, is this traffic that is hitting an Web Exception?

All traffic that goes through the proxy, even if it hits an exception, should still be logged with a source and destination FQDN.  Depending on the exception, it may or may not have a category.  So when you say "unidentified" I am wondering if you are talking about some other traffic not going through the proxy.

Hi Michael,

I am probably misleading in calling the traffic unidentified when it is really unclassified. I have searched the logs and cannot find any traffic on port 443 not going through the proxy.

If I expand the entry in the Report it show all going through the proxy (rule). 

I can identify the IP address of each entry.

Ian

Hi Michael,

I have identified some of the traffic and that is for Sophos Home and Sophos Home Premium products they are classified as NONE. Shouldn't they be classified as IT? All those Sophos sites will have to be added as exceptions when someone applies the Psiphon 3 blocking configuration?

Ian

This is an an unfortunate consequence of exceptions.  When a site is placed in an exception for Policy we no longer do categorization on it, which means that a lot of Sophos sites show up as None/Uncategorized.  The rationale is that we will not be applying any categorization policy decisions, and we want to streamline and have the highest efficiency for any traffic that hits those exceptions.

You can double check, but I willing to bet that the Sophos traffic you see falls under an exception, which is why it is None.

Therefore there should be no problem if someone blocks uncategorized/none, or with Psiphon.

Hi Michael,

I am probably misleading in calling the traffic unidentified when it is really unclassified. I have searched the logs and cannot find any traffic on port 443 not going through the proxy.

If I expand the entry in the Report it show all going through the proxy (rule). 

I can identify the IP address of each entry.

Be aware of the difference between a website (and website category) and an application (and application category).

Websites are categorized by the awarrenhttp proxy, only applies to port 80 and 443, and are done by looking up the URL in a categorization database.

Applications are normally identified by IPS/snort and can occur on any port.  In the case of HTTPS traffic on port 443, IPS is not about to decode the traffic and therefore relies on awarrenhttp to identify any application.  awarrenhttp may say there is no application because the website is not associated with any application (wafflehouse. com is not an application) or because there is a web exception that says not to do policy.

In this report I am not entirely sure what TCP:443 represents.  But I suspect it is any HTTPS traffic that does not have an application associated with it.  Which is perfectly normal, there are millions of HTTPS websites that are not applications.

For website categorization, the uncategorized can mean that Sophos has not categorized the site yet (you could complain that Sophos needs to do more) or that there is an exception.

For applications, the uncategorized can means that Sophos does not detect an application for those site (perfectly normal) or that there is an exception.

No Website categorization = bad.  No application = normal.

Hi Micheal,

thank you for those detailed explanations. I have been investigating further and found some of the 443 traffic appears rot be streaming video/audio as well as other exclusions.

So in summary, using the http proxy in direct mode does not aid or assist with application/web classification or category?

Ian