Youtube and ADS - Sometimes they are not blocked

Hi All,

Configure a Web Policy to block "Advertisement" category and enable HTTP & HTTPS scanning. Configure a DROP rule for UDP 443 service and make sure to place it on TOP. YouTube ADS are successfully blocked and during instances of incorrect web categorization, take the steps as suggested by Aditya Patel in his previous post. Refer the screenshots below:

Thanks

Sachin,

I am already using this configuration (apart the UDP 443 blocking rule) which is catched by the default rule at the bottom.

I will check the config and post back again.

Thanks

sachingurung 

I tested your config. Still ADS are not blocked on youtube video. XG is not able to catch the same ADS like McAfee engine on UTM.

Some ADS are blocked, but not all of them.

Let me know if ylou need logs.

Thanks

I will definitely need logs and screenshot. Please DM me. If there are any certain ADS link that is wrongly categorized, which is the reason why ADS are not blocked then, I request all to raise a reassessment request and submit the URL.

Thanks

Hi sachin, this is not directed at you since you are doing the best you can do and other people decide the product efficiency (webfilting in this case). While I agree that we should help sophos by raising a reassessment request if we see something categorized incrorrectly, I think Luk has a larger point in general. Every beta release, we hear that Sophos is fixing and incorporating the features requested by most partners and users.

By raising the ineffectiveness of XG filtering compared to the UTM filtering, he is just pointing out the shortcomings of the in house webfilter in certain configurations. Since you guys are our direct communication line to the higher ups/ concerned parties within sophos organization, we are trying for you to relay the message that some users are sending negative feedback on the webcategorization database... that is all. 

While most admins can submit a bad categorization request, it is not feasible in every case where someone bought the XG as set it and forget it from a reseller that has to constantly explain to them why ad blocking is not working as advertised.

Hope you understand

Regards

Bill 

Sachin,

ADS urls are changing everyday and in order to give you feedback regarding the URL that are not blocked, I have to keep a tail -f command opened on advanced shell and see which ADS URL during Youtube video are displayed, so it is almost impossible.

This is a process that you should take care about. What Billybob wrote is what you should follow as advice. Web Filtering on XG is not enough powerful and cannot even be compared with other UTM Web Filtering engine.

We understand that an URL is not categorized correctly......but ADS should be blocked correctly. At the moment blocking or not blocking ADS on youtube video, produce me the same output.

This is an ongoing process that should take care.

Time for me to put in a few cents...  :)

There are two underlying issues here, functionality and data.

The UTM and the XG use the same underlying web proxy.  There is customization and configuration is different for each product, but the underlying engine is the same.
If anyone is finding that the Web Filtering on UTM is more powerful than the XG, please let me know.  I'm curious what the feature gap is for things that customers use.

The UTM and the XG use different sources of data for categorization.
The UTM uses data from McAfee.
The XG uses data from Sophos.
For reasons both obvious and not, going forward we want to be using data from Sophos in as many products as we can.
We will not be bringing McAfee to the XG.
We want to improve the quality of the Sophos data.

Now Youtube wants people to see ads.  They are going to try to create unblockable ads.
Users don't want to see ads.  So companies like Sophos try to block them.
Some of the blocking of ads is done by categorizing certain URLs and blocking them.
But..  Youtube then is going to change the URLs... or mix real content and ad content within the same URL.
Its an arms race, and its an evolving landscape.

One point is that when v17 comes out (or any new features) that probably wont affect youtube ads.

Hi Michael, I had posted something similar before here about cyren formerly commtouch that was used in UTM for email spam. I understand using sophos engine makes sense moving forward, but maybe the web categories need re-categorization. What I mean by that is XG has a few extra categories that UTM doesn't have. Some of them are redundant (to me atleast) for example uncategorized and none are two seperate categories in XG.

I think none means the website doesn't fall in ANY category while uncategorized means there is no record of it in the database. I guess there could be websites about nothing out there that can't be classified, but it was confusing to me atleast initially. There are others that classify correctly as ads in UTM but are classified IT (information technology) in XG. Here is one

https ://cnn.stream1.fyre.co

It is classified as IT in XG, while UTM classifies it correctly as streaming media. So while XG has extra categories, seems that the categories that matter are not categorized correctly. You wouldn't want to block Information Technology in your work environment whereas streaming media will most likely be under scrutiny. 

This is not a complain for the sake of complaining, I pulled that url within a few seconds by going to cnn .com. You guys don't realistically expect us to give you feedback on every URL do you? As I explained earlier, this creates problems for resellers where bad categorization creates unexpected results and customers complain. Its your product and you can do as you please... we are just sending you a heads up that the categorization database needs some major work and is not comparable to your other flagship UTM product. 

Michael Dunn said:

One point is that when v17 comes out (or any new features) that probably wont affect youtube ads.

Youtube ads is a metaphor for a bigger problem but I have said my piece on the subject. 

I've raised the issue with data quality of advertising internally.  I don't promise anything, but the comments in this thread have been forwarded.  The good news is that web categorization data improvement is not on the release cycle and has a different set of people working on it.

I don't know too much about how the Sophos categorization team works, but I believe most of the work is using automated tools and web crawlers, and that most of the work will focus on categorizing new websites, not on re-evaluating current ones.  For changes to existing categories it works more on reports of poor categorization.  There are millions of websites currently categorized - which ones need to be updated?  It is impractical to try and re-analyze all of them.  They need to rely on complaints.  In addition, some category changes (like to/from Business and malicious) are much more important than others (like to/from Business and Information Technology).  Things like Ads and Streaming Media fall in between those priorities.

The other complication is that a domain may be used for multiple things.  For example, you found that steaming media came from cnn.stream1.fyre.co.  But maybe also the stock ticker comes from there.  Or maybe even html news articles pull some content from that domain.  A categorizer may need to be careful to use the lowest common denominator.  Otherwise you may find that categorizing that domain as streaming media and then blocking it, several parts of cnn.com stop working.  Which then people would complain about -- why is Sophos incorrectly classifying that url as streaming media.  :)  It is a pretend example, but it goes to show that the problem is more difficult than it may first appear.

I agree that that category names, etc, are maybe a bit confusing.  Partly that is just a learning experience for anyone switching from one product to another  Aside from improving documentation, I don't know if we can do anything about that.

For those of you with a long memory.  Do you recall that in UTM the original Astaro it used a categorization service known as CFFS.  In 9.2 we switched it to SXL.  It used the same McAfee database but had several improvements, including the addition of Sophos Labs data source for malicious urls.  We are doing the same thing for XG.  Currently XG uses a service called WINGc.  In v17 we will be switching this to SXL.  This will give us the better data source for malicious URLs.  The evolution of XG is following the same evolution as UTM.

Thanks Michael,

yours is a great news. I think that you should add multiple and different web filtering engine to XG/UTM, specially for customers that are using Sophos Heartbeat. A dual defense on Web Filtering nowadays is a must and if XG and Endpoint use the same engine is not so safe.

As NGFW, XG should "search" web traffic from more than one DB. Think about to take an agreement with other products. The same applies to Sophos Web Appliance (SWA).

Regards

Hi Michael, as always thanks for the concise thoughtful response. You are right about the URL that I mentioned in my post and I am not saying that in certain cases XG won't classify something better. I also realize that change is hard for some of us folks[:$]

Michael Dunn said:

For those of you with a long memory.  Do you recall that in UTM the original Astaro it used a categorization service known as CFFS.  In 9.2 we switched it to SXL.  It used the same McAfee database but had several improvements, including the addition of Sophos Labs data source for malicious urls.  We are doing the same thing for XG.  Currently XG uses a service called WINGc.  In v17 we will be switching this to SXL.  This will give us the better data source for malicious URLs.  The evolution of XG is following the same evolution as UTM

Thanks again for taking the time and raising the issue with the correct team.

Hi Michael, as always thanks for the concise thoughtful response. You are right about the URL that I mentioned in my post and I am not saying that in certain cases XG won't classify something better. I also realize that change is hard for some of us folks[:$]

Michael Dunn said:

For those of you with a long memory.  Do you recall that in UTM the original Astaro it used a categorization service known as CFFS.  In 9.2 we switched it to SXL.  It used the same McAfee database but had several improvements, including the addition of Sophos Labs data source for malicious urls.  We are doing the same thing for XG.  Currently XG uses a service called WINGc.  In v17 we will be switching this to SXL.  This will give us the better data source for malicious URLs.  The evolution of XG is following the same evolution as UTM

Thanks again for taking the time and raising the issue with the correct team.

The main change from WINGc to SXL is we are changing the cloud servers and the method of communicating with them.  The data backing them is 99% the same.
The difference is that WINGc (from Cyberroam days) does not contain Sophos Labs security data.
Rather than adding the Sophos Labs data to WINGc, we have added the Cyberoam-now-Sophos categorization data to SXL, and are moving XG to SXL which now has both.
Sophos Labs data is based on emerging threats and is managed more like antivirus data.  Security data is often shared between companies because security is more important than proprietary.
It is a small drop in the total categorization data, yet it is an important one.

Endpoints are complicated because where they get categorization data can change depending on the product they are associated with.  There are now also several different endpoint products.

I'm not in the newsgroups daily, my involvement ebbs and flows with my other workload.  I've got a fine line to walk about what I can/should talk about and not.  I am also sometimes limited in what I can actually do about issues, Support and Partner channels sometimes being the better options.  That being said, I think teasers and insights like this are good.  I'm running SXL on my test v17 XG box right now.  :)

Thanks again Michael for sharing your knowledges and news. In this community, we should see more people like you from each department (Email, WAF, Network, etc...) so you can read forums, get in contact with people here and take notes of new ideas/bugs.

We are all here to share our point of view and to improve Sophos products. Take our criticisms are constructive and not disruptive!

Thank again! We are looking forward to testing v17!

Regards

After almost a year I would inform Sophos and suggest to them to do something with ADS blocking. Web Filtering is not blocking ADS, simple! Why Top management have to pay for something that simply does not work? This is the question I receiving with XG.

Dear Sophos's Guys, you have to improve it. We pay for Web Filtering and I had enough to receive complaints.

I guess I am not alone, here....

Hi Luk

As you might know, I have (when I´m bored or have too much time to waste) as kind of a "hobby" blocking away www advertisers and trackers - I also wrote in the past a "guide" to give people a base to start blocking www ballast away ==> Blocking Ads in UTM

However, besides of the differences in the used URL databases behind UTM and XG, both products are not designed to operate as adblockers as main function. There are some nice features built in to start to make websurfing faster and cleaner...

From my personal experience the XG URL database became quite good over the past 1-2 years. Many of the advertisers URLs or domains gets blocked out of the box using the "Advertisements" category. Maybe if I get some spare time, I might start once with a "tweaking guide" for the XG firewall or a "how to block advertisers, trackers and other WWW ballast" guide...

However, as base to start you might get best results with following rough steps:

- Use HTTPS scanning/MITM in the webproxy whenever possible (as this give more granular control for filtering content)

- Blocking, or at least set to "Warn" for the "Advertisements" Category already blocks lot of the unnecessary ballast away

- I also use a "URL Group" named "Additional_Advertisers_Trackers" containing some additional domains delivering ads or tracking users, which otherwise might be categorized as Content Delivery or General Business in the URL DB, and block them away too. Actually for my surf behaviour I added following domains, which might be a start for your own domain collection:

quantserve.com
spiceworks.com
outbrain.com
cxense.com
ensighten.com
chartbeat.com
visualwebsiteoptimizer.com
gigya.com, adtech.de
clicktale.net
krxd.net
mpstat.us
usabilia.com
supersonicads-a.akamaihd.net
unityads.unity3d.com
supersonicads.com
adcolony.com
external-frt3-1.xx.fbcdn.net
pixel.facebook.com
l.facebook.com
google-analytics.com

However keep in mind, some of those domains also might deliver some legitimate content which also gets blocked away, so use that "mini guide" at your own risk ;o)

Filtering that way the websurfing experience is already quite clean and fluid. I anyway still use "real" adblockers in browsers as uBlock in Firefox to further clean up websites, as adblockers got designed and optimized to get one single and specific task done right.

/Sascha

Hi Sascha, Luk,

I use the XG ad blocking feature and seem to block most ads, though not all and I have to add some exceptions because as Sascha advises some site use ad sites to deliver genuine product.

This was yesterday's little collection. I do note that firefox and safari block different ads after they get through the XG.

Ian

Thanks Sascha for your response but for the web filtering, I have enough...me and of course my customers. XG is deployed using HTTPS scan since v16 but ADS and several URL are not correctly categorized. I encourage my customers to submit false positive using Sophos website but this engine does not its job. I understand that you moved to your own engine, but remember that customers are the more important part here and if they are not happy, they move to something else.

Using custom url blocking is a workaround but you have to sit with the XG console, check logs, reports and submit so many false positive and fill the url group? We have time for that but not all day x 365 days.

Check rfcat_vk feebacks.

Sorry guys but XG web filter at the moment is unusable for its catching quality rate.

On UTM 9 we can block additional ads via application control. Why there isn't an similar option (app classification for web-ads and trackers) in the XG firewall?

I would like to block the Ads in the youtube videos, but from prior discussions on this subject, youtube figured out a way of bypassing ad blocking by incorporating them in the videos. So to block a youtube AD you will end up blocking youtube videos, maybe not a bad idea.

Ian