Youtube and ADS - Sometimes they are not blocked

Hi All,

Configure a Web Policy to block "Advertisement" category and enable HTTP & HTTPS scanning. Configure a DROP rule for UDP 443 service and make sure to place it on TOP. YouTube ADS are successfully blocked and during instances of incorrect web categorization, take the steps as suggested by Aditya Patel in his previous post. Refer the screenshots below:

Thanks

Sachin,

I am already using this configuration (apart the UDP 443 blocking rule) which is catched by the default rule at the bottom.

I will check the config and post back again.

Thanks

I've raised the issue with data quality of advertising internally.  I don't promise anything, but the comments in this thread have been forwarded.  The good news is that web categorization data improvement is not on the release cycle and has a different set of people working on it.

I don't know too much about how the Sophos categorization team works, but I believe most of the work is using automated tools and web crawlers, and that most of the work will focus on categorizing new websites, not on re-evaluating current ones.  For changes to existing categories it works more on reports of poor categorization.  There are millions of websites currently categorized - which ones need to be updated?  It is impractical to try and re-analyze all of them.  They need to rely on complaints.  In addition, some category changes (like to/from Business and malicious) are much more important than others (like to/from Business and Information Technology).  Things like Ads and Streaming Media fall in between those priorities.

The other complication is that a domain may be used for multiple things.  For example, you found that steaming media came from cnn.stream1.fyre.co.  But maybe also the stock ticker comes from there.  Or maybe even html news articles pull some content from that domain.  A categorizer may need to be careful to use the lowest common denominator.  Otherwise you may find that categorizing that domain as streaming media and then blocking it, several parts of cnn.com stop working.  Which then people would complain about -- why is Sophos incorrectly classifying that url as streaming media.  :)  It is a pretend example, but it goes to show that the problem is more difficult than it may first appear.

I agree that that category names, etc, are maybe a bit confusing.  Partly that is just a learning experience for anyone switching from one product to another  Aside from improving documentation, I don't know if we can do anything about that.

For those of you with a long memory.  Do you recall that in UTM the original Astaro it used a categorization service known as CFFS.  In 9.2 we switched it to SXL.  It used the same McAfee database but had several improvements, including the addition of Sophos Labs data source for malicious urls.  We are doing the same thing for XG.  Currently XG uses a service called WINGc.  In v17 we will be switching this to SXL.  This will give us the better data source for malicious URLs.  The evolution of XG is following the same evolution as UTM.

Thanks Michael,

yours is a great news. I think that you should add multiple and different web filtering engine to XG/UTM, specially for customers that are using Sophos Heartbeat. A dual defense on Web Filtering nowadays is a must and if XG and Endpoint use the same engine is not so safe.

As NGFW, XG should "search" web traffic from more than one DB. Think about to take an agreement with other products. The same applies to Sophos Web Appliance (SWA).

Regards

Hi Michael, as always thanks for the concise thoughtful response. You are right about the URL that I mentioned in my post and I am not saying that in certain cases XG won't classify something better. I also realize that change is hard for some of us folks[:$]

Michael Dunn said:

For those of you with a long memory.  Do you recall that in UTM the original Astaro it used a categorization service known as CFFS.  In 9.2 we switched it to SXL.  It used the same McAfee database but had several improvements, including the addition of Sophos Labs data source for malicious urls.  We are doing the same thing for XG.  Currently XG uses a service called WINGc.  In v17 we will be switching this to SXL.  This will give us the better data source for malicious URLs.  The evolution of XG is following the same evolution as UTM

Thanks again for taking the time and raising the issue with the correct team.

The main change from WINGc to SXL is we are changing the cloud servers and the method of communicating with them.  The data backing them is 99% the same.
The difference is that WINGc (from Cyberroam days) does not contain Sophos Labs security data.
Rather than adding the Sophos Labs data to WINGc, we have added the Cyberoam-now-Sophos categorization data to SXL, and are moving XG to SXL which now has both.
Sophos Labs data is based on emerging threats and is managed more like antivirus data.  Security data is often shared between companies because security is more important than proprietary.
It is a small drop in the total categorization data, yet it is an important one.

Endpoints are complicated because where they get categorization data can change depending on the product they are associated with.  There are now also several different endpoint products.

I'm not in the newsgroups daily, my involvement ebbs and flows with my other workload.  I've got a fine line to walk about what I can/should talk about and not.  I am also sometimes limited in what I can actually do about issues, Support and Partner channels sometimes being the better options.  That being said, I think teasers and insights like this are good.  I'm running SXL on my test v17 XG box right now.  :)

Thanks again Michael for sharing your knowledges and news. In this community, we should see more people like you from each department (Email, WAF, Network, etc...) so you can read forums, get in contact with people here and take notes of new ideas/bugs.

We are all here to share our point of view and to improve Sophos products. Take our criticisms are constructive and not disruptive!

Thank again! We are looking forward to testing v17!

Regards

After almost a year I would inform Sophos and suggest to them to do something with ADS blocking. Web Filtering is not blocking ADS, simple! Why Top management have to pay for something that simply does not work? This is the question I receiving with XG.

Dear Sophos's Guys, you have to improve it. We pay for Web Filtering and I had enough to receive complaints.

I guess I am not alone, here....

Hi Luk

As you might know, I have (when I´m bored or have too much time to waste) as kind of a "hobby" blocking away www advertisers and trackers - I also wrote in the past a "guide" to give people a base to start blocking www ballast away ==> Blocking Ads in UTM

However, besides of the differences in the used URL databases behind UTM and XG, both products are not designed to operate as adblockers as main function. There are some nice features built in to start to make websurfing faster and cleaner...

From my personal experience the XG URL database became quite good over the past 1-2 years. Many of the advertisers URLs or domains gets blocked out of the box using the "Advertisements" category. Maybe if I get some spare time, I might start once with a "tweaking guide" for the XG firewall or a "how to block advertisers, trackers and other WWW ballast" guide...

However, as base to start you might get best results with following rough steps:

- Use HTTPS scanning/MITM in the webproxy whenever possible (as this give more granular control for filtering content)

- Blocking, or at least set to "Warn" for the "Advertisements" Category already blocks lot of the unnecessary ballast away

- I also use a "URL Group" named "Additional_Advertisers_Trackers" containing some additional domains delivering ads or tracking users, which otherwise might be categorized as Content Delivery or General Business in the URL DB, and block them away too. Actually for my surf behaviour I added following domains, which might be a start for your own domain collection:

quantserve.com
spiceworks.com
outbrain.com
cxense.com
ensighten.com
chartbeat.com
visualwebsiteoptimizer.com
gigya.com, adtech.de
clicktale.net
krxd.net
mpstat.us
usabilia.com
supersonicads-a.akamaihd.net
unityads.unity3d.com
supersonicads.com
adcolony.com
external-frt3-1.xx.fbcdn.net
pixel.facebook.com
l.facebook.com
google-analytics.com

However keep in mind, some of those domains also might deliver some legitimate content which also gets blocked away, so use that "mini guide" at your own risk ;o)

Filtering that way the websurfing experience is already quite clean and fluid. I anyway still use "real" adblockers in browsers as uBlock in Firefox to further clean up websites, as adblockers got designed and optimized to get one single and specific task done right.

/Sascha

Hi Sascha, Luk,

I use the XG ad blocking feature and seem to block most ads, though not all and I have to add some exceptions because as Sascha advises some site use ad sites to deliver genuine product.

This was yesterday's little collection. I do note that firefox and safari block different ads after they get through the XG.

Ian

Thanks Sascha for your response but for the web filtering, I have enough...me and of course my customers. XG is deployed using HTTPS scan since v16 but ADS and several URL are not correctly categorized. I encourage my customers to submit false positive using Sophos website but this engine does not its job. I understand that you moved to your own engine, but remember that customers are the more important part here and if they are not happy, they move to something else.

Using custom url blocking is a workaround but you have to sit with the XG console, check logs, reports and submit so many false positive and fill the url group? We have time for that but not all day x 365 days.

Check rfcat_vk feebacks.

Sorry guys but XG web filter at the moment is unusable for its catching quality rate.

On UTM 9 we can block additional ads via application control. Why there isn't an similar option (app classification for web-ads and trackers) in the XG firewall?

On UTM 9 we can block additional ads via application control. Why there isn't an similar option (app classification for web-ads and trackers) in the XG firewall?