Thread Info
  • State Suggested Answer
  • Locked Locked
  • Replies 30 replies
  • Answers 2 answers
  • Subscribers 37 subscribers
  • Views 49676 views
  • Users 0 members are here
Options
Suggested
This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

Bad URLs Categorizations Error on Web

FormerMember
FormerMember

Hi Sophos

Customers of XG Firewalls are reporting a bad url clasification, for example, this sites are detected as Porn/Sexuality/Nudity in other Sophos products but the clasification in XG firewall is bad.

http://www.xvideos.com

http://www.felizporno.com/mas-vistos

https://www.bonitaporno.com/pornostar-famosas/

https://www.pornoperra.com/videos-porno/castellano/

http://www.pornhub.com

Support say me that "Send a URL Request" but the troubble is that these URLs are Porn.

Please check.

Regards



This thread was automatically locked due to age.
Parents
  • 0

    WEB filtering is a broken thing in XG.

    Ultra known web sites are wrongfully categorized.  This one among many others :

    If you can't categorize Intel properly, you have no business to do whatsoever in WEB filtering.  Before we decommissioned all Sophos products here, we would have similar results with ultra-known IBM (and many others) as well.  Another example: our Corporate bank Account categorized as "Job Search".  Users would knock at my door many times a day.  In the end I removed WEB filtering.  Also, users were able to access forbidden web sites indirectly.  By doing a Google search, and clicking on the resulting page.  It was not reliably repeatable however.  Symantec old WEB gateway, "Spywall", would do that too.  This why I checked Sophos' WEB appliance.  Symantec new WEB gateway, "Blue Coat", do not do that.

     

    PJR

  • 0 in reply to Big_Buck

    Forgot to mention ... XG WEB filtering is a moving target.  One day a WEB site could be ok.  The day after, it is not. Then the following day, it is ok ...

    PJR

  • 0 in reply to rfcat_vk

    Hi,

    this is very strange. i dont know why Sophos don't use their own categorization from other products?

    They seem way more accurate than the cyberoam classifications.

    UTM uses the mcafee database which is fine. It's really OK when Sophos is trying to push their own database. But why they don't use the database from the entpoint protection, instead of cyberoam? [:O]

  • 0 in reply to TheBalmasque

    Hi,

    Sophos are trying to move away from Mcafee because they have to pay for it. I have been a player in the UTM environment for many years, so have a bit of an understanding where they are going.

    Ian

  • 0 in reply to rfcat_vk

    Move away from McAfee is one thing.  But they were clearly not ready to do so.

    Now add to this XG and Intercept X.  Sophos ended swallowing a pill far too large for what they could actually handle.  They are sinking in a development boat.

    It will take years to stabilize.

    Meanwhile customers are paying for development more than support ...

    Paul Jr

  • 0 in reply to Big_Buck
    The person who started this thread is "Feliz Porno".

    If you open his profile, his Blog, his Twitter, his RSS feeds, and his Website are porn sites.
     
    He has posted only once.  He has not replied to anything in the thread that he started.
     
    All the sites that he posted are correctly categorized as Sexually Explicit in the XG.
     
    Although I have no specific proof, I believe the initial posted is a troller who is advertising or increasing linkages (SEO) to certain porn sites.
     
    I recently had the old "bad categorization" thread locked - which had a bunch of other similar posts by users who only did singular posts with links to porn sites.
     
    I recommend ignoring all posts containing links to porn sites, unless the person is a multiple poster who is clearly real.
     
    ----
     
    For the other people who are in this thread complaining about the categorization, I generally agree.  I have found the recategorization to be responsive (stuff I submitted has been changed when I check back a few days later) but the interface and feedback could be better.
     
    As I'm mentioned before in the past - some categorization of advertising can be problematic as it is difficult to determine the category of a 1x1 pixel.

    Although XG does not have the category override in exactly the same way as UTM, in practical terms you can do the same thing.  On the XG you can create a custom category and put your URLs in there.  Or you can create a URL Group and put them in there.  From that you can deal with them within your policy however you want.  Note that both of these are additive - it does not remove the original category.  Therefore after recategorizing you need to put in a new rule to match before whatever rule it used to match with the incorrect category.  An additional step, but it only needs to be done once.
     
    The sample that was shown - downloadcenter.intel.com is correctly categorized as a "Download Freeware and Shareware" site.  At least, there is a good argument for categorizing it as such.
     
     
     
  • 0 in reply to Michael Dunn

    M. Dunn

    Ok let's try to see it this way ...

    Downloading patches and updates is correctly categorized as "Freeware and Shareware" ?  This implies the firewall will block them, and these updates will not get installed on any desktops.  I hardly see how security gets improved in a mechanism like that ...  Because of this, either we have to disable that policy and allow-all "freeware and download".  Or at least IT managers faces the responsibility to disable and enable every time Intel post an update.  I do not have that much time to waste.

    Besides all other security products we are using - Symantec and Web Sense for example - do not pull IT manager in that extra unproductive management task waste.

    For now, Sophos struggle categorizing.

    Paul Jr 

  • 0 in reply to Big_Buck
    I don't want to get into a deep discussion on whether or not it is better to categorize that site as a Download site or Information Technology.
     
    My point is that it is easy to understand why it might be categorized as one versus the other.  Especially using automatic categorizers who crawl through the websites.
     
    However when I think about it, I doubt this was an error in the automatic categorizer.  When we categorize a website (intel.com, we generally have no reason to categorize subdomains differently.  Because at that point the subdomain is categorized (inherit from the parent) we don't run the automatic categorizer on it.  There is little reason we would look at changing the category in a subdomain - unless there was a specific complaint from a customer about it being wrong.  So my guess (and it is only a guess) is that Sophos now has two customers - one who in the past saw if as Information Technology and put in a request to recategorize it as Download, and another customer who sees it now as Download and says it should be Information Technology.
     
    Therefore the statement "If you can't categorize Intel properly, you have no business to do whatsoever in WEB filtering" is unfair - we are juggling multiple customer complaints here and there are perfectly valid arguments for both sides.
     
    In any case, it looks like that site has been set to Information Technology now.
  • 0 in reply to Michael Dunn

    I am generally satisfied with web categorization.  In the event I do run into false positives or whatnot, I have found that the categorizations are usually corrected within 24-48 hours after submitting them for recategorization.  I also do not seem to have the "Uncategorized" problem reported earlier in this thread, for instance yesterday 0.26% of my requests were categorized as "None."  I'm perfectly ok with that.  

    I'm also unclear why you would want your users downloading and installing their own updates from Intel but to each his own I guess.

  • 0 in reply to rfcat_vk

    Please note that this chart DOES NOT show that your users visited 22k distinct uncategorized websites.

    What this chart shows is that across your network, there were 22.37k requests to URLs that were not categorized. Those requests could well have been to a very small number of different sites.

    It also shows that the average size of the responses to those requests was a little over 1kb. 

    So this is just as likely to be a large number of small requests to a single site that was uncategorized - for example, an app that does many polling requests over time.

    Please note also that any web traffic that is subject to a Web Exception excluding it from policy checks may also appear as Uncategorized in the logs and reports. The product tends not to do category lookups for sites where it knows that the category will not be required for making policy decisions. This will include any Sophos endpoint products running on machines on your network that do various types of lookups for data updates, URL security lookups, management or policy updates and product version upgrades.

    You should be able to get a better picture of the actual number of uncategorized sites by going to Reports > Dashboards, scrolling down to the Web Categories chart, then click on the 'None' label in the table below. This will take you to a filtered report, which includes a table called 'Web Domains'. This will show you a list of the domains that make up the Uncategorized block and how many Hits (HTTP Requests/HTTPS connections) to each were recorded.

    I agree we may have a problem, but I think that the problem here is more likely to do with how the data is recorded and presented in reports. If your further investigations suggest that there really are thousands of different domains being uncategorized, then we will look into it further.

  • 0 in reply to Michael Dunn

    , 

    in XG we are not able to override URL and put them in another Category. This is very useful for Reporting.

    https://ideas.sophos.com/forums/330219-xg-firewall/suggestions/33264490-web-category-and-reputation-override-like-utm

    XG web filtering catching level, compared to UTM, is still far away. I hope you will improve it quickly as for the moment we spend time to submit URL to Sophos Website.

    Regards

  • 0 in reply to RichBaldry

    Hi Rich,

    you are correct, the sites are all Sophos which have default exceptions in web.

    The above screenshot is from today's activity from one device.

    Ian

Reply Children
No Data