Web Policy and Filtering Not Working at All

I found on my XG230 that the only Firewall rule it would use was the first (top) and then allow everything else under it.

Seems mental as usually a Firewall would do the filter thing and pass to the next rule.

I ended up making a Web Policy and adding all the categories into it and then adding that to the Firewall Rule and it works fine.

I was told by the Sophos vendor this was by design and not a bug - not quite sure I believe him though.

I don't know if he was in error, expressed himself poorly, or you misunderstood, but firewalls universally work by principal of first match.  If traffic matches a rule, yes, it stops further comparison of rules.  Therefore, you need to have the most specific rule first for any traffic that could match more than one rule.  For instance, if you want to block host A from HTTP, you'd first have a rule to block it before any a rule allowing all others to HTTP.  If you had the rules reversed where the "allow all" to HTTP first it would stop there and allow Host A's traffic, and not get to the rule to block that specific host.  I hope that makes sense.

Thanks David,

I do understand however the XG isn't working this way.

For example, I had rule 1 to block Social Media sites (web policy) and applied to a specific User Group, then Rule 2 was a Web Blacklist of sites I dont want anyone going to, Rule 3 was to block EXE downloads and so on.

If a user in that policy tried to go to Facebook it would fail (Rule1) however could download an EXE file (Rule3)

So I have one FW rule with seperate Web Policy and Application Controls added.

Firewall Rule 1 had a Web Policy.  That Web Policy was "Block Social Media and a Default Action of Allow".  The rule was matched based on the user and it correctly followed the logic of the Default Action, which is to allow.  It never hits the second Firewall Rule.

In v16 and later, as much as possible you should put all web decisions into a single Web Policy.  Within that policy you can do the Rule 1, Rule 2, Rule 3 that you describe.

Firewall rules basically have two parts:  Match this traffic THEN apply this action.  Firewall rules flow from one to the other on the "match this traffic" part until a Firewall Rule is matched.  It then follows all the logic in the "apply this action".  It does not then, after that, go the the next firewall rule to see if it matches that as well.  For a given packet, only one firewall rule is applied.

Web Policy rules work differently.  It also has a "match this traffic then apply this action".  However for those rules, traffic is always matched against all rules that apply to it.  For the Web Policy a "Block" will Block and "Allow" really means "I'm not going to block, continue evaluating the next rule".

EDIT: Update/Clarification.  Web Policy also applies the first rule that matches, whether it is allow or block.  Allow does mean Allow, not "continue processing".  This makes a difference if Rule 1 is allow Document Files and Rule 2 is block Adult sites.  If someone downloads a pdf from an adult site, it will be allowed.

Firewall Rule 1 had a Web Policy.  That Web Policy was "Block Social Media and a Default Action of Allow".  The rule was matched based on the user and it correctly followed the logic of the Default Action, which is to allow.  It never hits the second Firewall Rule.

In v16 and later, as much as possible you should put all web decisions into a single Web Policy.  Within that policy you can do the Rule 1, Rule 2, Rule 3 that you describe.

Firewall rules basically have two parts:  Match this traffic THEN apply this action.  Firewall rules flow from one to the other on the "match this traffic" part until a Firewall Rule is matched.  It then follows all the logic in the "apply this action".  It does not then, after that, go the the next firewall rule to see if it matches that as well.  For a given packet, only one firewall rule is applied.

Web Policy rules work differently.  It also has a "match this traffic then apply this action".  However for those rules, traffic is always matched against all rules that apply to it.  For the Web Policy a "Block" will Block and "Allow" really means "I'm not going to block, continue evaluating the next rule".

EDIT: Update/Clarification.  Web Policy also applies the first rule that matches, whether it is allow or block.  Allow does mean Allow, not "continue processing".  This makes a difference if Rule 1 is allow Document Files and Rule 2 is block Adult sites.  If someone downloads a pdf from an adult site, it will be allowed.

Michael Dunn said:

Web Policy rules work differently.  It also has a "match this traffic then apply this action".  However for those rules, traffic is always matched against all rules that apply to it.  For the Web Policy a "Block" will Block and "Allow" really means "I'm not going to block, continue evaluating the next rule".

Policy1 Allow lan to wan webfilter ->allow general business, block advertisements.

Policy2 Allow lan to wan webfilter-> allow advertsements, block everything else

Does that mean that advertisements and general business categories will be allowed?

Also, is the webfilter logic also applicable to application control?

Billybob said:

Policy1 Allow lan to wan webfilter ->allow general business, block advertisements.

Policy2 Allow lan to wan webfilter-> allow advertsements, block everything else

Does that mean that advertisements and general business categories will be allowed?

Also, is the webfilter logic also applicable to application control?

Thanks for the reply, that clarifies things a lot. With level 8, I can assign specific rules to specific people so I usually don't have to worry about policy compliance. The firewall rules incorporated with webfiltering, application filtering, qos, and IPS with further granular control of QoS is one my favorite features in XG. You can write a single rule for a lot of stuff that needed multiple rules before but on the other hand this makes things complicated and gives unexpected results if the admin is not paying attention.

After reading through your explanation I remembered the allow/deny rule setting in UTM9 web filtering and everything made a lot more sense. The main difference is that default allow doesn't carry over to the next firewall rule[:D]

I'm coming from the Web team and therefor am biased towards the Web Policy.

In v15 we carried along with the simpler implementation of making the user-aware firewall make all the user policy decisions.

In v16 we found that the firewall rules were just not powerful enough to do everything we wanted to, and that the real work in web policy decisions must be done in the web proxy.  So we did a lot of work making Web Policies and User Activities.  These are closer to the way that the UTM works.  Now you have a choice.

* You could use the XG fully like the UTM - where the UTM Web Profile is equivalent to an XG Firewall Rule (with no match users and a web policy selected).

* You could also use the XG with the user-aware firewall and do all your user policy decisions there.  The thing is that if you do make all your user policies decisions in the firewall you must make sure that the web policy you select fully defines their permissions.

From a pure policy perspective, doing it within Web is more powerful as you now have user constraints and time constraints defined in the same place.

Here is a basic example of the difference:

Firewall Rule 1 - Applies to Mike - Use Web Policy 1 - Block executable files, Allow everything else
Firewall Rule 2 - Applies to Mike - Use Web Policy 2 - Block adult sites, Allow everything else

-OR-

Firewall Rule 1 - Don't match users - Use Web Policy 3 - Applies to Mike - Block executable files
                                                       - Applies to Mike - Block adult sites

In the first case, Mike will not be blocked from Adult sites.

So if you wanted to put user decisions in the Firewall you should do the following.

Firewall Rule 1 - Applies to Mike - Use Web Policy 3 - Applies to All - Block executable files
                                                     - Applies to All - Block adult sites

If it helps, you can think of it a different way.  An incoming packet can only go to one Web Policy.  So you cannot define two Web Policies and have them both apply to that connection.  You need to have one Web Policy.  Then you have the freedom of doing user selection within the Web Policy or the Firewall Rule.

Thanks Michael for perfectly explaining things. This is the kind of interaction that I miss from astaro days. The devs/mods didn't just show you a workaround but the logic and thinking that went behind a decision to do things a certain way. Most people get caught up in the basic gui differences between UTM9 and XG and I don't blame them but by doing so they don't give the true gems like firewall policies a fair chance. I was not a fan of v15 but gave v16 a serious chance after reading 's praises of XG. Although I still miss certain things from UTM9, after using XG v16 exclusively for a few months, I realized that UTM9 is indeed showing its age.

Although you mostly deal with the webfiltering aspect, I am hoping you guys can inspire other teams with your forward thinking into other areas of XG. If we keep getting the XG firmwares at the pace we have been getting, I am very optimistic and can't wait to test what v17 will bring.

Thanks again and happy holidays

Regards
Bill

v16.5 will bring in Sandstorm for web and email.

v17.0 I can't give you any hints.  :)

Michael Dunn said:

If it helps, you can think of it a different way.  An incoming packet can only go to one Web Policy.  So you cannot define two Web Policies and have them both apply to that connection.  You need to have one Web Policy.  Then you have the freedom of doing user selection within the Web Policy or the Firewall Rule.

Sorry to bump such an old thread but this thread has some great info on how policies work. If I'm understanding this correctly, it really boils down to the statement quoted above but I'm still a bit confused as to the purpose behind "Allow All" or "None" and why policies are applied to individual firewall rules. Here's an example I created to illustrate what I'm confused about:

I created two firewall rules that apply to the same computer, we'll call it Firewall Rule 1 and Firewall Rule 2.

Firewall Rule 1 - Applies to Computer X - Allows all traffic - Web Policy set to 'None'

Firewall Rule 2- Applies to Computer X (same computer) - Allows all traffic - Web Policy set to 'Block Shopping Websites'

In this example, Firewall Rule 1 is applied when accessing the internet, therefore Firewall Rule 2 will never be applied and I can still access shopping related websites. However, I thought the purpose of setting a policy to 'None' is so that it can still be "eligible" for a web policy to apply, but how is that possible if it will never be assessed against another firewall rule?

There are two other factors you need to consider - the "Services" that the rule applies (which is really the port numbers), and the Malware scan HTTP and scan HTTPS setting.

Think of it as two sections - an "The rule applies to traffic matching criteria" section and an "enforcement" section.

Assuming that Firewall Rule 1 applies to the HTTP and HTTPS service (or TCP or Any), then Rule 1 would apply to HTTP traffic.

The Firewall looks at both the Malware setting and the Web Policy.  Assuming that Malware scan was on for HTTP and Web Policy was None, then it would go through the web proxy in order to do antivirus scanning but the web proxy would not enforce any policy (eg around website categories or filetypes).  If Malware Scan was off, then it would not go though the web proxy at all and would just be allowed through the firewall.

Once the matching part is done to select a firewall rule, then the firewall decides "Does this need to be sent to the web proxy".  If the malware scan is on, then it is sent to proxy.  If the Web Policy is not "None" then it is sent to Proxy.  If you want to create a rule that passes the traffic with no interference from the proxy, then you turn malware scan off and Web Policy None.

Web Policy "None" (which is part of enforcement) does not cause it to fall through to another firewall rule (which would be a part of traffic matching).  

Pretend for example that None did allow a fall through from Rule 1 to Rule 2.  Which rule's Malware setting would apply?  Would rule's Application policy?

If you do want to create a "Firewall Rule 1 - Applies to Computer X - Allows all traffic - Web Policy set to 'None'"  it needs to not apply to the service HTTP/HTTPS (eg not really Allows all traffic).  Or if you really need it to be the Service Any then you need a higher priority rule for HTTP/HTTPS.

Michael Dunn, thanks for taking the time to respond. I think I'm understanding everything you're saying, but it seems to further support the fact that there seems to be no practical purpose for setting a Web Policy to "None" versus "Allow All". The only difference it would make (assuming Malware scanning is off) is the fact that traffic would pass through the web proxy if it's set to "Allow All" but nothing would be blocked... meaning, there is no reason to NOT set it to "None" - why have traffic pass through the web proxy for no reason?

If I'm understanding you correctly, Sophos XG determines which Firewall Rule traffic will apply to based on "Source Zone", "Source Networks and Devices", "Destination Zone", "Destination Networks" and "Services". If traffic matches based on these variables, then whatever falls under that Firewall Rule will apply, to include the "enforcement" section as you mentioned (malware scanning, policies, etc.). If the traffic does NOT match based on these variables, then nothing in that Firewall Rule will apply.

So again, I'm still not understanding why "Allow All" versus "None" would ever be used.

As a side note, would this also mean you need to have the appropriate "Services" selected for the "enforcement" you're trying to utilize? For example, if I want to Scan HTTP for malware or utilize a web policy, "Services" would have to be set to either "Any" or "HTTP". I'm assuming the answer to this is yes (just seems to make sense logically as a different service/port would be a different connection even if it's from the same IP address).

Thanks again for your time. I'm just trying to understand how Sophos XG works and not actually having any specific issues with my setup so I hope I'm not wasting your time!