Web filtering: differences between Web policy of None and Web policy of Allow All

Question

I see differences in whether web requests work depending on whether a Web policy of None is used or a Web policy of Allow All is used. So there must be extra things that happen when there is a Web policy of Allow All. Can someone help explain what those things are? 
 The scenario I have is a specific mobile app not working correctly. By trial and error, I have been able to isolate a simple change in a firewall rule that makes the difference between whether the app works as expected or not. 
 I have a firewall rule for LAN to WAN traffic where the source IP address is for the mobile device with the app. The rule is for any destination network or service. When the Web policy for the rule is set to None, the app works. When the Web policy for the rule is set to Allow All, the app does not work. None of the other checkboxes under Web filtering are checked. 
 Logging is turned on for the rule. But when I use the log viewer I cannot find anything being blocked. I have checked the Firewall, Web filter and SSL/TLS inspection rules. 
 Are there other logs I should check? Is there other configuration that comes into effect when the Allow All Web policy is applied?

Vivek Jagad · Answer

Hey David Hay ,

By logs, I meant the logs under the /log/awarrenhttp_access.log
For instance if you set it to none, you'll not see website browsing logs under the webfilter as well as /log/awarrenhttp for the site you browsed, but if I keep allow all and browse to facebook.com I'll be able to capture the logs [as it goes through the proxy] as seen under the /log/awarrenhttp.log
for e.g -
1668686403.140310167 [ 7515/0x7f323aae7400] fwid=5 fwflag="VN" iap=1 aap=0 conn_id=1797583488 id="0001" name="http access" action="pass" method="CONNECT" srcip="192.168.97.104" dstip="31.13.79.35" user="administrator@sophos.creed" statuscode=200 cached=0 trxlen=581 rxlen=2411 url="">https://www.facebook.com/" referer="" type="" upload_file_name="" upload_file_type="" download_file_name="" download_file_type="" authtime=0 dnstime=2 cattime=141509 avscantime=0 fullreqtime=6216273 ua="" activity="" av_transaction_id="" categoryname="Social Networking" category="67" app_id=0 app_name="None" app_cat="None" exceptions=""

If there is a site works fine, with the option none, meaning there is no restrictions what-so-ever, working as a normal direct ISP connection to a laptop. With the allow all, a web proxy/DPI comes into the picture depending upon the option you enable it or not "Use web proxy instead of DPI engine" and with that "allow all" option if a site is not working meaning FW proxy/DPI is intervening the traffic and needs to be diagnosed further to understand the root cause !!

Long Story Short - None means it will not be processed by proxy/DPI whereas Allow All means it will be go through proxy/DPI.

Thanks & Regards,
_______________________________________________________________

Vivek Jagad | Team Lead, Global Support & Services

Sophos Community | Product Documentation | Sophos Techvids | SMS
If a post solves your question please use the 'Verify Answer' button.

Michael Dunn · Answer

If web policy is None and Malware scanning is unselected, you cannot turn on proxy mode (WebAdmin will not allow you). Basically, if you don't want the XG to do anything then it forces it into DPI mode. If for some reason you want it to go through proxy mode then you'll need to set Allow All. In DPI mode, there are several reasons that the XG may want to interpret the HTTP. Any web policy except None will cause DPI to look at HTTP. Malware scanning will cause DPI to look at HTTP. ATP will cause DPI to look at HTTP. When DPI looks at HTTP it will enforce the HTTP specification (traffic must conform to what it cab process) and it will log (in Web Filter logs, which will also power reports). But if there is no reason to look at the HTTP at all then it won't. So if web policy is None (and no malware or ATP) then web-in-snort DPI will not try to interpret the HTTP at all, will not enforce HTTP spec, and will not log web traffic. I am not positive, but that might then enable FastPath for that traffic.