Yet again: Download Throttling

I'm honestly still not sure on this. Hopefully Sophos can weigh in on it specifically.

My guess is, that your resulting rules would be correct, as long as you think of them in an egress vs. ingress manner.

Which, if I understood you right Bob, is the overall way you are looking at them, so yes, your rules would be correct.

Again, I had to move 2 of my QoS rules to the download throttling tab to correct them. I was still thinking in an ALTQ manner, after migrating my config to SUTM from a FreeBSD based system.

Hi bob, keith is right on all counts as far as the mechanism and the engineering aspect. Only thing I would change on your earlier post is that as keith pointed out, we are mostly playing with egress (outgoing packets). There is a lot more flexibility where we can assign different priorities, delay the packets and apply bandwidth limits etc on the queues when the traffic is leaving the firewall. With ingress policing, you really don't have much choice and the only option is dropping traffic to then let tcp/ip builtin controls to slow that traffic down.

So yes there are actual differences and throttling doesn't just drop packets in all cases. Also, the throttling is done on kernel level so ingress/egress shouldn't have any influence on proxies or other daemons as that should technically come after kernel has already sorted through all the traffic. A proxy cache may give you skewed results by caching/scanning some of the traffic and then releasing to the client in one chunk making it seem like you have a higher bandwidth than what is really available, but the actual throttling has already been done.

Of course, I have not written any iptables rules since i found shorewall and definitely don't even remember the syntax since jumping on astaro[:$] so my knowledge is outdated and limited.

EDIT: I know the confusion that you are describing in some of the earlier versions where the proxy seemed like it was doing its own thing while QoS was not having any effect. I think that has been fixed since later v8/ early v9 releases. The reason for that wasn't the proxy but they way rules were written for QoS. So for example a rule like 

QoS any traffic coming from internet with port 80 to ANY LAN got bypassed when proxies dnatted that traffic. That was not the shortcoming of QoS but the implementation and the complexity of applying that rule correctly for proxies with different configurations and profiles.

I will chip in here:

Download throttling does work even with "web filtering on"

The trick is to select "Application selector" and NOT "traffic selector" under Traffic selectors.

I have tried this successfully this morning using the following:

DOWNLOADS:
Traffic selector = application (http)
Download throttling = Bound to WAN > select above traffic selector and enter limit in kb/s

UPLOADS:
Same as above but you need to use Bandwidth Pools and ensure "Upper limit" is selected with appropriate kb/s

Doing both of the above limited my download and upload using http with web filtering enabled.

Just did a quick sanity check / test on SUTM current [9.413-4]:

Limiting upload bandwidth on an internal interface with a download throttling rule (ingress), limits the upload rate from the LAN host to the web proxy. The web proxy then uploads the memory cached (I have disk caching disabled) data at full interface speed. This is using service based traffic selectors.

Using the exact same rule(s), uploading a singe stream of data larger than the maximum antivirus scan size, works as expected. The traffic is forwarded through SUTM at the desired limited rate.

Without testing further, I'm fairly certain we can assume that this would also be the case when limiting download traffic, with bandwidth pools (egress), on an internal interface, using a service based traffic selector, for all data streams coming in under the maximum antivirus scan size. They would be downloaded by the web proxy at full speed, into memory, then sent from the web proxy to the LAN host at the limited rate.

QoS is indeed still "doing its own thing". I understand why, I just wanted to make sure this current behavior was noted, hopefully it will add to Bob's sanity, and the sanity of others reading this ;-)

It seems, the strait forward solution, as Louis pointed out, is to use layer 7 application selectors in your traffic selector definitions, rather than layer 4 service (mostly port based) selectors. This way, your limiting rules won't care what source / destination port the traffic is using, they'll only care what layer 7 type the traffic is, and http(s) will match, regardless of how the web proxy modifies said traffic.

As I understand it, however, layer 7 based rules (anywhere in SUTM) are far more computationally expensive, than layer 4 based rules. And, you have to think of them in terms of all traffic that the rule(s) would be evaluated against, not just the traffic they would match. So, this may not be a great solution on hardware with lower capacity CPU's.

Bill and Louis, consider https://community.sophos.com/products/unified-threat-management/f/network-protection-firewall-nat-qos-ips/73340/can-not-apply-qos-when-enable-web-filtering/331211#331211.

Cheers - Bob