Sophos XG 18 MR3 DPI slow download

This is just the way it is.  Also you get the bonus of it breaking random sites with no idea why or what to do about it.

are there any new ideas on this topic?

I don't think that this is not happening to lot's of other people if this would be a bug?

My issues started with MR4 fwiw

Same here,

XG125 fresh install with v18 MR4.

Download with ips + dpi enabled/running 2.5MB/s

Download with stopped dpi + ips service 15MB/s

Running on an 1GBit / 50 MBit Cable line

Direct download via Cable Modem from same site 80 MB/s

Mine too.

For me it is always the same story; I usually enable it following an MR release which promises DPI engine bug fixes and tweaks, then I wait for the calls and e-mails telling me the internet is acting strange or websites don't work.  It usually takes about 12 hours before the problems begin to manifest themselves, and its usually one of two things; either a site won't load (or a particular page within a site won't load), or downloads just don't work, it will download a little bit of a file and then just stop.  Of course the wonderful XG logs all report no problems at all with decryption, nothing being blocked, etc., but the moment you switch the firewall rule back to web proxy, all these mysterious problems vanish.  So there is something wrong with the DPI engine at least in my setup (and its fairly vanilla) and the logging is just too poor to capture it or display anything useful to the administrator as to what is happening or why.  

The only indication of something wrong I ever get in the logs is mounting numbers of "FLOW_TIMEOUT[5] errors.  To my knowledge Sophos has never publically disclosed what this error actually means and honestly I see these errors on sites that aren't having problems and don't see them on sites that do, so I am unsure if it is just a red herring.

This is a very unscientific statement but based on my observations of the problems I have it almost seems to me like the connections get hung or stuck and it results in sites never loading, downloads never downloading, etc.  When I first enable DPI it works and then slowly over the next hours it becomes worse and worse to where it becomes noticeable to the average user out there.

Its a real shame because I want to use it, on paper it sounds wonderful.

Did you every test if this also accours if you have Linux are MacOSX Clients?

For me the issues are there just for my windows machines.

BTW. This also happens if I start a virtual Linux machine on my Windows 10 Desktop. But starting Linux native on the same system with DPI enable there are no issues.

I went through the pcaps with Wireshark for hours without finding the issue :-(

We only have Windows clients in operation. I'm going to test Linux and try it.

@Bill Roland  It is exactly the same with us

Hi,

it happens on MAC clients, MBP, iPad, iPhones.

Mail - iMap/s and smtp/s fail with no error messages.

Ian

I have to admit, that I just testet DPI for HTTP(s) and as this was causing issues I did  not extend the coverage, so I can't talk about imap and smtp.

My current solution is just not to use DPI which is a petty because I would like to make use of it...

Just to confirm, if you use DPI, can you extend the speed of your Line with multiple sessions? For example Ookla uses different downloads at the same time. https://www.speedtest.net/ You can run multiple or single tests. 

Whats the performance with DPI on your appliances? 

Just to confirm, if you use DPI, can you extend the speed of your Line with multiple sessions? For example Ookla uses different downloads at the same time. https://www.speedtest.net/ You can run multiple or single tests. 

Whats the performance with DPI on your appliances? 

right, this works. If i start several downloads I can max out my connection

The reason for the speed difference is because the underlying software on Sophos XG  - who does the DPI is heavily single-core, if you start a download with a single connection the DPI will only be able to use a single core to do almost everything, from the decryption/encryption to IPS and so on.

Meanwhile with multiple connections the DPI Engine is able to share the load with all available cores of your appliance.

The day Sophos updates Snort to 3.1, this issue will be (probably) solved.

Might be, but in my case my cores are not maxed out, there is lots of performance left on every single core and when using Linux instead of windows DPI works at full speed.

just to add, I tested this with a single client connected to a 230 Appliance which also was a fresh install with 18.4.

I just got a troughput between 1,6 to 2,5Mbit/s with disabled DPI I get 12Mbit/s and as you can imagine one client on a 230 should not max out the machine.

These are interesting facts:

- Linux and Windows differ in behaviour
- Multiple parallel connections max out the connection
- CPU is not fully loaded (I'd expect that a single connection will always go on a single thread/core). In bigger installation this should not be an issue as there will be plenty of connections.


There are still other parameters that might have an impact:

- fastpath on/off
- avanced-firewall tcp settings (https://docs.sophos.com/nsg/sophos-firewall/18.0/Help/en-us/webhelp/cli/PDF/sfos_cliguide.pdf)
- SSL/TLS decryption

If the proxy is on there are probably two connections. One going from the client to the sophos and one from sophos to the website.

When DPI is used I'd assume that the connection is more "direct". As the behavior of linux and windows is different with respect to throughput I'd think that there might be some "live translation" between client <-> sophos and sophos <-> website which fits in case of a linux client and does not fit in case of newer Windows versions.  Analysing such a connection low level in wireshark might help to find the reason for this different behaviour (window size, window scaling, selective acknoledgement, mtu & mms, fragmentation, ...).

I have seen some bad throughput of Windows Servers through WAN lines with high latency (SMB - never tested with HTTP(S)) so this might be an issue of Sophos firewall OR Windows Server. Multiple connections also helped in this case to increase throughput off the underlying connection.

Besides some basic testing (and probably using the old proxy afterwards) I won't participate in this. Simply not my task to invest time here. As a lot of people seem to be able to reproduce this it is really surprising that this is lingering around in this forum for months (or even longer) and still seems not to be deeply analysed, troubleshooted and resoved by Sophos.

I did some more tesing:

direct download https://speed.hetnert.de ISO 1 GB

Win 7 Chrome Cable modem drect 70MB/s

Win 7 Chrome xg125 Web Proxy  32 MB/s

Win 7 Chrome xg125 dpi 2,5 MB/s (i read about snort being single core but this is really slow maybe the xg125 is at max cpu?)

Then i sat up an Ubuntu 20 VM and after fiddling with disabling ipv6 (no cert warning or appliance cert usage):
Firefox dpi 10 MB/s (I guess firefox uses several sessions for download?)

Firefox Webproxy 23 MB/s

If i have some more time i will test multiple sessions

Gunter Kaufmann thanks for testing this at your end.

Very strange that everyone seems to get different results while doing the tests.

If Firefox would use several sessions for a download this should also give good results on Windows 10 which it does not do at my end.

Currently I have no clue what I could test or change on top of the things I already did. Perhaps it is a bug or an issue I can't resolve, but as long as there is no feedback from Sophos I will stay at the legacy proxy configuration which works very well.

I recently updated my Home UTM to Sophos XG Home firewall, I had a few fun and games with the UEFI boot problem but when I had circumnavigated around that I was pleased with the look and feel of the new interface (despite there not being a SET STATIC option from the lease list in DHCP).  All appeared fine as I played around with the features and settings for a few days until I realised my download speeds via a windows client seemed severely limited from what I had before, I have a 200Mbps download link but via a windows client this was reported as 9Mbps, if I try the same speed test from a MAC computer on the same network it was revealing a 199Mbps download speed.  I tried switching of various features and disabling dpi and using traditional web proxy but could not get download speeds to match the MAC or what I have in reality, the actual system was not heavily loaded.  

I have now reverted back to my old UTM restore and download speeds are as they should be with 40GB downloads finishing in minutes again instead of the XG network predicted 10 hours.

This does appear to be an issue with the new XG which I am supposed to be rolling out in my company in a few weeks but am now hesitant if this issue carries over to the business environment.   

Could an engineer from Sophos confirm this is a known issue and when there is likely to be a fix or if anyone else has found a workaround could they let me know.

I took some time looking into this concept and found some interesting points. 

Of course IPS, App control and web filtering will cause some decrease of performance. 

I could found some limitation of single connection downloads (single stream). Using a speedtest, everything looks normal (speedtest.net). Using test downloads, which offers a download file, seems to point to this issue. It will slowly increase over time. 

So if i download for example a 1 GB test file (https://speed.hetzner.de/) which are encrypted, i can start with a slower speed but it increase over time to a acceptable speed. 

Also by starting multiple connection from this test page, i can quickly match the overall acceptable speed. 

It is important to understand about the DPI: The client actually communicate with the server. There is no redirect to any proxy what so ever (compared to proxy, which in direct/transparent, will redirect it to the proxy and the proxy build up a new connection).

Just to confirm: If you see different values on different OS, it could lead to a different handling of the flow state between the server/client. 

Another question, do you see any kind of increase over the time of a bigger download? 

And please: For the future discussions, please be specific in your numbers. There is MB/s and there is mbit/s. To mix this up is unhealthy to get a common sense of the actual throughput numbers. 

No long 40GB downloads did not appear to speed up, I left them for over two hours and they did not increase in speed HOWEVER when I reverted to UTM and tried again the downloads started and finished in minutes.  To my mind there is certainly an issue with the XG firewall, even with IPS and every other feature switched off the speed did not improve.

edited my original message to be more specific on numbers  :-)