Sophos Firewall: v18.5 MR4: Feedback and experiences

Since I can't wait for Sophos to release v19 MR1 and v18.5 MR4 won't fix the SASI hassle that's going on, I decided to downgrade to 18.5 MR2. However, this seems to be a problem sometimes.

I have a XGS126 that was upgraded from 18.5 MR2 to 19 GA. I could downgrade it back to 18.5 MR2 without any problem.

After that I tried to downgrade a XG210 that was upgraded from 18.5 MR3 to 19 GA. When uploading 18.5 MR2 or MR4 to the firewall it said "The firmware will boot the device with factory default configuration. Are you sure you want to continue?". Since I can't stand all the spam, I did it anyway and booted the uploaded 18.5 MR2. It indeed came up with factory defaults loaded. I tried to restore the latest config backup from 19 GA, but it just said that the firmware is not matching, so I restored an old backup from a time the firewall had installed 18.5 MR2, which worked. After that I just booted up the still present 19 GA firmware and the firewall came up with the restored MR2 config. I restored the latest 19 GA backup and booted the 18.5 MR2 firmware. I expected the firewall to come up with factory defaults again, but it still has the latest config running.

So thats the processes that work:
18.5 MR2 > 19 GA > 18.5 MR2/MR4 without factory reset
18.5 MR3 > 19 GA > 18.5 MR2/MR4 with factory reset  > 19 GA > restore latest config from 19 GA > 18.5 MR2/MR4 without factory reset

Why do I need a factory reset if 18.5 MR3 was previously installed, but not if I skipped it? In both scenarios I go 19 GA > 18.5 MR2 in the end.

I do not have the answer to this point but MR2 will go End of Life soon. Per definition of the lifecycle policy: https://support.sophos.com/support/s/article/KB-000035279?language=en_US#xgfirewallsoftware

https://community.sophos.com/sophos-xg-firewall/b/blog/posts/sophos-lifecycle-policy-changes-end-of-life-announcement-for-older-sfos-versions

So i assume, it is not smart to go back in time to MR2. 

The only thing I see is that a lot of very obvious spam mails are still getting through, which is not the case with MR2, so I hope that this is not as good as it gets

Check the Logs for those Emails and see, what the engine reports. 

I see mails in GUI Mail log that are not present in the smtpd_main.log, is that by design?

The mail log can eventually rotate. See: https://docs.sophos.com/nsg/sophos-firewall/18.0/Help/en-us/webhelp/onlinehelp/AdministratorHelp/Logs/LogFileDetails/index.html

In Webadmin the logs are a database, which can eventually store much more meta data compared to the plain text data. 

I found the last two obvious spam mails that got through:

MSG   Jun 21 03:58:24Z [1o3V1z-0005pO-SF]: spam scanning result: 'NonSpam'
MSG   Jun 21 03:58:45Z [1o3V2K-0005qt-IP]: spam scanning result: 'NonSpam'

So I guess the SASI engine is just bad then.

SASI is being used in UTM, Central Email, Email Appliance, PureMessage etc. 

It is now in the retrospective hard to compare those, as you do not have the comparison for this exact email. 

But: Do you use IP Reputation? 

That's not correct, I indeed can compare both engines for the very same email. I just send one of the two mails I posted above to an exchange server behind a XGS I downgraded to MR2, guess what the result was?

MSG   Jun 21 09:15:06Z [1o3ZyT-0006wW-Mn]: spam scanning result: 'Confirmed spam'

Could you provide this particular Email as a Lab Request to get this sorted out? 

Sure, I submitted a few samples to is-spam@labs.sophos.com that passed the SASI engine with ease.

Why is the detection rate of the SASI engine that poor? We're talking about the usual "buy cheap viagra here!!!!"-mails I haven't received in years, just until you guys decided it would be a good idea to switch the spam engine. Since that day me and our customers a pleased with the most obvious spam mails that we all thought were dead.

It looks to me like Sophos doesn't see this for what it is: a serious problem that needs a fast solution. Our customers were willing to wait for a patch, but now it seems like the engine is just bad and a fix will, as always with Sophos, take ages to get released. I simply can't expect our customers to wait for that long, so if you guys don't surprise us, then we will look for another product from another company, at least for mail protection. Not just because we're disappointed with Sophos, but also because our customers would not be willing to buy another solution from the very same company that broke the one they already bought. I'm still having trouble explaining why a company like Sophos would replace a reasonably functional antispam engine without thoroughly testing the replacement.

It looks to me like Sophos doesn't see this for what it is: a serious problem that needs a fast solution. Our customers were willing to wait for a patch, but now it seems like the engine is just bad and a fix will, as always with Sophos, take ages to get released. I simply can't expect our customers to wait for that long, so if you guys don't surprise us, then we will look for another product from another company, at least for mail protection. Not just because we're disappointed with Sophos, but also because our customers would not be willing to buy another solution from the very same company that broke the one they already bought. I'm still having trouble explaining why a company like Sophos would replace a reasonably functional antispam engine without thoroughly testing the replacement.

So do you use the reputation filter in MTA or not? You did not answer this question.

And also which kind of RBL do you use? 

Hi, 

The filter works in legacy mode.

RBL:
dnsbl-1.uceprotect.net
pbl.spamhaus.org
sbl-xbl.spamhaus.org

I don't see how this is even relevant, since the settings have not changed but the Spam Engine has. Anyway, reputation filter is enabled and I use all RBLs provided by Sophos + zen.spamhaus.org

Here you have a comparison of one week MR2 vs. one week MR4:

The detection rate is just plain bad and I highly doubt that you guys did any propper testing before releasing this to your customers!

Hi Dreamcatcher, a new Development ID NC-98258 has been assigned to investigate this issue of poor SPAM detection rate even after upgrading to 18.5.4. We will update this thread as we know more.

has there been any movement on this? I show last response about a month ago and a similar chain on the xg v19mr1 with no resolution either. We have noticed this new poor anti spam detection as well and have been consequently upgrading to latest MR in hopes it would get fixed but nothing yet.

No progress so far, 19.0 MR1 is as bad as 18.5 MR3/MR4. I have a case open with Sophos, but so far it doesn't seem to go anywhere in the near future. They just want me to send them spam mails that got through so they can submit them to Sophos labs. My customers get super obvious spam mails with the corresponding mail headers showing a SASI spam probability between 10-20%, it's ridiculous.

We've noticed this as well but have not generated a specific ticket for it, yet - mainly because we found another issue related to email security we'd like support to address first related to SPF, or lack thereof.  I really hope Sophos starts to get things turned around for maintenance/breakage items and less on enhancements as the list of things breaking is growing.

Hi Karlos, you have some news to share on this one?