Did the RAM limits change for Home edition for 16.05.2 to 4GB?

Hi All,

Investigating it further, the issue might be associated with the Kernel which is just identifying the RAM on 1 slot out of 2. For example ( 4 GB + 4 GB). Kernel only limits the RAM on single slot

discarding 2nd slot completely so when 6GB limit is imposed only 4GB RAM becomes visible. We are Investigating further. 

Thanks for the patience

What is frustrating about the whole thing is that

• First you guys said nothing has changed.
• Users point out the new 6G switch and then oh yeah something has changed.
• Well, some mysterious people were seeing 8GB ram when they should have been seeing 6 so we shafted everyone.
• That darn linux kernel. Certain switches make it only look at the first memory stick.

XG is already getting a black eye from most commercial users that regret the fact that they chose it over UTM. Now you guys are alienating the advanced home user base. Don't get me wrong, I appreciate the very generous hardware quota for home installations but with so many things that break with every release as evident from the posts in this form, I am sure the mystery 8GB user on home license was a not a top problem on Sophos' list.

Hey careful there BillyBob, I am no mysterious home user. Did have 8gb in reports, but never used that much, now have less than 6gb but more than 5gb.

I agree careful what you say, I am also an enterprise user (XG 310's), while I never used the UTM, the XG is years beyond our old firewall. Not to mention cost, Sophos does a great thing by giving it away to home users. Its a great way to get feedback and help build a more stable platform.

Brian

I have dealt with firewalls for a long time in some form or another(been an System admin for 20+ years).

The functionality Sophos gives you for free is very good. I looked at all the other "free" solutions and kept heading back to Sophos.

I've tried pfsense/opnsense/vyos and they are missing some of the anti-virus/web scanning features without jumping thru major hoops.

Untangle doesn't support ipv6 4in6 tunnels natively, and the good stuff cost $50 a year.

ClearOS/Zentayl are just too complex to keep secure as it tries to do too much.

The only reason i left UTM for XG is the IP limit in UTM and IPv6 addressing chewing up ip's toward the limit.

I really don't want to spin my own.

I'm hoping the 4GB memory issue in XG is worked out soon. Worst case i'd probably fall back to opnsense.

Before this i was using a TomatoUSB firmware variation(shibby) on my netgear WNR3500L v2, but updates have pretty much dried up for that router and it was time to move to something better and something that can run any firewall that can run on X86 hardware. I bought the Zotac Ci323 to run my home firewall of choice. It would be nice to have wireless support in sophos but i know that's asking a bit much.

Maybe sophos could do a low cost per year home use "unlimited" license similar to what Untangle did. Just an idea.

I agree, I would be willing to pay a reasonable yearly fee to have more memory and features, such as access to sandstorm, and Heartbeat on Sophos Free AV. Hopefully Sophos can help figure out soon this kernel limitation.

I too would pay a small nominal fee too, look at ClearOS Home, they have a small fee, 3-5 bucks a month.  Very reasonable and to have access to the other features would be great, hopefully Sophos would see the advantage of home users feeding into the larger data analytics which in turn further help their corporate customers.  

Guys, while agree with you on Sophos' slow response to the problem, the memory reduction shouldn't affect you that much in reality. I was using XG with 4GB in a vm and it ran fine with about 50ish IPs behind it. Granted I only have a few actual users and most of the other stuff is IoTs and streaming devices, but unless you are a heavy user with a lot of people like a LAN party, 4GB shouldn't impact you severely.

As far as a monthly fee, sophos has been generous to give the product for free. Every few years, someone asks for a home license that would cost 5-10 dollars a month, 100 dollars a year and so on. While I personally don't mind the idea, astaro and now sophos don't want to do home user market. By giving the product away free, they don't have to worry about supporting every home user and then showing them what is a subnet mask. I would rather use a full featured enterprise product with no support than a watered down home version of XG that then I have to pay for to get fewer features that are focused for a home user.

Yeah 4G isn't that big a deal for me in particular, but the licenses explicitly states 6G of memory is allowed and you can have more in the system. The mem=6G addition in my opinion was not fully researched on it's potential impact. If you look up the option in the linux documentation it specifically says it's a address limit. That address limit appears to break the ability to remap around the reserved space. Some people on the internet have said that it's sometimes required to actually set the limit higher that the amount of memory you want to actually get the targeted amount.

If i knew how to test varying kernel options using the kernel and boot loader that sophos uses i would. I may try testing this weekend with an ubuntu 14.04 image (3.13 kernel) and setting mem=6G and see what happens, but it's not an apples to apples comparison. 

The fix number quoted by sophos support wasn't even listed in the release notes for MR2, or MR1, which also makes me think it may not have gotten the full attention it deserved.

Makes me even wonder what would happen if mem= was used on their own XG hardware.

I'm very grateful for what they provide. This is simply a bug that needs to be fixed. Their fix to limit to 6G is broken depending on hardware. 

In the end they need to find a different solution to limiting memory or check the results of the mem= and increase the limit until the system boots with the wanted amount of usable memory. I'd imagine some program that just steals/allocates the total memory minus 6G might be another idea. There is so little documentation on mem= out there i personally wouldn't use it unless absolutely necessary. 

Yeah 4G isn't that big a deal for me in particular, but the licenses explicitly states 6G of memory is allowed and you can have more in the system. The mem=6G addition in my opinion was not fully researched on it's potential impact. If you look up the option in the linux documentation it specifically says it's a address limit. That address limit appears to break the ability to remap around the reserved space. Some people on the internet have said that it's sometimes required to actually set the limit higher that the amount of memory you want to actually get the targeted amount.

If i knew how to test varying kernel options using the kernel and boot loader that sophos uses i would. I may try testing this weekend with an ubuntu 14.04 image (3.13 kernel) and setting mem=6G and see what happens, but it's not an apples to apples comparison. 

The fix number quoted by sophos support wasn't even listed in the release notes for MR2, or MR1, which also makes me think it may not have gotten the full attention it deserved.

Makes me even wonder what would happen if mem= was used on their own XG hardware.

I'm very grateful for what they provide. This is simply a bug that needs to be fixed. Their fix to limit to 6G is broken depending on hardware. 

In the end they need to find a different solution to limiting memory or check the results of the mem= and increase the limit until the system boots with the wanted amount of usable memory. I'd imagine some program that just steals/allocates the total memory minus 6G might be another idea. There is so little documentation on mem= out there i personally wouldn't use it unless absolutely necessary. 

NOTE: I did some experimentation on my firewall hardware using an Ubuntu 16.04 mini ISO which is a newer kernel(4.4)

I booted the kernel with different values of mem= 

mem=7G  resulted in around 4.2G usable. I didn't write it down sorry.
mem=8G  resulted in 5963420 K usable
mem=9G  resulted in 6995608 K usable
mem=10G resulted in 8027796 K usable, which is what I get when booting without mem=

I think this also shows that even with a fairly recent kernel mem= isn't doing what i believe Sophos expected it to do. It simply lops off the ability to map any memory over 6G. If the system memory is mapped outside that range you are out of luck

Here's the memory map from dmesg, i made the values as close as possible. might be rounding errors.

BIOS-e820: [mem 0x0000000000000000-0x000000000009c7ff] usable => RANGE is 0K to 641023(640K) 1M
BIOS-e820: [mem 0x0000000000100000-0x000000001effffff] usable => RANGE is 1048576(1MB) to 520093695(520MB) 519MB
BIOS-e820: [mem 0x0000000020200000-0x000000007b218fff] usable => RANGE is 538968064(539MB) to 2065797119(2048MB) 1509MB
BIOS-e820: [mem 0x000000007b294000-0x000000007b3bbfff] usable => RANGE is 2066300928(2066MB) to 2067513343(2067MB) 1MB
BIOS-e820: [mem 0x000000007bb31000-0x000000007bffffff] usable => RANGE is 2075332608(2075MB) to 2080374783 (2080MB) 5MB
BIOS-e820: [mem 0x0000000100000000-0x000000027fffffff] usable => RANGE is 4294967296(4294MB) to 10737418239(10737MB) 6443MB


In my case the that last memory segment is 6G of my 8G of memory, but it starts at 4GB instead of directly after the previous block at 2G.

When the mem=6G kicks in the last memory segment looses 4G of capacity.

user: [mem 0x0000000100000000-0x000000017fffffff] usable => RANGE is 4294967296(4294MB) to 6442450943(6442MB) 2148MB

AND....

1+519+1509+1+5+6443 = 8478

1+519+1509+1+5+2148 = 4183

My math might be a little off but i think this proves my point without a shadow of a doubt. 

If we want to figure out why some are ok, and some are 4G, and others are somewhere in-between i'm pretty certain it's going to map back to where your usable memory maps in.

If you can't get the info from /var/tslog/syslog.log anymore the current RAM ranges can be found using the command below.

SFVH_SO01_SFOS 16.05.2 MR-2# grep RAM /proc/iomem
00001000-0009c7ff : System RAM
00100000-1effffff : System RAM
20200000-7b217fff : System RAM
7b293000-7b3bafff : System RAM
7bb30000-7bffffff : System RAM
100000000-17fffffff : System RAM


I believe this is likely what my bios is doing, and probably many others:

https://en.wikipedia.org/wiki/PCI_hole#Mapping_memory_to_addresses_above_4_GB

Kernel Code link for mem= 

http://lxr.free-electrons.com/source/arch/x86/kernel/e820.c#L839

Kevin Brierly said:

In the end they need to find a different solution to limiting memory or check the results of the mem= and increase the limit until the system boots with the wanted amount of usable memory. I'd imagine some program that just steals/allocates the total memory minus 6G might be another idea. There is so little documentation on mem= out there i personally wouldn't use it unless absolutely necessary.

They have the whole api. They don't need to do command line hacks to enforce licensing limits. This seems like a quick hack by someone who is not talking to the whole licensing team to narrow down the licenses that actually have a problem and then fix the problem. Sophos employees (atleast the ones that interact with us)not having any knowledge on the change is also a little astounding.

By the way, thanks for the extensive memory testing.

No big deal doing the testing. I figured if i didn't provide the data on what's happening the issue may have not proceeded.

I'm hoping we get some sort of answer on this one from sophos soon. To me this is probably impacting any home user that has high memory mapped starting at 4G and above.

The Bios on the zotac ci323 doesn't have any options for changing the memory remapping behavior, and honestly i don't think they need to. Not doing it leads to other issues.

I'd be interested in seeing "grep RAM /proc/iomem" of the people with various amounts of memory to make sure my conclusions follow to other hardware and other users.

I'm pretty sure they will.

The question in the end will be if Sophos will backout the hard limit at mem=6G and replace it with something else, or actually scale mem= based on individual systems memory maps. 

With a small script/program it should be VERY easy to determine a value to use.

Kevin Brierly said:

I'd be interested in seeing "grep RAM /proc/iomem" of the people with various amounts of memory to make sure my conclusions follow to other hardware and other users.

I have 2x4GB installed.  Supermicro X10SBA-L-O

SFVH_SO01_SFOS 16.05.2 MR-2# grep RAM /proc/iomem
00001000-00089bff : System RAM
00100000-1effffff : System RAM
1f100000-1fffffff : System RAM
20100000-78825fff : System RAM
78b81000-78b81fff : System RAM
78bc4000-78d2ffff : System RAM
78ffa000-78ffffff : System RAM
79000000-7affffff : RAM buffer
100000000-17fffffff : System RAM

DavidWilliams1 said:

 

I have 2x4GB installed.  Supermicro X10SBA-L-O

SFVH_SO01_SFOS 16.05.2 MR-2# grep RAM /proc/iomem
00001000-00089bff : System RAM
00100000-1effffff : System RAM
1f100000-1fffffff : System RAM
20100000-78825fff : System RAM
78b81000-78b81fff : System RAM
78bc4000-78d2ffff : System RAM
78ffa000-78ffffff : System RAM
79000000-7affffff : RAM buffer
100000000-17fffffff : System RAM