Hi folks,
my my XG's memory usage suddenly went from 44% to 70% and has now settled at 60%.
This has only happened since I started adjusting maxpkts as part of the TOR/Psiphon blocking setup.
In the graph below the grey section is immediately after a resart.
Thoughts please?
Ian
Check via atop the process, which uses this memory.
You can do a "ring buffer" atop.
atop -w /var/log/atop.log 60 &
https://linux.die.net/man/1/atop
Hi Toni,
there are two copies of snort running each using 12% of memory. That is interesting because I thought I set snort up sometime ago to have 3 copies running.
Looks like snort is the culprit., but why? So what do I do to reduce snort other than removing the TOR/Psiphon configuration?
Ian
I removed the 'none' from the web policy and the memory dropped back to 41% after a restart.
I realise the Psiphon block will not work effectively, there is an upside though, the number of exceptions that have to be added to allow access to some sites are reduced.
Ian
Hi, wondering about the same thing. I migrated my config to identially move from a 4 core / 4 thread UTM 425 to a Protectli FW6C 2 core / 4 thread i5 box. 8 gb memory. Same rules and same IPS config but now my base memory jumped from about 43% to 60%. There are 4 snort instances that say 20% on each of them, but the old box said 8%. Performance on the new box is great. Lowmem is off (was on the old box as well). Any insight into why snort seems to be grabbing more mem on the new box with the same config?
Ty
Hi,
I am currently running v18.0.2 MR-2 and my memory usage is around 50%. Memory usage is only an issue if you start seeing values of around 80-90% and performance degradation.
The snort instances will uses more CPU on you new hardware because it is lower power, with a slower CPU and also only two e=real cores. The 450 has an e3 CPU with 4 real cores.
Ian
Thanks, actually the CPU (even with only two physical cores) is handling everything superbly. I have 400/20 (down/up) and get full bandwidth downloads with AV, IPS, ATP, App/Web Filters. very smooth. Was just curious that the memory is 17% more with an identical config on two different pieces of hardware.
This small box works like a charm. Completely silent (passive cooling, large heatsink). The 425 with its high speed 40x40x28 fans was a noise machine.
Would recommend the protectli highly. I'm going to load MR-2 tonight.
Thanks
Thanks, actually the CPU (even with only two physical cores) is handling everything superbly. I have 400/20 (down/up) and get full bandwidth downloads with AV, IPS, ATP, App/Web Filters. very smooth. Was just curious that the memory is 17% more with an identical config on two different pieces of hardware.
This small box works like a charm. Completely silent (passive cooling, large heatsink). The 425 with its high speed 40x40x28 fans was a noise machine.
Would recommend the protectli highly. I'm going to load MR-2 tonight.
Thanks
Hi, another question. I have IPS enabled and when I look at top there are 4-5 instances of IPS, and they indicate an increase in CPU with high bandwidth transfers - all as expected.
However, when I go into the console under option 4, and issue the command show ips-settings, it doesn't list any IPS Intances, but you can see they are there in top. Any idea?
-------------IPS Settings-------------
stream on
lowmem off
maxsesbytes 0
maxpkts 8
enable_appsignatures on
http_response_scan_limit 65535
search_method ac-q
sip_preproc enabled
sip_ignore_call_channel enabled
inspect untrusted-content
-------------IPS Instances------------
IPS CPU
console>
top - 11:08:33 up 11:54, 1 user, load average: 0.07, 0.11, 0.10
Tasks: 450 total, 1 running, 368 sleeping, 0 stopped, 0 zombie
Cpu0 : 0.3%us, 0.3%sy, 0.0%ni, 99.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu1 : 0.0%us, 0.0%sy, 0.0%ni, 99.7%id, 0.3%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu2 : 0.3%us, 0.0%sy, 0.0%ni, 99.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu3 : 0.3%us, 1.3%sy, 0.0%ni, 98.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 6094116k total, 5301940k used, 792176k free, 285696k buffers
Swap: 8050804k total, 0k used, 8050804k free, 1357572k cached
PID PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
21820 20 0 2154m 1.1g 45m S 0.0 18.3 1:32.60 snort
21817 20 0 2154m 1.1g 45m S 0.3 18.3 1:25.77 snort
21819 20 0 2154m 1.1g 45m S 0.3 18.3 1:42.22 snort
21818 20 0 2154m 1.1g 44m S 0.3 18.3 1:24.88 snort
21667 20 0 1286m 1.0g 17m S 0.0 17.6 0:29.98 snort
8452 20 0 605m 396m 15m S 0.0 6.7 0:53.76 avd
7576 20 0 3774m 334m 16m S 0.3 5.6 0:47.59 java
8409 20 0 298m 127m 8716 S 0.0 2.1 1:19.74 awarrenhttp
7317 20 0 129m 117m 3316 S 0.0 2.0 0:42.62 dnscache
6831 20 0 80524 57m 13m S 0.0 1.0 1:04.68 garner
5861 20 0 46660 32m 32m S 0.0 0.5 0:00.24 postgres
25536 20 0 48116 32m 30m S 0.0 0.5 0:02.74 postgres
5893 20 0 46772 32m 31m S 0.0 0.5 0:03.05 postgres
5862 20 0 46660 31m 30m S 0.0 0.5 0:00.13 postgres
Thanks. I set the snort/cpu affinity with the command set ips ips-instance add IPS cpu for each of the four threads. Now the show ips-settings shows four IPS instances and the performance and CPU usage with high bandwidth traffic is the same. Top still shows five snort processes. I remember reading somewhere that XG uses snort for IPS and another function, but I can't remember what this is. Do you recall? (Notice how the 5th snort has a different virtual size and a non-sequential PID, while the first four seem to fit together).
console> show ips-settings
-------------IPS Settings-------------
stream on
lowmem off
maxsesbytes 0
maxpkts 8
enable_appsignatures on
http_response_scan_limit 65535
search_method ac-q
sip_preproc enabled
sip_ignore_call_channel enabled
inspect untrusted-content
-------------IPS Instances------------
IPS CPU
1 0
2 1
3 2
4 3
top - 19:04:36 up 19:50, 1 user, load average: 0.19, 0.15, 0.10K
Tasks: 452 total, 1 running, 370 sleeping, 0 stopped, 0 zombie
Cpu0 : 4.3%us, 1.3%sy, 0.0%ni, 94.4%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu1 : 6.7%us, 2.3%sy, 0.0%ni, 90.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu2 : 1.7%us, 0.7%sy, 0.0%ni, 97.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu3 : 3.3%us, 0.3%sy, 0.0%ni, 96.0%id, 0.3%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 6094116k total, 5625472k used, 468644k free, 378492k buffers
Swap: 8050804k total, 0k used, 8050804k free, 1586988k cached
PID PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
21818 20 0 2218m 1.1g 48m S 0.0 18.4 4:11.32 snort
21819 20 0 2218m 1.1g 48m S 0.3 18.4 4:44.75 snort
21820 20 0 2218m 1.1g 48m S 0.0 18.4 3:50.08 snort
21817 20 0 2218m 1.1g 48m S 0.3 18.4 4:13.62 snort
21667 20 0 1286m 1.0g 17m S 0.0 17.6 0:30.06 snort
8452 20 0 612m 406m 14m S 0.0 6.8 1:17.04 avd
7576 20 0 3776m 285m 17m S 0.3 4.8 1:14.15 java
8409 20 0 314m 129m 8716 S 0.0 2.2 3:03.61 awarrenhttp
7317 20 0 129m 117m 3316 S 0.0 2.0 0:57.17 dnscache
6831 20 0 78476 55m 13m S 0.0 0.9 1:51.30 garner
5861 20 0 46660 32m 32m S 0.0 0.5 0:00.25 postgres
5893 20 0 46888 32m 31m S 0.0 0.5 0:05.47 postgres
5862 20 0 46660 31m 30m S 0.0 0.5 0:00.19 postgres
8297 20 0 34244 30m 5784 S 0.0 0.5 0:06.25 awed [master]
Hi,
5 snort instances seems odd to me going on discussions from people with knowledge about snort. At this stage the snort in XG is only single threaded, so having a 5th one is vey add. A newer version of snort is multi-threading, but I am not sure about it production status and when it will be added to XG.
In XG snort is used for classification and as well intrusion reporting/blocking.
Ian