SNORT high load average killing all connections

Do you know of a way to completely restart SNORT via CLI?  I have been just killing processes off... is there another way?

OK, the graph definately peaks out!

I'm not aware of such a command to stop IPS on shell.

Is your setup HA cluster? Did you perform any (even minor) change at about 8:00 yesterday? Please check Admin log for changes.

I have an issue where a change on a RED would kill the cluster in a HA environment after some minutes. I sometimes noticed high SNORT CPU in such a situation.

I found this thread https://community.sophos.com/sophos-xg-firewall/f/discussions/82468/cluster-xg-310---snort-100---firewall-block-all-traffic-and-routing which is describing some of your problems.

Can you check ips-settings?

console> sh ips-settings 

-------------IPS Settings-------------

stream on

lowmem off

maxsesbytes 0

maxpkts 8

OK, the graph definately peaks out!

I'm not aware of such a command to stop IPS on shell.

Is your setup HA cluster? Did you perform any (even minor) change at about 8:00 yesterday? Please check Admin log for changes.

I have an issue where a change on a RED would kill the cluster in a HA environment after some minutes. I sometimes noticed high SNORT CPU in such a situation.

I found this thread https://community.sophos.com/sophos-xg-firewall/f/discussions/82468/cluster-xg-310---snort-100---firewall-block-all-traffic-and-routing which is describing some of your problems.

Can you check ips-settings?

console> sh ips-settings 

-------------IPS Settings-------------

stream on

lowmem off

maxsesbytes 0

maxpkts 8

OK, the graph definately peaks out!

I'm not aware of such a command to stop IPS on shell.

Is your setup HA cluster? Did you perform any (even minor) change at about 8:00 yesterday? Please check Admin log for changes.

I have an issue where a change on a RED would kill the cluster in a HA environment after some minutes. I sometimes noticed high SNORT CPU in such a situation.

When the issue is happening, I ask you to ping the aux/peer HA node Cluster IP from shell (note it before). Is it responding?

console> system ha show details
 HA status              : Enabled
 Current Appliance Key  : xxx
 Peer Appliance Key     : xxx
 Current HA state       : Primary
 Peer HA state          : Auxiliary
 HA Config Mode         : Active-Passive
 Load Balancing         : Not Applicable
 Dedicated Port         : PortX
 Current Dedicated IP   : xxx.xxx.xxx.xxx
 Peer Dedicated IP      : xxx.xxx.xxx.xxx

I found this thread https://community.sophos.com/sophos-xg-firewall/f/discussions/82468/cluster-xg-310---snort-100---firewall-block-all-traffic-and-routing which is describing some of your problems.

Can you check ips-settings?

console> sh ips-settings 

-------------IPS Settings-------------

stream on

lowmem off

maxsesbytes 0

maxpkts 8

can you also please post last IPS pattern date/time update on your machine?

 cat /log/u2d.log | grep "pt_dload_checker" | grep "ips"

Tue Aug 03 07:24:29 2021 pt_dload_checker: Download for file ips_16.0_v18_18.18. 41.tar.gz.gpg passed integrity and gpg checks
Tue Aug 03 07:24:29 2021 pt_dload_checker: Current ips patterns are at /content/ ips_16.0/18.18.40
Tue Aug 03 07:24:29 2021 pt_dload_checker: New updated patterns are now at /cont ent/ips_16.0/18.18.41
Tue Aug 03 07:25:45 2021 pt_dload_checker: Updated signature db for ips, version = 18.18.41.
Tue Aug 03 07:25:45 2021 pt_dload_checker: Deleted pattern for module ips, versi on = 18.18.40 at /content/ips_16.0/18.18.40.
Thu Aug 05 07:23:27 2021 pt_dload_checker: Starting download for file ips_16.0_v 18_18.18.42.tar.gz.gpg
Thu Aug 05 07:24:26 2021 pt_dload_checker: Download completed for file ips_16.0_ v18_18.18.42.tar.gz.gpg
Thu Aug 05 07:24:26 2021 pt_dload_checker: We are primary machine in HA. Syncing download for module ips to auxiliary machine
Thu Aug 05 07:25:28 2021 pt_dload_checker: Download for file ips_16.0_v18_18.18. 42.tar.gz.gpg passed integrity and gpg checks
Thu Aug 05 07:25:28 2021 pt_dload_checker: Current ips patterns are at /content/ ips_16.0/18.18.41
Thu Aug 05 07:25:28 2021 pt_dload_checker: New updated patterns are now at /cont ent/ips_16.0/18.18.42
Thu Aug 05 07:26:43 2021 pt_dload_checker: Updated signature db for ips, version = 18.18.42.
Thu Aug 05 07:26:43 2021 pt_dload_checker: Deleted pattern for module ips, versi on = 18.18.41 at /content/ips_16.0/18.18.41.
Wed Aug 11 01:24:27 2021 pt_dload_checker: Starting download for file ips_16.0_v 18_18.18.43.tar.gz.gpg
Wed Aug 11 01:25:49 2021 pt_dload_checker: Download completed for file ips_16.0_ v18_18.18.43.tar.gz.gpg
Wed Aug 11 01:25:49 2021 pt_dload_checker: We are primary machine in HA. Syncing download for module ips to auxiliary machine
Wed Aug 11 01:26:50 2021 pt_dload_checker: Download for file ips_16.0_v18_18.18. 43.tar.gz.gpg passed integrity and gpg checks
Wed Aug 11 01:26:50 2021 pt_dload_checker: Current ips patterns are at /content/ ips_16.0/18.18.42
Wed Aug 11 01:26:50 2021 pt_dload_checker: New updated patterns are now at /cont ent/ips_16.0/18.18.43
Wed Aug 11 01:28:08 2021 pt_dload_checker: Updated signature db for ips, version = 18.18.43.
Wed Aug 11 01:28:08 2021 pt_dload_checker: Deleted pattern for module ips, versi on = 18.18.42 at /content/ips_16.0/18.18.42.
Thu Aug 12 09:24:26 2021 pt_dload_checker: Starting download for file ips_16.0_v 18_18.18.44.tar.gz.gpg
Thu Aug 12 09:25:27 2021 pt_dload_checker: Download completed for file ips_16.0_ v18_18.18.44.tar.gz.gpg
Thu Aug 12 09:25:27 2021 pt_dload_checker: We are primary machine in HA. Syncing download for module ips to auxiliary machine
Thu Aug 12 09:26:27 2021 pt_dload_checker: Download for file ips_16.0_v18_18.18. 44.tar.gz.gpg passed integrity and gpg checks
Thu Aug 12 09:26:27 2021 pt_dload_checker: Current ips patterns are at /content/ ips_16.0/18.18.43
Thu Aug 12 09:26:27 2021 pt_dload_checker: New updated patterns are now at /cont ent/ips_16.0/18.18.44
Thu Aug 12 09:27:48 2021 pt_dload_checker: Updated signature db for ips, version = 18.18.44.
Thu Aug 12 09:27:48 2021 pt_dload_checker: Deleted pattern for module ips, versi on = 18.18.43 at /content/ips_16.0/18.18.43.

console> system ha show details
HA status : Enabled
Current Appliance Key : <removed>
Peer Appliance Key : <removed>
Current HA state : Primary
Peer HA state : Auxiliary
HA Config Mode : Active-Passive
Load Balancing : Not Applicable
Dedicated Port : Port8
Current Dedicated IP : 10.254.254.1
Peer Dedicated IP : 10.254.254.2
Monitoring Port : Port1
Auxiliary Admin Port : Port1
Auxiliary Admin IP : 192.168.100.6
Auxiliary Admin IPv6 :
HA Cluster ID : 0
Keepalive request interval : 250
Keepalive attempts : 16
Hypervisor assigned MAC addresses : Disabled
HA preemption : Disabled

console> sh ips-settings
-------------IPS Settings-------------
ac_atp_exception_fwrules 7
stream on
lowmem off
maxsesbytes 0
maxpkts 8
enable_appsignatures on
http_response_scan_limit 65535
search_method hyperscan
sip_preproc enabled
sip_ignore_call_channel enabled
inspect all-content

-------------IPS Instances------------
IPS CPU
1 0
2 1
3 2
4 3

AV Definitions updated overnight a bunch overnight, we were checking ever 2 hours so... it makes sense that these updates were happening. Maybe a bad AV definition update?

This login event was from the administrator that alerted me that the CPU was high.

OK, all looks good to me.

I suggest, you check the graphs again. There was something at about 8:00 on, I think it was Sunday.

Do you find peaking traffic on that time on any interface? If yes, I think there was a rule with IPS beeing hit.

And again, did you find something in the ips.log on that time?

It could be a core dump problem. You should check core dumps and create a support case.

Maybe update to V18.5 MR1 first and check, if this still occurs. 

cd /var/cores/

XG330_WP02_SFOS 18.0.5 MR-5-Build586# ls -la
drwxrwxrwt 2 root 0 4096 Oct 22 2020 .
drwxr-xr-x 44 root 0 4096 Aug 18 01:12 ..
-rw------- 1 root 0 20897792 Oct 22 2020 core.sandbox_reportd

Not sure what this means but there is only one core from last October in that folder...