Guest User!

You are not Sophos Staff.

Thread Info
  • State Verified Answer
  • Locked Locked
  • Replies 22 replies
  • Subscribers 30 subscribers
  • Views 56388 views
  • Users 0 members are here
Options
Suggested
This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

[Probably a bug] IPS engine restarting constantly

IeM

TLDR; since firmware SFOS 16.01.1 the IPS service dies and keeps restarting. Probably the reason why the service cannot init:
/bin/snort: /lib/libcrypto.so.1.0.0: no version information available (required by /bin/snort)

Related/Relevant forum threads:

[1] Upgrade Sophos XG from SFOS15 to SFOS16 and "lost internet access". Reverting back to SFOS15 as well ass disabling the IPS service fixed it. community.sophos.com/.../315511

[2] "IPS keeps restarting". community.sophos.com/.../version-16-beta-5-and-ga-ips-keeps-restarting

--

Hi there,

Upgraded my Sophos XG Firewall (from SFOF 15.x) to version SFOS 16.01.2. After the upgrade no changes have been made to the configuration. Short after the upgrade I found out that all in- and outgoing traffic is blocked. I have traced it back to the IPS service. Stopping these service will result in allowing traffic.

IPS
Nothing shows up in my log file (log viewer) despite that IPS logging is enabled. Also the IPS status gives the status back that 0 packets are dropped. Disabled the IPS policy for the rules on which the firewall should hit but that is not making any sense. 

Firewall
The firewall is blocking the traffic because it is 'invalid traffic'.

Using the device console I have dumped some traffic on the network. Below is an example packet (that is blocked):

2016-12-25 16:36:57 0102021 IP 172.17.29.3.33533 > 5.153.231.35.80 : proto TCP: P 3441838487:3441838
649(162) win 229 checksum : 52802                                                                   
0x0000:  4500 00d6 4fab 4000 4006 34a6 ac11 1d03  E...O.@.@.4.....                                  
0x0010:  0599 e723 82fd 0050 cd26 4997 2a9a a931  ...#...P.&I.*..1                                  
0x0020:  8018 00e5 ce42 0000 0101 080a 0011 5bc2  .....B........[.                                  
0x0030:  110f 6655 4745 5420 2f64 6562 6961 6e2f  ..fUGET./debian/                                  
0x0040:  6469 7374 732f 6a65 7373 6965 2f49 6e52  dists/jessie/InR                                  
Date=2016-12-25 Time=16:36:57 log_id=0102021 log_type=Firewall log_component=Invalid_Traffic log_sub
type=Denied log_status=N/A log_priority=Alert duration=N/A in_dev=Port6 out_dev= inzone_id=0 outzone
_id=0 source_mac=32:64:36:64:38:38 dest_mac=36:65:61:65:39:36 l3_protocol=IP source_ip=172.17.29.3 d
est_ip=5.153.231.35 l4_protocol=TCP source_port=33533 dest_port=80 fw_rule_id=0 policytype=0 live_us
erid=0 userid=0 user_gp=0 ips_id=0 sslvpn_id=0 web_filter_id=0 hotspot_id=0 hotspotuser_id=0 hb_src=
0 hb_dst=0 dnat_done=0 icap_id=0 app_filter_id=0 app_category_id=0 app_id=0 category_id=0 bandwidth_
id=0 up_classid=0 dn_classid=0 source_nat_id=0 cluster_node=0 inmark=0x0 nfqueue=0 scanflags=0 gatew
ay_offset=0 max_session_bytes=0 drop_fix=0 ctflags=0 connid=0 masterid=0 status=0 state=0 sent_pkts=
N/A recv_pkts=N/A sent_bytes=N/A recv_bytes=N/A tran_src_ip=N/A tran_src_port=N/A tran_dst_ip=N/A tr
an_dst_port=N/A  >

What can be the cause? Any help is appreciated.


Regards, Ilias



This thread was automatically locked due to age.
Parents
  • 0

    Hi Ilias,

    Make sure the IPS is up2date. Restart IPS service from System > services> IPS. Alongside, the drop logs tells that the drop traffic is invalid which means any one of the condition is mismatched. Check #2 here.

    Thanks

  • 0 in reply to sachingurung

    Experiencing the same issue on a fresh installation of Sophos XG.
    The logs shows that the IPS service (Snort) is constantly restarting.

    Setup details
    Software install with firmware image: SW-SFOS_16.05.0_RC-1-098.iso (sha256: c1afd3fe74ff80e30701941d0ce80838153a93d61004362c17c0c065921732ec)

    IPS configuration

    Enable Spoof Prevention: not selected
    Restrict Unknown IP on Trusted MAC: not selected
    Spoof Protection Trusted MAC: no records
    DoS Settings: nothing selected
     
    Attack Type    Source                 Destination
    Applied           Traffic Dropped     Applied     Traffic Dropped
    SYN Flood       No         0             No             0
    UDP Flood       No         0             No             0
    TCP Flood        No         0             No             0
    ICMP Flood      No         0             No             0
    IP Flood           No         0             No             0

    (Active) Firewall rule

    Rule Name: #Default_Network_Policy
    Action: Acccept

    Source
    Source Zones: LAN
    Source Networks and Devices: ANY
    During Scheduled Time: All the time

    Destination & Services
    Destination Zones: WAN
    Destination Networks: ANY
    Services: HTTP, HTTPS

    Identity
    Match known users: not selected

    Malware scanning
    Scan HTTP: not selected
    Decrypt & Scan HTTPS: not selected
    Detect zero-day threats with Sandstorm: not selected
    Scan FTP: not selected

    Advanced
    Intrusion Prevention: None
    Traffic Shaping Policy: None
    Web Policy: None
    Application Control: None

    Rewrite source address (Masquerading)
    Use Outbound Address
    MASQ

    Figure 1: Overview

    Syslog
    Dec 26 10:43:56 (none) user.info kernel: [ 1887.482362] traps: snort[10487] trap invalid opcode ip:677245 sp:7fff9fa24698 error:0 in snort[400000+720000]
    Dec 26 10:43:56 (none) user.info kernel: [ 1887.495792] traps: snort[10491] trap invalid opcode ip:677245 sp:7fff9fa24698 error:0 in snort[400000+720000]
    Dec 26 10:43:56 (none) user.info kernel: [ 1887.857172] traps: snort[10496] trap invalid opcode ip:677245 sp:7fff9fa24698 error:0 in snort[400000+720000]
    Dec 26 10:43:56 (none) user.info kernel: [ 1887.876527] traps: snort[10494] trap invalid opcode ip:66b495 sp:7fff9fa242d0 error:0 in snort[400000+720000]
    Dec 26 10:43:56 (none) user.info kernel: [ 1887.900984] traps: snort[10499] trap invalid opcode ip:677245 sp:7fff9fa24698 error:0 in snort[400000+720000]

    IPS logs
    [Dec 26 10:43:56 :10499]:Commencing packet processing (pid=10499)
    [Dec 26 10:43:56 :10499]:Decoding Raw IP4
    [Dec 26 10:43:56 :8743]:readfd cdata.cpipe[1] for pid 10499 set
    [Dec 26 10:43:56 :8743]:child 10494 dead
    [Dec 26 10:43:56 :8743]:cdata[1].lstatus for pid 10494 set
    [Dec 26 10:43:56 :8743]:1391:My childrens are dieing too fast
    [Dec 26 10:43:56 :10499]:Total preallocated memory : 2036.1094KB
    [Dec 26 10:43:56 :10499]:Max memory alloc by webcat: 3732.1094KB
    *** Caught Term-Signal
    [Dec 26 10:43:56 :8743]:IPS: finally after all childrens grand father is going down
    Snort exiting
    fd 4 size 528384
    size 1022 maxapp 4096 counter 7 bytesize 512
    ERROR[8741]:Dec 26 10:43:57:s_worker.c:1160:csccom_loop:Snort master goes down: state: 1: exited, status=1
    INFO[8741]:Dec 26 10:43:57:s_worker.c:1169:csccom_loop:on error
    /bin/snort: /lib/libcrypto.so.1.0.0: no version information available (required by /bin/snort)

    and the second time:

    IPS
    [Dec 26 10:55:16 :12410]:Decoding Raw IP4
    [Dec 26 10:55:16 :11615]:readfd cdata.cpipe[1] for pid 12410 set
    [Dec 26 10:55:16 :12410]:Total preallocated memory : 2036.1094KB
    [Dec 26 10:55:16 :12410]:Max memory alloc by webcat: 3732.1094KB
    [Dec 26 10:55:16 :11615]:child 12410 dead
    [Dec 26 10:55:16 :11615]:cdata[1].lstatus for pid 12410 set
    [Dec 26 10:55:16 :11615]:1391:My childrens are dieing too fast
    INFO[12408]:Dec 26 10:55:16:daq_pktq.c:747:pktq_daq_acquire:[12408] packet recv contents failure: dlen 0 : 14:Bad address
    *** Caught Term-Signal
    Snort exiting
    fd 4 size 528384
    size 1022 maxapp 4096 counter 7 bytesize 512
    [Dec 26 10:55:18 :11615]:IPS: finally after all childrens grand father is going down
    Snort exiting
    fd 4 size 528384
    size 1022 maxapp 4096 counter 7 bytesize 512
    ERROR[11613]:Dec 26 10:55:19:s_worker.c:1160:csccom_loop:Snort master goes down: state: 1: exited, status=1
    INFO[11613]:Dec 26 10:55:19:s_worker.c:1169:csccom_loop:on error
    /bin/snort: /lib/libcrypto.so.1.0.0: no version information available (required by /bin/snort)

  • 0 in reply to IeM

    Reverting back to SFOS 15.01.0 MR-3 fix this issue.
    After upgrade to firmware SFOS 16.01.1 /SFOS 16.01.2 this issue is observed again.

  • 0 in reply to IeM

    HI All, 

    Could you set the Settings of IPS as follows 

    Default IPS settings

    stream on
    lowmem off
    maxsesbytes 0
    maxpkts 100
    enable_appsignatures on
    http_response_scan_limit 65535

    Run Commands on Console 

    set ips maxsesbytes-settings update 8192
    set ips maxpkts 8

    IPS settings after changes 

    -------------IPS Settings-------------
    stream on
    lowmem off
    maxsesbytes 8192
    maxpkts 8
    enable_appsignatures on
    http_response_scan_limit 65535

    This should help 

  • 0 in reply to IeM

    Hi, I see you experienced the same issue that I am currently experiencing. I am seeing several RST Packets and drops on connections that were previously working with no issues. My question is how do I revert back to SFOS 15.01.0 MR-3 from SFOS 16 without performing a clean install? SF OS 15 is no longer in my repository.

  • 0 in reply to BobbyDigital

    Just turned off UDP flood as well as TCP

    The UDP setting was also affecting the TCP setting. The log showed UDP and TCP IPS issues. I went to investigate why my work secure tunnel kept dying.

    Not happy.

  • 0 in reply to IeM

    Hi IeM,

    Upgrade to the latest available firmware on your XG. If that doesn't help, please provide us details about:

    1. Which XG device do you use? Is that a software appliance?

    2. If software; what is the system specifications?

    3. Have you ever bind CPUs for the IPS instances? Run the command in console: show ips-settings (you can see how many IPS instances are available and how many CPU are used).

    Thanks

  • 0 in reply to sachingurung

    Hi Sachingurung,

    Already running the latest available firmware (to be exactly: SFOS 16.05.1 MR-1).

    Answer on your questions:

    1. XG Firewall. It's a software appliance.

    2. System specs are: memory 6GiB, processor 8 (4 sockets, 2 cores)...

    3. No I have never bind CPUs for the IPS instances.

    Output of the command `show ips-settings`

    -------------IPS Settings-------------                                                       
            stream on                                                                            
            lowmem off                                                                           
            maxsesbytes 0                                                                        
            maxpkts 8                                                                            
            mmap off                                                                             
            enable_appsignatures on                                                              
            http_response_scan_limit  65535                                                      
                                                                                                 
                                                                                                 
    -------------IPS Instances------------                                                       
    IPS CPU                                                                                      
     1  4                                                                                        
     2  5                                                                                           
  • 0 in reply to Aditya Patel

    Changed the IPS settings (maxsesbytes-settings and maxpkts) but it doesn't solve the issue.

  • 0 in reply to IeM

    The fact that this IPS engine issue has not been fixed since version 16 came out is RIDICULOUS! It seems like most people who experience this are running an AMD environment. I'm starting to think Sophos has no inclining to actually fix the issue.

  • 0 in reply to red281gt

    Unknown said:

    The fact that this IPS engine issue has not been fixed since version 16 came out is RIDICULOUS! It seems like most people who experience this are running an AMD environment. I'm starting to think Sophos has no inclining to actually fix the issue.

     

     

    I am co-signing your response. I currently had to revert back to SFOS 15.01.0 MR-3 and I am running fine with no issues. I will not upgrade until I have confirmation that this bug is fixed.

    Secondly, I tested this problem on my AMD PC and the issue came back once upgrading to the lastest 16 GA. Now the interesting part is on my much older Intel based PC upgrading to v.16 I did not experience the IPS flapping issue.

    I believe Matthew Brooks is on to something.

    My disclaimer is I did not add any rules or settings on any of the test boxes as the IPS issue is instant upon upgrading in my previous experiences.

  • +1 in reply to IeM

    Resolved after upgrading to MR4.

Reply Children
No Data