Guest User!

You are not Sophos Staff.

Thread Info
  • State Suggested Answer
  • Locked Locked
  • Replies 17 replies
  • Answers 2 answers
  • Subscribers 29 subscribers
  • Views 2003 views
  • Users 0 members are here
Options
Suggested
This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

XG115W daily crashes after v18 MR5 upgrade Build 586

BrentMagnant

We updated XG115W from 17.5 MR15 to v18 MR5 on Sunday. Since that upgrade the firewall has crashed every evening. The entire system stops responding.

- VPN goes down
- WebAdmin page does not respond.
- Firewall logs stop logging any activity in any log
- All internet access is down

After a reboot, the system acts normally for a period of time.

Firewall has an SSL site to site VPN and a basic rule set to allow most traffic to the internet with ATP and IPS.

It is a fairly basic setup that was working fine before the v18 upgrade.

They are a public utility and daily outages are not really acceptable.

Anyone got any ideas?



This thread was automatically locked due to age.
  • 0

    Hello Brent,

    Thank you for contacting the Sophos Community, sorry to hear you are having issues with your device.

    If you have a case open with Support could you please share the Case ID with me, so I can follow up, if you haven't please open one and share the Case ID with me.

    Can you please submit the following files:

    csc.log, applog.log, syslog.log, msync.log and networkd.log

    Memory and CPU graph and all this detail with exact date and time when issue observed.

    If you have any log under /var/cores, please submit the output of the command.

    Also the output of this command:  grep 'NMI\|backtrace' /log/syslog.log

    Additionally please run the following command, to disable Firewall-Acceleration and monitor if the issue happens again.
    console> system firewall-acceleration disable
    To see if the Firewall Acceleration is enabled, please run
    console> system firewall-acceleration show

    Note: If disabling Firewall Acceleration, does temporarily resolve the problem, this still needs to be investigated. 

    I'd also suggest you set up a console connection to capture the next restart event if it ever happens. 

    Regards,

  • 0 in reply to emmosophos

    Sophos ticket# is 04038376, but I have had no response since opening it.

    Firewall acceleration was enabled but has now been disabled:

    console> system firewall-acceleration show
    Firewall Acceleration is Disabled.

    Under  /var/cores

    . .. core.sslvpn

    The firewall became unresponsive @  2021-05-28 01:00:58 and was power cycled @ 2021-05-28 07:45:20

    grep 'NMI\|backtrace' /log/syslog.log

    May 25 07:31:22 (none) user.info kernel: [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x01] high level lint[0x1])
    May 25 07:31:22 (none) user.info kernel: [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x02] high level lint[0x1])
    May 25 07:31:22 (none) user.info kernel: [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x03] high level lint[0x1])
    May 25 07:31:22 (none) user.info kernel: [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x04] high level lint[0x1])
    May 25 07:31:22 (none) user.info kernel: [ 0.070537] NMI watchdog: Enabled. Permanently consumes one hw-PMU counter.
    May 25 07:48:10 (none) user.info kernel: [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x01] high level lint[0x1])
    May 25 07:48:10 (none) user.info kernel: [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x02] high level lint[0x1])
    May 25 07:48:10 (none) user.info kernel: [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x03] high level lint[0x1])
    May 25 07:48:10 (none) user.info kernel: [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x04] high level lint[0x1])
    May 25 07:48:10 (none) user.info kernel: [ 0.070526] NMI watchdog: Enabled. Permanently consumes one hw-PMU counter.
    May 25 09:08:48 (none) user.info kernel: [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x01] high level lint[0x1])
    May 25 09:08:48 (none) user.info kernel: [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x02] high level lint[0x1])
    May 25 09:08:48 (none) user.info kernel: [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x03] high level lint[0x1])
    May 25 09:08:48 (none) user.info kernel: [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x04] high level lint[0x1])
    May 25 09:08:48 (none) user.info kernel: [ 0.070527] NMI watchdog: Enabled. Permanently consumes one hw-PMU counter.
    May 26 08:35:44 (none) user.info kernel: [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x01] high level lint[0x1])
    May 26 08:35:44 (none) user.info kernel: [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x02] high level lint[0x1])
    May 26 08:35:44 (none) user.info kernel: [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x03] high level lint[0x1])
    May 26 08:35:44 (none) user.info kernel: [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x04] high level lint[0x1])
    May 26 08:35:44 (none) user.info kernel: [ 0.070518] NMI watchdog: Enabled. Permanently consumes one hw-PMU counter.
    May 27 07:21:02 (none) user.info kernel: [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x01] high level lint[0x1])
    May 27 07:21:02 (none) user.info kernel: [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x02] high level lint[0x1])
    May 27 07:21:02 (none) user.info kernel: [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x03] high level lint[0x1])
    May 27 07:21:02 (none) user.info kernel: [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x04] high level lint[0x1])
    May 27 07:21:02 (none) user.info kernel: [ 0.070526] NMI watchdog: Enabled. Permanently consumes one hw-PMU counter.
    May 27 15:05:31 (none) user.info kernel: [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x01] high level lint[0x1])
    May 27 15:05:31 (none) user.info kernel: [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x02] high level lint[0x1])
    May 27 15:05:31 (none) user.info kernel: [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x03] high level lint[0x1])
    May 27 15:05:31 (none) user.info kernel: [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x04] high level lint[0x1])
    May 27 15:05:31 (none) user.info kernel: [ 0.070527] NMI watchdog: Enabled. Permanently consumes one hw-PMU counter.
    May 27 17:20:52 (none) user.info kernel: [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x01] high level lint[0x1])
    May 27 17:20:52 (none) user.info kernel: [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x02] high level lint[0x1])
    May 27 17:20:52 (none) user.info kernel: [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x03] high level lint[0x1])
    May 27 17:20:52 (none) user.info kernel: [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x04] high level lint[0x1])
    May 27 17:20:52 (none) user.info kernel: [ 0.070529] NMI watchdog: Enabled. Permanently consumes one hw-PMU counter.
    May 28 07:43:26 (none) user.info kernel: [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x01] high level lint[0x1])
    May 28 07:43:26 (none) user.info kernel: [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x02] high level lint[0x1])
    May 28 07:43:26 (none) user.info kernel: [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x03] high level lint[0x1])
    May 28 07:43:26 (none) user.info kernel: [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x04] high level lint[0x1])
    May 28 07:43:26 (none) user.info kernel: [ 0.070540] NMI watchdog: Enabled. Permanently consumes one hw-PMU counter.

  • 0 in reply to emmosophos

    It looks like the logs might be too large to attach.

    How can I submit them?

  • 0 in reply to BrentMagnant

    Hello Brent,

    Thank you for the screenshots and some info on the Logs.

    You can upload the files to our FTP, you should be able to see the FTP Credentials in the case now.

    I also left a note for the engineer to check, but if you could just update the ticket with the REVision number of your appliance, that should help.

    Regards,

  • 0 in reply to emmosophos

    Uploaded the logs. It looks like one of them kept running while the others all stopped during the outage. So it wasn't entirely crashed... just mostly crashed.  Slight smile

    I don't know the REV #, I need to get that of the chassis right?

  • 0 in reply to BrentMagnant

    Hello Brent,

    Thank you for the follow-up.

    You can click the link I added on the previous Post, it is on Blue letters "Revision number of your appliance" you just need the first  6 letters of the S/N of your device to find it the REV on that KB, or yes on the device itself you can find it.

    Regards,

  • 0 in reply to emmosophos

    Thanks, I didn't notice that was a link....  IT is a REV 3 and I added that to the ticket! 

  • 0 in reply to emmosophos

    Support asked for a copy of cores.sslvpn from /var/cores but I am unable to transfer that file, I don't have permissions. Thoughts?

  • 0 in reply to BrentMagnant

    How big is this file? Copy it to /tmp/ and use (p)scp to copy it from there. 

  • 0 in reply to LuCar Toni

    I copied it to temp, but I had to CHMOD before I could download it... Thanks for the idea to make a copy!  :)