Thread Info
  • State Suggested Answer
  • +3 person also asked this people also asked this
  • Locked Locked
  • Replies 20 replies
  • Answers 2 answers
  • Subscribers 34 subscribers
  • Views 37253 views
  • Users 0 members are here
Options
Suggested
This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

How to find cause of strange reboots?

Brad Shannon

Hi folks, appreciate your help in advance.

 

I have an XG85 running SFOS 17.0.2 MR-2.

It is set up to send logs to a Syslog server I set up just yesterday. [https://i.imgur.com/BoNeNsB.png] [https://i.imgur.com/0h1MnvD.png]

That server is receiving some logs. [https://imgur.com/TRA2qDl]

 

Twice in the last 48 hours, the device has rebooted itself during times of high usage. What log settings should I configure to see what could be causing this, in case it happens again?

 

(Also, maybe it's related to SFOS 17?)

 

2/27/18 UPDATE:

  • updated to 17.0.5, same reboot problem.
  • replaced the hardware, restored settings to replacement, same reboot problem.
  • tried factory reset and manually re-entering the settings, same reboot problem.

I have a replacement non-Sophos FW en route. However, I got some interesting data from the COM port just now, which I'll add as a new reply below.



This thread was automatically locked due to age.
  • 0

    Possible causes are

    1/. failing disk drive

    2/. running out of disk space under load eg swapping too much.

    3/. how many users, how many rules, what is the base memory and cpu use?

    4/. have you logged a fault with Sophos support?

    Ian

  • 0 in reply to rfcat_vk

    Thanks, Ian!

     

    1. I have not added a disk drive to the device. Storage is internal/default.
    2. As far as I know, there's no issue of disk space. Should I monitor internal storage somehow?
    3. No users set up -- my admin account only. Only one firewall rule: allow all from anywhere to anywhere (temporary -- was troubleshooting unrelated issue, haven't set up new rules). CPU usage hovers around 35%. Memory hovers around 85%. Currently 260 "live connections", 55 "sessions".
    4. Not yet! I wanted to do some research first.

    Thanks for thinking through this with me.

    It sounds like the rebooting is strange, as I suspected. But it also sounds like there isn't a clear way to diagnose the problem if it happens again. Is there any data I should be collecting that would help identify the cause?

  • 0 in reply to Brad Shannon

    Hi Brad,

    please remember this device only has an 8gb ssd. there is another thread with someone having issues with an XG85 and returning it.

    Ian

  • 0 in reply to rfcat_vk

    Here's disk usage on the device. It looks to me like there's plenty of space available on each file system, but I don't know for sure.

    XG85_XN01_SFOS 17.0.2 MR-2# df -h

    Filesystem                Size      Used Available Use% Mounted on

    rootfs                  358.8M      2.5M    333.4M   1% /

    df: /newroot: No such file or directory

    df: /newroot/dev: No such file or directory

    df: /newrootrw: No such file or directory

    none                    358.8M      2.5M    333.4M   1% /

    none                    959.0M     20.0K    959.0M   0% /dev

    none                     96.0M     23.3M     72.7M  24% /tmp

    none                    959.0M     12.5M    946.4M   1% /dev/shm

    /dev/conf               385.4M     58.2M    327.1M  15% /conf

    /dev/content              5.0G    780.1M      4.2G  15% /content

  • 0

    The firewall just rebooted again. I don't know why. I don't know how to find out why.

     

    This is a work-critical device malfunctioning with no evidence of any problem on the network, no evidence of any issues that would affect hardware (eg, power outage), and extremely minimal use of the firewall's advertised capabilities.

     

    I'm now going to open a support ticket with Sophos and attempt to replace this device with a firewall that does not exhibit unpredictable behavior.

  • 0 in reply to Brad Shannon

    Hi Brad,

    In my experience, the reason for restarts is too small RAM memory in XG85 (2GB). Restarts occur when the XG85 performs an automatic update of the IPS patterns and needs more RAM. What to do ? (1) Turn off automatic pattern updates and do it manually or (2) Turn off unused processes and functions, eventually turn off ATP and IPS.

    Regards
    Jan

  • 0

    Here is my experience, I have had 2 XG85W in the last 3 weeks. Countless restarts on its own and finally Sophos has told me that that the X85 is most likely going to be dropped due its lack of resources especially with the latest OS. I am returning the device and going to try the XG105W.

  • 0 in reply to SophosStorm

    My customer is having the exact problem. Are the going to replace it for you with the 105 ? I must solve this problem too ...

     

    Thx.

  • 0

    This was printed to the COM port:

     

    [433020.181445] Kernel panic - not syncing: Out of memory: compulsory panic_on_oom is enabled
    [433020.181445]
    [433020.213879] CPU: 1 PID: 16933 Comm: lcdd Tainted: G O 3.14.22-Aum #1
    [433020.237985] Hardware name: Sophos XG/Aptio CRB, BIOS 5.6.5 03/20/2017
    [433020.259508] 000201da 8140abf2 00000002 81408bed 814c5d5c 8166ec40 00000002 000201da
    [433020.285376] 00000000 00000000 8108dc4f 814c95e8 814c8020 e2cee180 000201da 8108de14
    [433020.311241] 00000000 00000000 000201da 00000000 8157b75c 00077dfb 00000000 00000001
    [433020.337107] Call Trace:
    [433020.345444] [<8140abf2>] ? dump_stack+0x3e/0x4e
    [433020.360946] [<81408bed>] ? panic+0x7d/0x15f
    [433020.375301] [<8108dc4f>] ? check_panic_on_oom+0x4f/0x50
    [433020.393098] [<8108de14>] ? out_of_memory+0x94/0x2d0
    [433020.409749] [<81091999>] ? __alloc_pages_nodemask+0x7b9/0x7d0
    [433020.429269] [<8108c6d6>] ? filemap_fault+0x196/0x3b0
    [433020.446206] [<810a4c79>] ? __do_fault+0x59/0x3e0
    [433020.461996] [<810a7821>] ? handle_mm_fault+0x161/0x730
    [433020.479509] [<8104d6c4>] ? lock_hrtimer_base.isra.29+0x14/0x30
    [433020.499316] [<8102cd9e>] ? __do_page_fault+0xfe/0x410
    [433020.516540] [<8104d630>] ? hrtimer_get_res+0x40/0x40
    [433020.533476] [<8104dfee>] ? SyS_nanosleep+0x4e/0x60
    [433020.549841] [<8102d1e0>] ? vmalloc_sync_all+0x130/0x130
    [433020.567640] [<8140efac>] ? error_code+0x30/0x38
    [433020.583144] [<8102d1e0>] ? vmalloc_sync_all+0x130/0x130
    [433020.600949] Kernel Offset: 0x0 from 0x81000000 (relocation range: 0x80000000-0xf7bfdfff)
    [433020.627932] Rebooting in 3 seconds..
    [433023.673641] ACPI MEMORY or I/O RESET_REG.
    $▒

     

     

    So, the device is running out of memory (I'm assuming RAM), while running a config changed minimally from factory-default, on latest firmware. So, the firmware has a memory leak bug, or it just uses more memory than is available on this device.

     

    Sophos should not have rolled out a firmware update that breaks their product.