Guest User!

You are not Sophos Staff.

Thread Info
  • State Verified Answer
  • Locked Locked
  • Replies 9 replies
  • Subscribers 28 subscribers
  • Views 3257 views
  • Users 0 members are here
Options
Suggested
This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

XG Home Edition v18 - ReportDB Dead

gavo_nz

Hi,

When I logged into my XG Home Edition tonight I noticed the console had flagged the ReportDB service as dead. I havent logged on since I did the upgrade from SFOS 18.0.4 MR-4 to SFOS 18.0.5 MR-5-Build586 a few weeks ago - at the time of the upgrade I didn't notice a problem after the firewall restarted - so not sure if this is related or not.

Anyway, went through KB-000035777 and it appears that in this state, I should contact the Sophos support team - unfortunately with the home edition, I don't think that's an option. Disk space is all good on the box.

Just wondering if anyone had any thoughts on how to resolve Slight smile

Cheers,



This thread was automatically locked due to age.
  • FormerMember
    0 FormerMember

    Hi ,

    Thank you for reaching out to Sophos Community.

    Request to follow the steps below and share session output here or in DM.

    ==> Login to SSH > 5. Device Management > 3. Advanced Shell

    => Put csc service in debug.

    # csc custom debug

    => Try to start reportdb service manually.

    # service reportdb:start -ds nosync

    => After that run the below commands.

    # df -kh

    # ls -lahr /var/cores

    # tail -30 /log/reportdb.log

    # tail -n 500 /log/csc.log | grep -i "reportdb"

  • 0

    Please follow this link "on-box reporting issues" https://support.sophos.com/support/s/article/KB-000035777?language=en_US and share output of all the commands. 

    In addition, please capture the output of the following commands from the CLI console

    console> system diagnostics show version-info

    console> show report-disk-usage watermark

    console> system diagnostics show disk

    In addition, please capture the output of the following commands from the Advance shell console

    service reportdb:start -ds nosync

    You can check the output of 'less /log/garner.log.0' to see if it shows memory corruption on garner, as below:

    ERROR     [4120435520]: SSL_read failed ret:5
    *** Error in `garner': double free or corruption (!prev): 0xf331bc68 ***
    *** Error in `garner': malloc(): memory corruption: 0xf331c1a0 ***


     

  • 0 in reply to FormerMember

    Thanks Yash, please see output below.

    SFVH_SO01_SFOS 18.0.5 MR-5-Build586# csc custom debug
    SFVH_SO01_SFOS 18.0.5 MR-5-Build586# service reportdb:start -ds nosync
    503 Service Failed
    SFVH_SO01_SFOS 18.0.5 MR-5-Build586# df -kh
    Filesystem Size Used Available Use% Mounted on
    none 235.2M 4.5M 214.3M 2% /
    none 2.9G 28.0K 2.9G 0% /dev
    none 2.9G 35.2M 2.9G 1% /tmp
    none 2.9G 14.6M 2.9G 0% /dev/shm
    /dev/boot 127.7M 49.0M 76.0M 39% /boot
    /dev/mapper/mountconf
    385.4M 74.1M 307.3M 19% /conf
    /dev/content 20.8G 558.1M 20.2G 3% /content
    /dev/var 179.1G 38.3G 140.8G 21% /var
    SFVH_SO01_SFOS 18.0.5 MR-5-Build586# tail -30 /log/reportdb.log
    29193 2021-08-23 21:19:25.063 GMTLOG: database system was shut down at 2021-08-23 21:19:09 GMT
    29192 2021-08-23 21:19:25.067 GMTLOG: database system is ready to accept connections
    29197 2021-08-23 21:19:25.067 GMTLOG: autovacuum launcher started
    29205 2021-08-23 21:19:26.166 GMTFATAL: could not open file "base/16386/11811": No such file or directory
    29206 2021-08-23 21:19:26.168 GMTFATAL: could not open file "base/16386/11811": No such file or directory
    29192 2021-08-23 21:19:26.181 GMTLOG: received fast shutdown request
    29192 2021-08-23 21:19:26.181 GMTLOG: aborting any active transactions
    29197 2021-08-23 21:19:26.181 GMTLOG: autovacuum launcher shutting down
    29194 2021-08-23 21:19:26.181 GMTLOG: shutting down
    29194 2021-08-23 21:19:26.192 GMTLOG: database system is shut down
    29567 2021-08-23 21:21:05.611 GMTLOG: database system was shut down at 2021-08-23 21:19:26 GMT
    29566 2021-08-23 21:21:05.614 GMTLOG: database system is ready to accept connections
    29571 2021-08-23 21:21:05.615 GMTLOG: autovacuum launcher started
    29576 2021-08-23 21:21:06.714 GMTFATAL: could not open file "base/16386/11811": No such file or directory
    29577 2021-08-23 21:21:06.716 GMTFATAL: could not open file "base/16386/11811": No such file or directory
    29566 2021-08-23 21:21:06.729 GMTLOG: received fast shutdown request
    29566 2021-08-23 21:21:06.729 GMTLOG: aborting any active transactions
    29571 2021-08-23 21:21:06.729 GMTLOG: autovacuum launcher shutting down
    29568 2021-08-23 21:21:06.731 GMTLOG: shutting down
    29568 2021-08-23 21:21:06.740 GMTLOG: database system is shut down
    29618 2021-08-23 21:21:21.902 GMTLOG: database system was shut down at 2021-08-23 21:21:06 GMT
    29617 2021-08-23 21:21:21.905 GMTLOG: database system is ready to accept connections
    29622 2021-08-23 21:21:21.905 GMTLOG: autovacuum launcher started
    29626 2021-08-23 21:21:23.004 GMTFATAL: could not open file "base/16386/11811": No such file or directory
    29627 2021-08-23 21:21:23.006 GMTFATAL: could not open file "base/16386/11811": No such file or directory
    29617 2021-08-23 21:21:23.019 GMTLOG: received fast shutdown request
    29617 2021-08-23 21:21:23.019 GMTLOG: aborting any active transactions
    29622 2021-08-23 21:21:23.019 GMTLOG: autovacuum launcher shutting down
    29619 2021-08-23 21:21:23.020 GMTLOG: shutting down
    29619 2021-08-23 21:21:23.029 GMTLOG: database system is shut down
    SFVH_SO01_SFOS 18.0.5 MR-5-Build586# tail -n 500 /log/csc.log |grep -i "reportdb"
    DEBUG Aug 24 09:21:05 [reportdb:1612]: read_packet: read() 52 bytes from listener
    MESSAGE Aug 24 09:21:05 [reportdb:1612]: Toggling log level to: WARNING
    MESSAGE Aug 24 09:21:05 [reportdb:1612]: {"reportdb":{"method":"service","name":"reportdb:start","version":"1.2","type":"text","length":0}}
    ERROR Aug 24 09:21:06 [reportdb:1612]: execute_prepare_query:DB handle returned from perl is not OK.
    ERROR Aug 24 09:21:06 [reportdb:1612]: get_query_status: DB has returned error code: 1
    ERROR Aug 24 09:21:06 [reportdb:1612]: get_query_status:Query Error: FATAL: could not open file "base/16386/11811": No such file or directory
    ERROR Aug 24 09:21:06 [reportdb:1612]: csc_prep_query: execute_prepare_query failed for SELECT txid_current().
    ERROR Aug 24 09:21:06 [reportdb:1612]: execute_prepare_query:DB handle returned from perl is not OK.
    ERROR Aug 24 09:21:06 [reportdb:1612]: get_query_status: DB has returned error code: 1
    ERROR Aug 24 09:21:06 [reportdb:1612]: get_query_status:Query Error: FATAL: could not open file "base/16386/11811": No such file or directory
    CRITICAL Aug 24 09:21:06 [reportdb:1612]: csc_prep_query: execute_prepare_query failed for Execute Query.
    ERROR Aug 24 09:21:06 [reportdb:1612]: do_prep_query: Failed PREPSTMT: 'alter database iviewdb set timezone to ?'
    ERROR Aug 24 09:21:06 [reportdb:1612]: do_real_start: after_start failed. aborting!
    MESSAGE Aug 24 09:21:06 [reportdb:1612]: do_stop(): status = RUNNING
    MESSAGE Aug 24 09:21:06 [reportdb:1612]: do_waitpid: Service with pid 29566, exited with status 0.
    ERROR Aug 24 09:21:06 [reportdb:1612]: do_start: service failed to start
    ERROR Aug 24 09:21:23 [reportdb:1612]: execute_prepare_query:DB handle returned from perl is not OK.
    ERROR Aug 24 09:21:23 [reportdb:1612]: get_query_status: DB has returned error code: 1
    ERROR Aug 24 09:21:23 [reportdb:1612]: get_query_status:Query Error: FATAL: could not open file "base/16386/11811": No such file or directory
    ERROR Aug 24 09:21:23 [reportdb:1612]: csc_prep_query: execute_prepare_query failed for SELECT txid_current().
    ERROR Aug 24 09:21:23 [reportdb:1612]: execute_prepare_query:DB handle returned from perl is not OK.
    ERROR Aug 24 09:21:23 [reportdb:1612]: get_query_status: DB has returned error code: 1
    ERROR Aug 24 09:21:23 [reportdb:1612]: get_query_status:Query Error: FATAL: could not open file "base/16386/11811": No such file or directory
    CRITICAL Aug 24 09:21:23 [reportdb:1612]: csc_prep_query: execute_prepare_query failed for Execute Query.
    ERROR Aug 24 09:21:23 [reportdb:1612]: do_prep_query: Failed PREPSTMT: 'alter database iviewdb set timezone to ?'
    ERROR Aug 24 09:21:23 [reportdb:1612]: do_real_start: after_start failed. aborting!
    MESSAGE Aug 24 09:21:23 [reportdb:1612]: do_stop(): status = RUNNING
    MESSAGE Aug 24 09:21:23 [reportdb:1612]: do_waitpid: Service with pid 29617, exited with status 0.
    ERROR Aug 24 09:21:23 [reportdb:1612]: do_start: service failed to start

  • 0 in reply to NM_1987

    Thanks Nilesh, probably best if I only follow one troubleshooting process at a time - so I'll just see how I get on with Yash before I give your suggestions a try :) Cheers.

  • 0 in reply to gavo_nz

    Thanks ;

    From the logs, it looks like reportDB migration got failed.

    Suspected logs:

    SFVH_SO01_SFOS 18.0.5 MR-5-Build586# service reportdb:start -ds nosync
    503 Service Failed

    ERROR Aug 24 09:21:06 [reportdb:1612]: get_query_status:Query Error: FATAL: could not open file "base/16386/11811": No such file or directory
    CRITICAL Aug 24 09:21:06 [reportdb:1612]: csc_prep_query: execute_prepare_query failed for Execute Query.
    ERROR Aug 24 09:21:06 [reportdb:1612]: do_prep_query: Failed PREPSTMT: 'alter database iviewdb set timezone to ?'


    Please capture the output of the following commands from the CLI console

    console> system diagnostics show version-info

    console> show report-disk-usage watermark

    console> system diagnostics show disk

    You can check the output of 'less /log/garner.log.0' to see if it shows memory corruption on garner, as below:

    ERROR     [4120435520]: SSL_read failed ret:5
    *** Error in `garner': double free or corruption (!prev): 0xf331bc68 ***
    *** Error in `garner': malloc(): memory corruption: 0xf331c1a0 ***

  • 0 in reply to NM_1987

    Thanks ,

    I cant see any of those messages in the /log/garner.log.0 file. Also, here is the output from those other commands.

    Cheers,

    console> system diagnostics show version-info

    Serial Number: <removed>
    Device-Id: 4a32c96c-73b2-49f9-859d-14edf63d786b
    Appliance Model: SFVH
    Firmware Version: SFOS 18.0.5 MR-5-Build586
    Firmware Build: 586
    Firmware Loader version:
    HW version: SO01
    Config DB version: 18.034
    Signature DB version: 18.034
    Report DB version: 18.031
    Webcat Signature version: Not Available
    Web Proxy version: compiled
    SMTP Proxy version: 1.0
    POP/IMAP Proxy version: 1.0.0.3.4
    Logging Daemon version: 0.0.0.17
    AP Firmware: 11.0.005
    ATP: 1.0.0371
    Avira AV: 1.0.417338
    Authentication Clients: 1.0.0019
    Geoip ip2country DB: 2.0.005
    IPS and Application signatures: 18.18.48
    Sophos Connect Clients: 2.1.001
    odt: -
    RED Firmware: 2.0.016
    Sophos AV: 1.0.17062
    SSLVPN Clients: 1.0.009
    Hot Fix version: 3

    console> show report-disk-usage watermark
    Lower watermark percentage for report partition is 80%

    console> system diagnostics show disk
    Partition Utilization(%)
    ===============================
    configuration 19%
    content 3%
    report 21%

  • 0 in reply to gavo_nz

    Hi,

    pease edit your post and remove your serial number.

    Ian

  • +1 in reply to gavo_nz

    Thanks for the logs and requested information .

    Report Database has not migrated to latest one which is 18.034.

    Config DB version: 18.034
    Signature DB version: 18.034
    Report DB version: 18.031

    As report partition is 21% only, please flush the Report partition to resolve this issue.

    Flushing device reports from CLI : https://support.sophos.com/support/s/article/KB-000035779?language=en_US

    There will be network outage of 15-20 minutes while formating the report partition.

    Hope this helps!!

  • 0 in reply to NM_1987

    Many thanks !!

    That did the trick and everything seems happy again Slight smile

    Thanks again for your help, it is much appreciated.