Guest User!

You are not Sophos Staff.

Thread Info
  • State Suggested Answer
  • Locked Locked
  • Replies 12 replies
  • Answers 1 answer
  • Subscribers 10 subscribers
  • Views 13348 views
  • Users 0 members are here
Options
Suggested
This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

Linux / PostgreSQL / on-access scan resulting in "Operation not permitted" errors while using "talpa"

edrevpac

Hello team,

We have Sophos AV installed on a few CentOS 7 machines and it seems to be causing some errors when trying to export the PostgreSQL data. The issue is not systematic and seems to happen randomly a few times a week on each VM, but is causing our database backups to fail when they occur. This issue started happening on the day Sophos AV was installed and affects only PostgreSQL.

I have added the PostgreSQL ../base* and ../global* paths to the exclusion list, but this appears to have had no effect.

We are using "talpa" instead of "fanotify" based on the information from here community.sophos.com/.../118216. We are not currently using CIFS/SMB on any of out PostgreSQL linux machines.

Please assist.


OS: CentOS 7
Kernel: 3.10.0-862.11.6.el7.x86_64
AV version: Sophos Anti-Virus = 9.15.0, Build Revision = 2767612
PostgreSQL version: 9.6

Savlog:
2018-09-04 11:02:59: savd.daemon Sophos Anti-Virus daemon started.
2018-09-04 11:03:06: savd.daemon On-access scanning enabled using talpa.

PostgreSQL error:
pg_dump: archiver (db) connection to database "postgres" failed: FATAL: could not open file "global/2676": Operation not permitted
pg_dumpall: pg_dump failed on database "postgres", exiting


Regards,
Radu



This thread was automatically locked due to age.
Parents
  • 0

    Hello Radu,

    can't say if the exclusions would help but how did you add[] the PostgreSQL [...] paths to the exclusion list, i.e. what exactly did you specify? The patterns you've posted definitely don't do what you expect. Please see also chapter 15.2 in the SAV for Linux Configuration Guide.

    Christian

  • 0 in reply to QC

    Hi Christian,

     

    Thank you for replying so fast :)

     

    The commands used to exclude paths are as per below:

    /savconfig add ExcludeFilePaths /var/lib/pgsql/9.6/data/base/*

    /savconfig add ExcludeFilePaths /var/lib/pgsql/9.6/data/global/*

    /etc/init.d/sav-protect restart

     

    Also, when I use "/savconfig get ExcludeFilePaths" I do see all the folder and subfolders of both /global and /base in the list.

     

    Regards,

    Radu

  • 0 in reply to edrevpac

    Hello Radu,

    thought as much ...

    ExcludeFilePaths doesn't "need" a wildcard pattern. A directory is indicated by the trailing slash, that's all. If you need to use wildcards then it's ExcludeFileOnGlob.

    I do see all the folder and subfolders - this is because you didn't quote the pattern/expression, the shell expands it and passes the list of current files/directories to savconfig.

    Christian

  • 0 in reply to QC

    I see, so for my usecase with dynamic subfolder structures, I need to remove the current exclusion list, and add it again but only using:

     

    /savconfig add ExcludeFilePaths /var/lib/pgsql/9.6/data/base/

    /savconfig add ExcludeFilePaths /var/lib/pgsql/9.6/data/global/

     

    Is that correct?

     

    Radu

  • 0 in reply to edrevpac

    Hello Radu,

    as far as I can tell this should achieve what you want.

    Christian

  • 0 in reply to QC

    Hi Christian,

     

    I have made the changes on our PostgreSQL servers. Will monitor for the next few days and mark this thread as "Answered" if no more errors show up.

     

    Thank you for your help with this.

     

    Regards,

    Radu

  • 0 in reply to edrevpac

    Hello again,

     

    I'm afraid that the issue has reoccurred today.

    The on-access scan seems to be ignoring the exclusion and still locking the files for a short amount of time when they are accessed by the pg_dump utility.

     

    Can you perhaps suggest some other solution?

     

    Regards,

    Radu

  • 0 in reply to edrevpac

    Hello Radu,

    I'm ignorant of the inner workings of Talpa and how they could conflict with pg_dump. And I can't even guess which Operation is not permitted. As for ignoring the exclusion - before an exclusion can be honoured the path has to be determined so there might be some interception (necessary).

    Perhaps can give some advice.

    Christian

Reply
  • 0 in reply to edrevpac

    Hello Radu,

    I'm ignorant of the inner workings of Talpa and how they could conflict with pg_dump. And I can't even guess which Operation is not permitted. As for ignoring the exclusion - before an exclusion can be honoured the path has to be determined so there might be some interception (necessary).

    Perhaps can give some advice.

    Christian

Children
  • 0 in reply to QC

    Hi,

     

    The /var/log/syslog or /var/log/kern.log might be able to give some idea of what Talpa is blocking.

     

    Operation Not Permitted is the errno that Talpa returns if we are blocking access to a file because it is infected.

     

    Thanks,

    Douglas. 

  • 0 in reply to DouglasLeeder

    Hello Douglas,

     

    The only difference I could find was the below, from a failed attempt and a successful attempt. Besides the bolded log entry, I do not spot any other differences.

    Looking forward to your feedback on this.

     

    FAILED
     
    /var/log/messages

    Sep 12 10:20:38 <hostname> systemd: Starting "Sophos Anti-Virus update"…

    Sep 12 10:20:56 <hostname> savd: update.updated: Updating from versions - SAV: 9.15.0, Engine: 3.72.1, Data: 5.54

    Sep 12 10:20:56 <hostname> savd: update.updated: Updating Sophos Anti-Virus....#012Updating SAVScan on-demand scanner#012Updating Virus Engine and Data#012Updating Manifest#012Update completed.

    Sep 12 10:20:56 <hostname> savd: update.updated: Updated to versions - SAV: 9.15.0, Engine: 3.72.1, Data: 5.54

    Sep 12 10:20:56 <hostname> savd: update.updated: Successfully updated Sophos Anti-Virus from sdds:SOPHOS

    Sep 12 10:20:56 <hostname> systemd: Started "Sophos Anti-Virus update".

    Sep 12 10:21:17 <hostname> systemd-logind: New session 1340 of user <user>.

    Sep 12 10:21:18 <hostname> su: (to postgres) <user> on none

    Sep 12 10:21:24 <hostname> systemd-logind: Removed session 1340.

     

    /var/log/secure

    Sep 12 10:21:17 <hostname> sshd[31153]: Connection from <remote_ip> port 51868 on <local_ip> port 22

    Sep 12 10:21:17 <hostname> sshd[31153]: Accepted publickey for <user> from <remote_ip> port 51868 ssh2: RSA SHA256:<rsa_key>

    Sep 12 10:21:17 <hostname> sshd[31153]: pam_unix(sshd:session): session opened for user <user> by (uid=0)

    Sep 12 10:21:17 <hostname> sshd[31153]: User child is on pid 31158

    Sep 12 10:21:17 <hostname> sshd[31158]: Starting session: command for <user> from <remote_ip> port 51868 id 0

    Sep 12 10:21:17 <hostname> sudo:   <user> : TTY=unknown ; PWD=/home/<user> ; USER=root ; COMMAND=/usr/local/bin/cron/postgres_backup.sh <secret_key><user>

    Sep 12 10:21:18 <hostname> su: pam_unix(su-l:session): session opened for user postgres by (uid=0)

    Sep 12 10:21:22 <hostname> su: pam_unix(su-l:session): session closed for user postgres

    Sep 12 10:21:24 <hostname> sshd[31158]: Received disconnect from <remote_ip> port 51868:11: disconnected by user

    Sep 12 10:21:24 <hostname> sshd[31158]: Disconnected from <remote_ip> port 51868

    Sep 12 10:21:24 <hostname> sshd[31153]: pam_unix(sshd:session): session closed for user <user>

     

     

    SUCCESS

    /var/log/messages

    Sep 12 11:20:38 <hostname> systemd: Starting "Sophos Anti-Virus update"...

    Sep 12 11:20:48 <hostname> savd: update.check: Successfully updated Sophos Anti-Virus from sdds:SOPHOS

    Sep 12 11:20:48 <hostname> systemd: Started "Sophos Anti-Virus update".

    Sep 12 11:21:22 <hostname> systemd-logind: New session 1343 of user <user>.

    Sep 12 11:21:23 <hostname> su: (to postgres) <user> on none

    Sep 12 11:21:28 <hostname> systemd-logind: Removed session 1343.

     
     

    /var/log/secure

    Sep 12 11:21:22 <hostname> sshd[2304]: Connection from <remote_ip> port 51934 on <local_ip> port 22

    Sep 12 11:21:22 <hostname> sshd[2304]: Accepted publickey for <user> from <remote_ip> port 51934 ssh2: RSA SHA256:<rsa_key>

    Sep 12 11:21:23 <hostname> sshd[2304]: pam_unix(sshd:session): session opened for user <user> by (uid=0)

    Sep 12 11:21:23 <hostname> sshd[2304]: User child is on pid 2309

    Sep 12 11:21:23 <hostname> sshd[2309]: Starting session: command for <user> from <remote_ip> port 51934 id 0

    Sep 12 11:21:23 <hostname> sudo:   <user> : TTY=unknown ; PWD=/home/<user> ; USER=root ; COMMAND=/usr/local/bin/cron/postgres_backup.sh <secret_key><user>

    Sep 12 11:21:23 <hostname> su: pam_unix(su-l:session): session opened for user postgres by (uid=0)

    Sep 12 11:21:26 <hostname> su: pam_unix(su-l:session): session closed for user postgres

    Sep 12 11:21:28 <hostname> sshd[2309]: Received disconnect from <remote_ip> port 51934:11: disconnected by user

    Sep 12 11:21:28 <hostname> sshd[2309]: Disconnected from <remote_ip> port 51934

    Sep 12 11:21:28 <hostname> sshd[2304]: pam_unix(sshd:session): session closed for user <user>

     

     

    Regards,

    Radu

     

  • 0 in reply to edrevpac

    I'm afraid none of those are the talpa/kernel lines that are required.

     

    There should be some log file that contains talpa and kernel syslog lines, that should help explain why access to the file is being blocked.

  • 0 in reply to DouglasLeeder

    Hello Douglas,

     

    I have instituted kernel logging on 2 machines, but as luck would have it, the error did not reappear on them.

    I did however occur on another PostgreSQL machine, and this time It took down the whole database service.

     

    For the time being we have opted to disable on-access scanning completely on all PostgreSQL machines as it is starting to affect our operation.

     

    I will continue to perform some localized testing and will return with updates, if any useful information comes out of it.

     

    Thank you for your help with this.

     

    Regards,

    Radu

Unfiltered HTML