Guest User!

You are not Sophos Staff.

Thread Info
  • State Verified Answer
  • +1 person also asked this people also asked this
  • Locked Locked
  • Replies 3 replies
  • Subscribers 9 subscribers
  • Views 8784 views
  • Users 0 members are here
Options
Suggested
This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

Sophos Server Protection on Hyper-V VM with Azure Site Recovery replication - generating large amounts of replication traffic

ITOps

Hi,

We're experiencing a replication issue which appears to be due to Sophos Endpoint, with one VM in particular causing concern beyond the others.

As background, our platform is a Server 2016 Hyper-V cluster hosting 2008 R2, 2012 R2 and 2016 VMs.  All (30+) business-critical VMs are replicating to Azure Site Recovery (ASR) over a 100Mb/s MPLS, and the system has been in and working reliably for just under a year although we've only recently (last week) completed the transition from McAfee Endpoint Security to Sophos Endpoint.

The issue is that we're now seeing a lot more replication traffic than previously, with a variety of VMs hosting different roles (SQL, IIS, file server, Exchange CAS and DAG) affected.  After some investigation I believe I've traced the cause of the problem to the temporary files Sophos appears to constantly update, these files are located in C:\ProgramData\Sophos\Sophos Anti-Virus\Temp, which contains files suffixed with .$$$

If I monitor performance on the affected VMs, these files are being written to at quite a high rate, between 500KB-20MB/s per each instance of "SavService.exe".  Most worryingly this activity continues even when all AV components are disabled in Sophos using the "Override Sophos Central Policy". 

The majority of VM replicas are just about managing to stay current, however our primary file server is gradually falling further and further behind (it's built up around 100GB backlog since Saturday) and it looks like ultimately it'll require resync and I'll likely have to remove Sophos in order to recover replication.  This fileserver previously generated less than 10GB of change per day, it's now generating about 20GB an hour.  The file server VM is Server 2012 R2, using Storage Spaces and deduplication (backgroup dedupe is disabled, schedule dedupe job runs at 2am so it isn't running during the day and I've excluded the System Volume Information folders from Sophos scanning).

From reading around it seems these temporary files are supposed to be used when Sophos is scanning inside archives and such, so it's doubly puzzling why the build up continues when Sophos is disabled.

Has anyone come across this or anything similar?  Any ideas on how to stop it happening or reduce the impact (other than removing Sophos)?  If this were a new VM to be replicated I'd simply setup a separate un-replicated vDisk but that's not an option with ASR without restaging the whole 1.8TB VM.

Thanks in advance for any assistance.

Dean.

 

*ETA* - Sure enough the file server VM has just gone into "resync" status.



This thread was automatically locked due to age.
Parents
  • +1

    Hello,

    The files you are seeing are generated by Sophos when scanning particular files (over a certain size, type, etc); they can be removed safely but they should be removed when the scan of the file is complete. 

    You could manually remove them (if you havent already) and then see if they return. 

    1. Stop the Sophos Anti-Virus service (SavService.exe).
    2. Delete the $$$ files.
    3. Restart the Sophos Anti-Virus service

    If they return, you might run Process Monitor to see which other process is trying to access the files ($$$) and thus preventing Sophos from completing its action

  • 0 in reply to StephenMcKay

    Hi Stephen,

     

    Thanks very much for the reply, and the advice.  The issue isn't so much that the files are created, it's the constant writing to them that creates the replication workload but I think from what you've said above this isn't expected behaviour.  An odd thing I noticed after posting was that the temporary files were all zero bytes in size despite the constant writes by SAVService.  I'd seen this previously but presumed it was due to the habit of Windows to display files still being written to as zero bytes in size, but viewing the file/folder properties also showed zero bytes consumed.

    I restarted SAVService.exe yesterday so I could resync the replica, and monitored after restarting - the constant writes to the temporary files have ceased.  They've also stopped of their own accord on the other servers too. This leads me to believe the problem may stem from scheduled scans, as all the affected VMs ran one over the weekend.  I'll test and monitor, running ProcMon if the issue reoccurs to see if anything else is locking out the files.

     

    I'll report back when I have any information.

    Best regards,

     

    Dean

     

     

Reply
  • 0 in reply to StephenMcKay

    Hi Stephen,

     

    Thanks very much for the reply, and the advice.  The issue isn't so much that the files are created, it's the constant writing to them that creates the replication workload but I think from what you've said above this isn't expected behaviour.  An odd thing I noticed after posting was that the temporary files were all zero bytes in size despite the constant writes by SAVService.  I'd seen this previously but presumed it was due to the habit of Windows to display files still being written to as zero bytes in size, but viewing the file/folder properties also showed zero bytes consumed.

    I restarted SAVService.exe yesterday so I could resync the replica, and monitored after restarting - the constant writes to the temporary files have ceased.  They've also stopped of their own accord on the other servers too. This leads me to believe the problem may stem from scheduled scans, as all the affected VMs ran one over the weekend.  I'll test and monitor, running ProcMon if the issue reoccurs to see if anything else is locking out the files.

     

    I'll report back when I have any information.

    Best regards,

     

    Dean

     

     

Children
  • 0 in reply to ITOps

    Hi Dean,

    Thanks for reporting back; i'm happy to hear that the issue isn't present now. Please do check if your scheduled scan policy is including archive files; 'Enable deep scanning - scans inside archive files (.zip, .cab, etc.)', this would be a likely culprit of the issue you saw.

    Regards,

    Stephen

Unfiltered HTML