Sophos apparently has trouble dealing with "deleted" messages

Thank you for the feedback!

Is the MyDoom item identified in Quarantine the one at /Users/magill/Library/Mail/IMAP-magill@spacecat.mcgillsociety.org/Mail/Incoming.imapmbox/Attachments/156047/2/mail.zip ?

The basic phenomenon is ... it isn't identified as as single message, but as multiple messages -- which is part of what doesn't make sense.

It appears that the Quarantine is some how "remembering"  something which the scan does not find. 

(And naturally, I can't capture that information -- even with a screens shot.

---

sublime323 wrote:
I am having the exact same problem with Sopho's Quarantine Manager somehow remembering deleted files. In my case, it is Troj/Iframe-DV. The actual file associated with the virus will disappear, and then re-appear. And I will have clean scans for an hour or so, then Quarantine Manager pops up again with the same file. I've done countless Custom Scans. I've manually deleted the file. I've turned off Constant Scanning and Sophos, and then deleted the file. I've cleared all caches, and cookies, searched and scanned various libraries. Yet this file always returns. Any advice for resolving this once and for all? It is very frustrating.

---

This sounds like a slightly different issue; Troj/iFrame-DV detects malicious iFrames on web pages.  Most likely you had a web page refresh and re-add the malicious file to your web cache.  The only way to resolve it once and for all is to avoid visiting the website with the hidden iFrame (via your web browser, or possibly via an app you're using that grabs content via HTTP).

The IMAP issue is a bit trickier as he's got both a server and a client running, and has multiple processes managing the same files.

Here's another data point.

           i have 125 mail boxes for various mailing list, with incoming mail sorted into them by procmail. 

           Typically only 4 or 5 receive 10-15 messages a day while "incoming" receives probably around 100 a day.

           Primary mail server is a Dec Alpha running Tru64 Unix and an IMAP server.

           Additionally, my Gmail and .Mac mail accounts forward to this IMAP server but the Apple Mail app also queries

    them directly.

           All spam filtering is done on the iMac. (Except for whatever Gmail and .Mac do.)

Doing some fairly systematic mail reading last evening.

1- launch mail app. launch "activity window." 

2- begin reading mail while waiting for misc messages to clear from activity.

3- Sophos pops up with a quarantine Manager listing during reading.

4- finish reading mail, having probably deleted 75% of messages (most marked by the mail app as "junk).

6- empty mail trash

5- again wait for activity to complete.

6- quit mail

7- Open quarantine manager -- note that there are 5 "W32/MyDoom-O" messages listed.

      Four are listed "Clean up manually." Each lists a single file in the "Threat Details Section"

      One is listed "Scan Local Drives." This one lists two files in the "Threat Details."

8- Run "Custom Scan" -- "mail+java" -- as it scans, those listed as "Clean up Manually" are removed from the Quarantine Manager window.Quarantine

Scan name: "mail + java"

Scan items:

     Path: /Users/magill/Library/Mail enabled: yes

     Path: /Users/magill/Library/Caches/Java enabled: yes

Configuration:

     Scan inside archives and compressed files: Yes

     Automatically clean up threats: No

     Action on infected files: Delete
Scan started at 2011-05-22 18:02:55 -0400
2011-05-22 18:03:26 -0400 Threat: 'W32/MyDoom-O' detected in /Users/magill/Library/Mail/IMAP-

         magill@spacecat.mcgillsociety.org/Mail/Deleted Messages.imapmbox/Attachments/157283/2/message.zip/message.exe                                                            Deleted threat

2011-05-22 18:03:26 -0400 Threat: 'W32/MyDoom-O' detected in /Users/magill/Library/Mail/IMAP-

         magill@spacecat.mcgillsociety.org/Mail/Deleted Messages.imapmbox/Attachments/157283/2/message.zip

                             Deleted threat

2011-05-22 18:03:43 -0400 Threat: 'W32/MyDoom-O' detected in /Users/magill/Library/Mail/IMAP-

        magill@spacecat.mcgillsociety.org/Mail/Incoming.imapmbox/Attachments/157030/2/message.zip/message.exe

                             Deleted threat

2011-05-22 18:03:43 -0400 Threat: 'W32/MyDoom-O' detected in /Users/magill/Library/Mail/IMAP-

        magill@spacecat.mcgillsociety.org/Mail/Incoming.imapmbox/Attachments/157030/2/message.zip

                             Deleted threat

Scan completed at 2011-05-22 18:05:14 -0400.22151 items scanned, 4 threats detected, 0 issues

=============

9 - at this point only one item remains in the Quarantine Manager window -- the W32/MyDoom-O item, which lists two files in "Thereat details."

/Users/magill/Library/Mail/IMAP-magill@spacecat.mcgillsociety.org/Mail/Incoming.imapmbox/Attachments/157030/2/message.zip

/Users/magill/Library/Mail/IMAP-magill@spacecat.mcgillsociety.org/Mail/Deleted Messages.imapmbox/Attachments/157283/2/message.zip

10- Re-run "mail+java" scan.

11- no threats detected, but the Quarantine manager listing is unchanged.

Scan name: "mail + java"Scan items

. . .
Scan started at 2011-05-22 18:10:36 -0400
. . .
Scan completed at 2011-05-22 18:11:34 -0400.22149 items scanned, 0 threats detected, 0 issues

12 - 

The file:  magill@spacecat.mcgillsociety.org/Mail/Deleted Messages.imapmbox/Attachments/157283/2/message.zip

does not exist -- but the folder "Attachments 157283/2/ Does exist.

157283/2:

total 0

0 drwxr-xr-x  2 magill  magill   68 May 22 18:03 .

0 drwxr-xr-x  3 magill  magill  102 May 22 17:48 ..

13- Same result for the attachment in the incoming box.

14 - So I highlight and  "Clear from list".  .... until the next time.

Note, the Forum software tagged my message as follows:

"Your post has been changed because invalid HTML was found in the message body. The invalid HTML has been removed. Please review the message and submit the message when you are satisfied. "

I did not have any HTML code in the message and see no obvious changes in my text.... who knows what just happened ... :(

Thank you for the update; I'll make sure dev support knows about this thread next week.  I'm still not sure exactly what's going on, as I've attempted to replicate this on a minor scale and found that the Quarantine Manager correctly cleared the items for me after I'd deleted them.  This definitely looks like it needs closer attention.

One question: are you purging the files from your IMAP server on re-sync, or do they get pulled down and cached locally like a rich-man's popmail service?

Added another "twist."

After yesterday's update, I then did a "rebuild" on the inbox and deleted boxes.

Prior to that, yes there was quite a bit of "crud" in both of them, especially "deleted." -- old messages which in fact were not there, but

did exist on disk. (visible via terminal and "ls -als")

The rebuild  clears the directory and then re-downloads and re-indexes the messages. ... all of the "crud" was gone.  Went from about 75 entries

visible in "deleted" (after the "trash" had been emptied) to the current deleted messages, i.e. zero.

One thing which I noticed in "today's" launch of mail.app ...  The Quarantine Manager has multiple My/Doom messages "clean up manually," all for single files, but  seems to always have one My/Doom entry "scan local drives" which contains multiple files (in the Threat Details section.

Also...

The Quarantine Manager has 10 entries -- all stamped 4:05PM. However, the "/Library/logs/Sophos Anti-Virus.log" file only shows 8 entries.

half of which are stamped 16:05 and half 16:36.

--------<start log>-------

com.sophos.intercheck: 2011-05-23 16:05:11 -0400 Threat: 'W32/MyDoom-O' detected in /Users/magill/Library/Mail/IMAP-magill@spacecat.mcgillsociety.org/Mail/Incoming.imapmbox/Attachments/158490/2/document.zip

com.sophos.intercheck:                              Access to the file denied

com.sophos.intercheck:                              Scan of local drives required to complete cleanup

com.sophos.intercheck: 

com.sophos.intercheck: 2011-05-23 16:05:11 -0400 Threat: 'W32/MyDoom-O' detected in /Users/magill/Library/Mail/IMAP-magill@spacecat.mcgillsociety.org/Mail/Incoming.imapmbox/Attachments/158495/2/tedbf.zip

com.sophos.intercheck:                              Access to the file denied

com.sophos.intercheck:                              Scan of local drives required to complete cleanup

com.sophos.intercheck: 

com.sophos.intercheck: 2011-05-23 16:05:11 -0400 Threat: 'W32/MyDoom-O' detected in /Users/magill/Library/Mail/IMAP-magill@spacecat.mcgillsociety.org/Mail/Incoming.imapmbox/Attachments/158503/2/text.zip

com.sophos.intercheck:                              Access to the file denied

com.sophos.intercheck:                              Scan of local drives required to complete cleanup

com.sophos.intercheck: 

com.sophos.intercheck: 2011-05-23 16:05:11 -0400 Threat: 'W32/MyDoom-O' detected in /Users/magill/Library/Mail/IMAP-magill@spacecat.mcgillsociety.org/Mail/Incoming.imapmbox/Attachments/158514/2/message.zip

com.sophos.intercheck:                              Access to the file denied

com.sophos.intercheck:                              Scan of local drives required to complete cleanup

com.sophos.intercheck: 

com.sophos.intercheck: 2011-05-23 16:36:59 -0400 Threat: 'W32/MyDoom-O' detected in /Users/magill/Library/Mail/IMAP-magill@spacecat.mcgillsociety.org/Mail/Incoming.imapmbox/Attachments/158490/2/document.zip

com.sophos.intercheck:                              Access to the file denied

com.sophos.intercheck:                              Scan of local drives required to complete cleanup

com.sophos.intercheck: 

com.sophos.intercheck: 2011-05-23 16:36:59 -0400 Threat: 'W32/MyDoom-O' detected in /Users/magill/Library/Mail/IMAP-magill@spacecat.mcgillsociety.org/Mail/Incoming.imapmbox/Attachments/158495/2/tedbf.zip

\com.sophos.intercheck:                              Access to the file denied

com.sophos.intercheck:                              Scan of local drives required to complete cleanup

com.sophos.intercheck: 

com.sophos.intercheck: 2011-05-23 16:36:59 -0400 Threat: 'W32/MyDoom-O' detected in /Users/magill/Library/Mail/IMAP-magill@spacecat.mcgillsociety.org/Mail/Incoming.imapmbox/Attachments/158503/2/text.zip

com.sophos.intercheck:                              Access to the file denied

com.sophos.intercheck:                              Scan of local drives required to complete cleanup

com.sophos.intercheck:

 com.sophos.intercheck: 2011-05-23 16:36:59 -0400 Threat: 'W32/MyDoom-O' detected in /Users/magill/Library/Mail/IMAP-magill@spacecat.mcgillsociety.org/Mail/Incoming.imapmbox/Attachments/158514/2/message.zip

com.sophos.intercheck:                              Access to the file denied

com.sophos.intercheck:                              Scan of local drives required to complete cleanup

com.sophos.intercheck: 

com.sophos.autoupdate: Info:Checked primary server at 16:49 on 23 May 2011com.sophos.autoupdate: Sophos Anti-Virus is up to date

----------<end log> ----------

What is interesting (besides the fact that the two groups of 4 are the same files... is that they are the same files showing in the

Quarantine Manager window under the single "My/Doom - scan local drives" entry.

Apparently, the reason that "access is denied" is ...

----------<cut here> ----------

ls -ale@ 158514/*

total 32

drwxr-xr-x  3 magill  magill    102 May 23 16:05 .

drwxr-xr-x  3 magill  magill    102 May 23 16:05 ..

-rw-r--r--@ 1 magill  magill  29104 May 23 16:05 message.zip

            com.apple.quarantine   70 

----------<cut here> ----------

xattr -l

158490/2/document.zip: com.apple.quarantine: 0000;4dd9d4cc;Mail;BA111E79-7B99-471C-BCCB-08925E0D6F14|com.apple.mail158495/2/tedbf.zip: com.apple.quarantine: 0000;4dd9f184;Mail;8F51489E-044C-4B24-8C43-61642C4DC7AA|com.apple.mail158503/2/text.zip: com.apple.quarantine: 0000;4dda21ed;Mail;58BAEC14-158A-47AD-8B9B-5BDA2BD9A166|com.apple.mail158514/2/message.zip: com.apple.quarantine: 0000;4dda3ccb;Mail;D47EFB9B-27C9-40B1-B0ED-6AC487D645FA|com.apple.mail

----------<cut here> ----------

At any rate, only those 4 files show up in the log, not the "clean up manually" files.

Then when I run the scan it cleans them all up, but doesn't delete that last entry from the Quarantine Manager. (All 9 entries are gone. Only the last entry containing the multiple entries (apparently of the same files listed individually).

----------<cut here> ----------

Scan name: "mail + java"

Scan items:

     Path: /Users/magill/Library/Mail enabled: yes

     Path: /Users/magill/Library/Caches/Java enabled: yes

Configuration:

    Scan inside archives and compressed files: Yes

    Automatically clean up threats: No

    Action on infected files: Delete
Scan started at 2011-05-23 17:59:03 -0400
2011-05-23 17:59:44 -0400 Threat: 'W32/MyDoom-O' detected in /Users/magill/Library/Mail/IMAP-magill@spacecat.mcgillsociety.org/Mail/Incoming.imapmbox/Attachments/158490/2/document.zip/DOCUMENT.COM

                             Deleted threat

2011-05-23 17:59:44 -0400 Threat: 'W32/MyDoom-O' detected in /Users/magill/Library/Mail/IMAP-magill@spacecat.mcgillsociety.org/Mail/Incoming.imapmbox/Attachments/158490/2/document.zip

                             Deleted threat2

011-05-23 17:59:44 -0400 Threat: 'W32/MyDoom-O' detected in /Users/magill/Library/Mail/IMAP-magill@spacecat.mcgillsociety.org/Mail/Incoming.imapmbox/Attachments/158495/2/tedbf.zip/tedbf.htm                                      .com

                             Deleted threat

2011-05-23 17:59:44 -0400 Threat: 'W32/MyDoom-O' detected in /Users/magill/Library/Mail/IMAP-magill@spacecat.mcgillsociety.org/Mail/Incoming.imapmbox/Attachments/158495/2/tedbf.zip 

                            Deleted threat

2011-05-23 17:59:44 -0400 Threat: 'W32/MyDoom-O' detected in /Users/magill/Library/Mail/IMAP-magill@spacecat.mcgillsociety.org/Mail/Incoming.imapmbox/Attachments/158503/2/text.zip/text.scr

                             Deleted threat

2011-05-23 17:59:44 -0400 Threat: 'W32/MyDoom-O' detected in /Users/magill/Library/Mail/IMAP-magill@spacecat.mcgillsociety.org/Mail/Incoming.imapmbox/Attachments/158503/2/text.zip

                             Deleted threat

2011-05-23 17:59:44 -0400 Threat: 'W32/MyDoom-O' detected in /Users/magill/Library/Mail/IMAP-magill@spacecat.mcgillsociety.org/Mail/Incoming.imapmbox/Attachments/158514/2/message.zip/message.zip/MESSAGE.SCR

                             Deleted threat

2011-05-23 17:59:44 -0400 Threat: 'W32/MyDoom-O' detected in /Users/magill/Library/Mail/IMAP-magill@spacecat.mcgillsociety.org/Mail/Incoming.imapmbox/Attachments/158514/2/message.zip/message.zip

                             Deleted threat

2011-05-23 17:59:44 -0400 Threat: 'W32/MyDoom-O' detected in /Users/magill/Library/Mail/IMAP-magill@spacecat.mcgillsociety.org/Mail/Incoming.imapmbox/Attachments/158514/2/message.zip

                             Deleted threat
Scan completed at 2011-05-23 18:01:08 -0400.

21171 items scanned, 9 threats detected, 0 issues

 ----------<cut here> ----------

Let me know if there is anything else I can send you.

I just thought of something... some mailboxes tend to have an index file and a data file... when you delete a message, it deletes the index entry, but the data is still inside the data file.

Since the index entry is gone, you can no longer "see" the malware in the mailsystem... but if you open the data file with a text editor, you'll find the actual email is still there.

Since mailboxes of this sort contain more than one message, you often cannot "delete" the message without wiping out the entire data file.  If the file is still open for access from your mail app, it won't be deletable.

However, Mail.app can still extract the attachments from this file, and those can be detected and deleted.

SAV will detect many malicious files right through a base64 or MIME encoded wrapper inside a larger database.

I wonder if some of this is what's going on here.... I haven't investigated recently how Apple is currently handling local indexes of IMAP folders.

Of course, this might all be way off, and we're dealing with some Quarantine Manager logfile index management corruption here.

I just thought of something... some mailboxes tend to have an index file and a data file... when you delete a message, it deletes the index entry, but the data is still inside the data file.

(Someday I'll figure out how to quote and insert text on this System. Hand editing the poor HTML generated is a pain.)

In theory... that is what the "rebuild" option will do.

Select a mail box, click - mailbox-pulldown/rebuild.

Mail.app deletes all of the files in the associated directory: ~/LibraryMail/<mailbox>

the index files, data files and attachments.

This is easy enough to observe via the Terminal.

Since mailboxes of this sort contain more than one message, you often cannot "delete" the message without wiping out the entire data file.  If the file is still open for access from your mail app, it won't be deletable.

This is (can be)  true for Unix systems -- i.e. the IMAP server itself, but not for the Mail.app database -- which is the only thing Sophos looks at.

In the Mail.app "database" i.e. the ~/Library/Mail/<mailbox> - each message is a single ".emlx" file with the attachments stored separately.

Apple's Mail.app [Mail User Agent (MUA) ] retrieves each message from the from both POP and IMAP  servers (Mail Transport Agent (MTA)] as individual files and attachments. It maintains an index (.plist) of those retrieved. The only difference between POP and IMAP is that the IMAP server retains a copy of the mail message and any attachments, whereas the POP server does not. The mail client and server each operate independently. 

Ok, here's my current analysis...

What I believe I am seeing, now after the last couple of days of serious "debugging," is that the Quarantine Manager is DISPLAYING an additional "finding" for some reason. That additional finding is what is not being deleted from the Quarantine Manager.

Sophos first finds and lists  the individual files (but doesn't log them anyplace I have found except in the Quarantine Manager window), and then finds them again (logging them in a log file as well as the Quarantine Manager window this time) and builds the "collective" entry. As my last posting showed, that collective entry is apparently a duplicate of the files which are found individually. Then when the "Scan of local drives" deletes those individual messages and updates the Quarantine Manager, the "collective message" entry is not updated.

I don't know for certain, but here is a guess as to what is happening... Sophos is doing two different things, which generate two different situations, visible in the Quarantine Manager.

As the Mail.app retrieves each individual mail message it is scanned and an entry generated for the Quarantine Manager -- but this activity is not logged. (Or at least I haven't found evidence of it being logged.) This appears to be the action of the  "On-Access Scanner."

Then Sophos Scans the entire ~/Library/Mail structure and generates the "collective" entry, this time logging what it is doing. (I don't know the "period" here, but it appears to happen at least hourly.)

The "Scan Local Drives (custom scan)" logs to "~/LIbrary/Logs/Sophos Anti-Virus/Scans/<scan-name>/<time-stamp>

This log file contains the entries the scan actually deletes and they are deleted from the Quarantine Manager window as well.

The "collective scan" appears to be occurring when Sophos does its hourly check to update the virus database.

This log file (entries are tagged "com.sophos.intercheck " -  contains the entries which appear in the Quarantine Manager as that single collective entry. This "collective" entry is not updated by the "Scan Local Drives" (custom scan) action.

In short, there is a "communication" problem between "Scan Local Drives (Custom Scan)" and the Quarantine Manager display.

Guessing from the "Clear from list" button, this is "expected behavior" (i.e. a "known issue") with Sophos.

I suspect, but haven't tried, that the issue may be with the "Custom Scan" -- with a 320 gig drive, I have only ever done one full run of "Scan Local Drives" as it took something like 14 hours as I recall.