Thread Info
  • State Not Answered
  • Locked Locked
  • Replies 60 replies
  • Subscribers 1 subscriber
  • Views 102188 views
  • Users 0 members are here
Options
Suggested
This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

Sophos 9's Buffer Overflow Protection System (BOPS) and Internet Explorer 8

JonKloske

Hi,

Just wondering if anyone else has noticed any odd behavior of Internet Explorer since upgrading from Sophos 7 to Sophos 9?

We've found that on a substantial portion of our machines (Windows XP, Windows Server 2003, and possibly even Windows 7) suffer from a reproducible but statistical-in-nature deadlock in Internet Explorer.

I originally reported the problem on Microsoft's Technet forums (http://social.technet.microsoft.com/Forums/en/w7itpronetworking/thread/ef0a7af9-26b9-45b5-a05e-0cfd88c42aee) however it soon became clear that the culprit was Sophos_detoured.dll, part of the BOPS in Sophos 9 (and the timeline of the problem occurred very shortly after upgrading substantial portions of our machines to Sophos 9 from the original Sophos 7 clients they had installed.)

I notice someone else on the MS forums said they had the same problem, so I thought I'd post this here to achieve two things:

1. If other people experience this problem, they may know what's causing it instead of tearing their hair out about it

and

2. Perhaps other people have suggestions or advice

I should point out that we have an open ticket with Sophos about this at the moment [#2345512], and I've discovered at least two workarounds (which are both effectively the same workaround from different directions), so I'm not necessarily after assistance, just healthy discussion and information sharing.

As an aside, after I posted about this issue in a similar way on our internal forums, another IT staff member from a completely unrelated area responded along the lines of "Ahhh! That's what's causing it! We've just been reformatting and re-installing windows since we were stumped by the problems, and we never connected it with the recent upgrade to 9!"

And finally, the easiest workaround to the problem is to disable BOPS and reboot. There's a reghack that achieves effectively the same thing involving the AppInit_DLLs key (see the referenced microsoft forum post for details if you're brave.)

Cheers,

Jon.

:3208


This thread was automatically locked due to age.
  • 0

    Hi to everyone watching this thread,

    I'd just like to report that we've found a number of instances where turning off BOPS through the client (either from centrally managed policy or manually at the clients through the system notification area GUI) has NOT removed the AppInit_DLLs entry, and therefore does NOT resolve this problem.

    To clarify, turning off BOPS via the GUI or in policy appears to merely disable any actions the system may have taken, not prevent the system from being installed and injecting itself into newly created processes via the AppInit_DLLs registry key (which is what's actually causing the problem with Internet Explorer.)

    Regards,

    Jon.

    :3518
  • 0

    Hi Jon,

    yes, the same problem that I read in this thread!

    :3520
  • 0

    Hi James,

    Sorry to hear that! Hopefully some of the tips in this thread were helpful to you.

    Cheers,

    Jon.

    :3521
  • 0

    Hi Lestat,

    We haven't tried turning that option off, but it's worth noting that that option is the only way to detect viruses that Sophos does not have definitions for. This happens surprisingly often on our network (I recently spent two weeks looking at the information provided by this system and found almost 10 previously undetected viruses on our network - a large portion of which other AV products picked up, sadly.)

    The basic procedure for detection of these previously unknown viruses is to watch the Sophos Enterprise Console (go to Endpoints, click on the Status tab and sort by Alerts and errors) and investigate any “Suspicious behavior detected” or “Suspicious file detected” or other alerts. These alerts often (but not always) identify the local path of the file in question, which you should isolate inside a password protected archive file (I named mine “(password=virus)blah.7z”) which should be emailed to samples@sophos.com with the subject line “Sample submitted for analysis“. The body of the email should include the HIPS or other trigger Sophos was tripping on, your full contact details (including your client number or other indication that you're a current customer) and any other information you think is relevant or helpful (such as a rudimentary analysis of the suspected malware, especially any actions you notice it taking like opening browser windows).

    Regards,

    Jon.

    :3550
  • 0

    Have you tried to use internet explorer without add ons? Disabling all add ons worked for me. Now I shoud find the add on that causes the problem....

    :3615
  • 0

    Hi Alberto,

    As per the thread I linked to in my original post, yes, I've tried with addons disabled and it made no difference. Furthermore, the problem is with a DLL that's injected into the process at load time, which is completely separate from browser addons which are loaded and attached after the process has started.

    Regards,

    Jon.

    :3632
  • 0

    Good afternoon everyone.

    In order to look into this issue further (I can appreciate some posters have active case references for this issue) , can you ensure that cases are opened - ideally via www.sophos.com/support/query as this will automatically create a case reference for you, with Sophos diagnose logs (http://www.sophos.com/support/knowledgebase/article/33533.html), msinfo32 output (please save as .nfo file) and answers to the following questions.

    Brief rundown:

    Sophos Detoured is used by Sophos Anti-Virus for Buffer Overflow Protection (BOPS) and Host Intrusion Prevention System (HIPS).

    HIPS and BOPS information: http://www.sophos.com/support/knowledgebase/article/25044.html

    Registry information: http://www.sophos.com/support/knowledgebase/article/36501.html

    NOTE: On 64-bit platforms the Sophos_detoured.dll file is it also found under HKLM\Software\Wow6432Node\Microsoft\Windows NT\Current Version\Windows\AppInit_Dlls


    In order to investigate please provide inline answers to these questions, when logging a case.

    1. Were there any actual Buffer Overflow Detections?
    2. What was the first version of Sophos Anti-Virus this issue was discovered with?
    3. What change to the installation of Sophos Anti-Virus is necessary to resolve the conflict?
      1. Disabling the BOPs option in the Sophos Anti-Virus configuration (then reboot)?
      2. Disabling the HIPs option in the Sophos Anti-Virus configuration (then reboot)?
      3. Disabling both the BOPS and HIPs option in the Sophos Anti-Virus configuration (then reboot)?
      4. Removing the Detoured entry from the registry (then reboot)?
        • If system is 64-bit, are both registry entries required to be removed or just one?
    4. Are there any other values present in AppInit_DLLs?
      1. Does changing the ordering of the values change the outcome of the issue (i.e. Sophos_detoured first/ last in order)?
    5. Is the problem platform dependant? (Example: 32-bit only, Vista only etc…)
    6. Are there a succinct set of steps to cause/ recreate the conflict (please list)?
      1. What is the reproduction rate (i.e. 100% everytime, or one in five attempts, etc.)?
    7. What is the normal/expected behaviour ?
      1. Does Detoured exacerbate a problem rather than cause it?
    8. Is the affected application on the latest version?
      1. Have you contacted the other vendor in case an issue is already known?
    9. If there is a server/client relationship relevant to the problem, which PC does Detoured need to be removed from?
      (Examples: Remote desktop connection, VPN connections, database connections, applications that require a server connection when ran from the client)

    The more information we receive from yourselves the quicker we can isolate and fix this frustrating issue.

    Will

    :3764
  • 0

    Hi,

    > Were there any actual Buffer Overflow Detections?

    No.

    > What was the first version of Sophos Anti-Virus this issue was discovered with?

    9. Version 7 appears unaffected.

    > What change to the installation of Sophos Anti-Virus is necessary to resolve the conflict?

    Nothing short of simply not using Sophos Antivirus works. Even the workaround I described at the beginning of this thread is reverted by the AV client after a week or so, meaning you have to keep applying the workaround (this is the removing the entry from AppInit_DLLs workaround.)

    > Disabling the BOPs option in the Sophos Anti-Virus configuration (then reboot)?

    Makes no difference - disabling this option and rebooting simply means that BOPS will allow overflows, not that it won't be loaded into applications.

    > Disabling the HIPs option in the Sophos Anti-Virus configuration (then reboot)?

    No, though doing this is incredibly stupid given the huge number of malware items that Sophos does not detect (I found almost 10 undetected viruses on our network during a two week period, a reasonable portion of which other AV vendors detected no troubles.)

    > Disabling both the BOPS and HIPs option in the Sophos Anti-Virus configuration (then reboot)?

    Again, no effect.

    > Removing the Detoured entry from the registry (then reboot)?

    Works for a while, but then the AV client re-adds the entry and therefore it starts failing again.

    > If system is 64-bit, are both registry entries required to be removed or just one?

    Haven't tried reproducing the problem on 64bit architecture.

    > Are there any other values present in AppInit_DLLs?

    A small portion (probably less than about 10-20%) of the affected clients also have a google desktop search DLL in there.

    > Does changing the ordering of the values change the outcome of the issue (i.e. Sophos_detoured first/ last in order)?

    No, and additionally most of them only have a single entry so this doesn't apply in those cases.

    > Is the problem platform dependant? (Example: 32-bit only, Vista only etc…)

    We haven't tried reproducing on 64bit (though we've had anecdotal reports that 64bit OSs are also affected), but we can confirm the problem affects 32bit windows and windows 7.

    > Are there a succinct set of steps to cause/ recreate the conflict (please list)?

    Yes. Make sure the registry entry is there, open up ProcessExplorer, then hammer away on the quick launch icon for IE as fast as you can until about 50 have been started. Close down any that have started up successfully, wait for a bit, and then examine the process list or any IE windows that aren't responding - generally we get 2-3 processes in this state after this.

    The faster you open up IE the better chance you have of triggering this. If you're doing it via remote desktop on a slow link, or on a slow VM server it might reduce the chances of you triggering this behavior. Multicore machines seem more susceptible to the problem than single core machines.

    I  managed to reproduce this problem over a slow remote desktop link with a Sophos level 2 tech in England. He seemed to think there was a known issue with Sophos_detoured.dll deadlocking when Windows Live Signon Assistant addon in IE was enabled (which it is by default in most microsoft OS's). We tried with this addon disabled and we got a much lower chance of deadlocking ,but we still managed to reproduce a deadlock with this addon disabled, so that's not the only problem with Sophos_detoured.dll.

    > What is the reproduction rate (i.e. 100% everytime, or one in five attempts, etc.)?

    As I've repeatedly stated, it's about 5% on average if you do it right, though sometimes you'll get a long run of stability, and then suddenly you'll get almost an 80% failure rate. You need patience and a quick trigger finger on the mouse.

    Note, the users seem to manage to hit the condition frequently enough that they're asking us to get a different AV vendor, so perhaps the "in-the-wild" rate is higher than that, or something else triggers it as well that we haven't identified.

    > What is the normal/expected behaviour ?

    IE runs without deadlocking?

    > Does Detoured exacerbate a problem rather than cause it?

    As far as we can tell it's a 1:1 correlation. If that entry exists, the problem exists. If that entry is removed, we have been unable to reproduce the problem no matter how hard we try.

    > Is the affected application on the latest version?

    Yes. Fully updated.

    > Have you contacted the other vendor in case an issue is already known?

    As I said at the top, I've contacted the microsoft forums about IE, and their assessment was that it was a Sophos problem. Additionally, other users reported the same problem, and they too were running Sophos.

    > If there is a server/client relationship relevant to the problem, which PC does Detoured need to be removed from?
    (Examples: Remote desktop connection, VPN connections, database connections, applications that require a server connection when ran from the client)

    There is no dependencies. All you need is IE, Sophos_detoured.dll, and some patience.

    Regards,

    Jon.

    :3864
  • 0

    Related issue possibly:

    IE8, Windows 2003 SP2, Citrix Presentation Server, Sophos 9.0.5.

    Setting the homepage on any user (Citrix or not) to our local intranet page, the page will not load when IE8 loads. Hitting "Refresh" once IE8 is up will load the page instantly. After spending a long, long time trawling around I found this thread and discovered that either disabling Sophos Web Content Scanner add-in OR removing the Sophos entry from the AppInit_DLLs registry key will make the problem go away. This problem can be reproduced 100% of the time on this server.

    (We also have intermittent hanging issues with IE7 on two other Citrix servers, ongoing for months, which I am now thinking may be related to this issue too - although this is un-tested. That issue is about 2-3% of the time IE7 will hang while loading.)

    :3882