Sophos Anti-Virus for Mac version 9.0 Preview

---

ZRL1 wrote:

I suspect that SophosWebIntelligence has to clear each of those links in the page before it lets them through so they seem to appear one at a time.

---

Mostly correct, although we can process things in parallel if the browser does. There are three points where the system does a lot more processing for web traffic: (1) each network socket is filtered by SophosWebIntelligenceD to pick out the URL and the content; (2) the URL is given to SophosSXLD which communicates with our cloud infrastructure; and (3) the content is given to SophosScanD for the anti-malware scan.

Each of these components is multi-threaded so we can absorb as much work as possible, but the necessary CPU cycles still add up. The URL lookups are backed by a sophisticated caching mechansim to avoid looking up things too often (that would interfere with performance) but often enough to give real-time reaction to threats. We typically see more than 90% caching even for complex sites when you start browsing them, although if you stop browsing then return to the site later we have to requery. We don't bother trying to avoid re-scanning (by caching content) because the browsers do an excellent job of caching themselves. We never re-scan cached content because the browser never fetches it from the remote server. But anything the browser can't or won't cache will get scanned (which is the corrrect behavior).

Our typical "worst case" sites are like Amazon's front page or Yahoo!'s front page. Both are full of small bits fetched from lots of different servers. The front page of the Washington Post is similar, it references content from more than sixty different servers. Seems like a good site for more performance testing.

---

ZRL1 wrote:

I suspect that SophosWebIntelligence has to clear each of those links in the page before it lets them through so they seem to appear one at a time.

---

Mostly correct, although we can process things in parallel if the browser does. There are three points where the system does a lot more processing for web traffic: (1) each network socket is filtered by SophosWebIntelligenceD to pick out the URL and the content; (2) the URL is given to SophosSXLD which communicates with our cloud infrastructure; and (3) the content is given to SophosScanD for the anti-malware scan.

Each of these components is multi-threaded so we can absorb as much work as possible, but the necessary CPU cycles still add up. The URL lookups are backed by a sophisticated caching mechansim to avoid looking up things too often (that would interfere with performance) but often enough to give real-time reaction to threats. We typically see more than 90% caching even for complex sites when you start browsing them, although if you stop browsing then return to the site later we have to requery. We don't bother trying to avoid re-scanning (by caching content) because the browsers do an excellent job of caching themselves. We never re-scan cached content because the browser never fetches it from the remote server. But anything the browser can't or won't cache will get scanned (which is the corrrect behavior).

Our typical "worst case" sites are like Amazon's front page or Yahoo!'s front page. Both are full of small bits fetched from lots of different servers. The front page of the Washington Post is similar, it references content from more than sixty different servers. Seems like a good site for more performance testing.