NSS Labs Endpoint Protection Test Results

On Monday, NSS Labs released the results of their anti-malware Endpoint Protection Product tests. The test results are separated into consumer and corporate product lines, with the consumer report available for download from their web site after free registration.

The test put each product through a 17-day rolling assessment, where each day the latest updates to the product were applied and a fresh list of malware-serving URLs were processed. This provides a clear view of how these products fare in the real world, and not just against a static list of well-known samples. Each product had two opportunities to block the malware, once during download, and again once it was written to disk and executed by the user. The score for a given product is calculated as the sum of both methods of blocking the sample, for example, if it was missed during download, but caught on execution, it still counts as being blocked. Each of these products also contains an anti-virus engine, which should provide some basic protection for unknown samples, based on heuristics and behavior.

The top-ranking product in the consumer test was Trend Micro, which caught a whopping 96.4% of all malware samples, followed by Kaspersky at 87.8%. Most of the major-brand consumer products had an average closer to 80%, with AVG, Panda, and ESET all coming in below the average. These results show that on average, two out of every ten pieces of malware will slip past consumer-grade security solutions. Users who rely on cheaper products like AVG and ESET have an even lower level of protection, while those using Trend are well above the average. The corporate product test results are a bit different (and somewhat surprising, compared to the consumer results), but are only available for a fee from NSS Labs. If you rely on Sophos for your enterprise endpoint security, this report may be worth purchasing.

From my own testing with Metasploit-generated payload executables, both Trend and Kaspersky seem to rely on heuristics and behavior more than the other products in the field. For example, this VirusTotal report shows the results of a reverse connect shell generated by the latest version of Metasploit. While two products misclassified the executable as "Win32:Tipa" (due to the read/write/exec section), Trend Micro was the only product to clearly identify the file as "packed" using what looks like an entropy signature. Two McAfee products flagged the file as suspicious, but in most scenarios the file would have been allowed anyways. Unique hashing doesn't work in this case, as the executable is randomized every time it is generated by Metasploit.

From a penetration testing perspective, the NSS reports are useful in determining not only how robust a client's endpoint protection is, but what the probability of existing infections are for their workstations. A company using a product on the weaker end of the scale (AVG, ESET, etc) is likely to have a higher chance of botnet agents and credential sniffers.

Some easy ways to determine what filtering software is in use at a given organization are to send an email to a bogus address at the domain, solicit an email response from an internal user, or find a sent email archived online -- any of these methods should allow access to the MIME headers, which security products often insert their product name and version into. For example, if we wanted to see what a particular government agency is using, all we have to do is send an email to a bogus address, wait for the bounce reply, and look at the headers:

X-IronPort-AV: E=Sophos;i="4.44,431,1249272000"; d="scan'208";a="9936347"

This line indicates that Sophos is being used with an IronPort appliance and includes the version number of the product. The "1249272000" value after the version is a UNIX timestamp, which converted to a human-readable date becomes "2009-08-02 23:00:00 -0500". This is likely the date on which the product was last updated. From a penetration testing perspective, we need to find a way to bypass detection of our malware by this version of Sophos in order to reach the endpoint. We still don't know what endpoint software is in use, but we can either guess that it too is Sophos-based, or try to solicit an email response from an internal user and then craft our malware so that it avoids both the gateway and the endpoint product. In most cases, bypassing a specific anti-virus is just a matter of hex-editing a few bytes of the executable.

If we rolled back the clock 10 years, I don't believe anyone expected their anti-virus product to become the end-all of desktop and gateway security. However, the popularity of social media sites has triggered a bloom in social-engineering malware attacks, forcing the anti-virus industry to expand its scope. The products that scored the highest results in the consumer report all used cloud-backed signature sets to detect and block malware, removing the normal window of exploitation between signature updates. The disparity between vendors is surprising, considering the age of the anti-virus industry and the relatively equivalent price points. Penetration testers and system administrators both need to be aware of the strengths and weaknesses of the technology as well as specific products on the market.