Are Malware Test Files Useful?
Several of the major anti-malware developers frequently participate in independent tests conducted by Virus Bulletin. Numerous companies have received VB 100 certification, including Sophos and Symantec. Since many products on the market today are effective, some have posed the question of how useful malware test files really are. We have discovered some information that will shed light on the subject and hopefully answer this question
False Positives
Earlier this year, each of the three anti-malware products submitted by Trend Micro into Virus Bulletin's independent test produced failed results because of false positives. A total of 20 products were submitted, with six generating false positives when scanning a batch of known clean files. Aside from Trend Micro, other products that failed to meet requirements for VB 100 certification were FortiClient, Ikarus Utilities and VirusBuster.
As one of the big four anti-malware developers, Trend Micro's products falsely identified a Microsoft development tool as a piece of spyware. The test files were the first conducted by Virus Bulletin on the 64-bit Windows Vista system. The malware test files included were known to be clean and chosen from the "most popular" lists on various free download sites.
The Detriment of False Positive Alerts
The fact that some of today's best software resulted in such an upsurge of false positives raises huge concerns. False detections can actually stir up as much chaos as genuine malware, causing end users to panic and needlessly delete legitimate files under the assumption they are being attacked. The results of such a situation can be very damaging. According to John Hawes, senior consultant at Virus Bulletin, most products that failed with the malware test files showed a significant reliance on heuristic detection techniques. He goes on to say that anti-malware developers have a long way to go if they expect to minimize this alarming number of false detections.
In February of 2007, Microsoft received criticism from all angles after their OneCare conumser AV software failed to meet the requirements for VB 100 certification when tested on the 32-bit Windows Vista platform. In the most recent test, ForeFront, their enterprise product, gave a strong performance and was awarded VB 100 certification on the 64-bit Vista platform.
Requirements for meeting VB 100 status calls for anti-virus products to scan test files of numerous viruses from the "in the wild" list, strains that are known to be circulating on systems around the world. In order to earn VB 100 certification, these products must be able to detect 100% of the malware categorized in the wild, all without generating any false positives when scanning a set of clean files.
Conclusion
Are malware test files useful? Companies like Trend Micro and Microsoft would respond with a resounding yes. An outbreak of false positives resulting from different programs would be an absolute nightmare for the industry. Unfortunately, three more anti-virus developers recently failed to meet VB 100 requirements for malware detection on Windows XP. This is clear evidence that the struggle continues.
Post a comment