Recently I put out a
Vodcast about anti-virus and anti-malware testing. I also explained very carefully what not to do ... and this is exactly what has been done by
www.untangle.com in their latest anti-virus test.
So the people at untangle.com decide to “test” anti-virus product in an effort to prove their dedication to open source software at least that is what you can read between the lines... I’m not against open source, but if you want to promote it then be honest about it.
So what did they do ...
• 10 anti-virus vendors were tested (ClamAV, FProt, Fortinet, Global Hauri, Kaspersky, McAfee, SonicWall, Sophos, Symantec, Watchguard)
• 35 samples were used (6 EICAR samples, 12 from Untangle, and 17 user-submitted samples)
• It appears they performed an on-demand scan of the sample set.
What did I found out:
- Small Sample Size .. This was the basic of my Vodcast, please use a representative testset.
- Possible Biased Samples .. With just 35 viruses you can bias the complete test, again the testset must be representative.
- Comparing the Wrong Products .. The test compares 5 Linux, 2 Windows, and 3 Gateway products. That's again one of the main mistakes made by 'beginners'.
- Conflict of Interest .. The fact that this test was performed by Untangle who develops, markets, and sells an anti-virus solution with their gateway product is a blatant example of a conflict of interest.
And than the most problematic one in my own eyes ...
was the inclusion of EICAR test files. In their report untangle says “The first set was a basic test set (from eicar.org) that is a universal virus test.” This is completely incorrect. To call the EICAR test file a universal test virus is really showing complete incompetence! The EICAR test set is there to be used to test if the scanner is functioning. EICAR doesn’t tell you anything you can use to conclude that something is excellent in detection.
As EICAR Director Press & Information I can assure you that this is nearly unbelievable and really misleading to customers!
Finally, these people who cannot competently test software, and who run blatantly biased and incompetent tests are putting viruses up on their web site for anyone to download. By offering a link to these live viruses on their company’s public website, they are in violation of the Computer Fraud & Abuse Act which prohibits the distribution of computer viruses because it is endangering public safety. There is a reason why only trained security professionals should handle computer viruses.
Conclusion: Please ignore such tests or sites in the future. They are just showing their incompetence of handling security in a good way. However they did one thing correct... everybody knows now untangle.com .
I'm back home from Corsica ... and did you really think that I didn't do anything malware related over there? Well I've been contacted by the press for an interview with VRT (Belgian Broadcast station). I will report more on this shortly. And that's not all .. I'm just home and guess what? Indeed, 'De Standaard' newspaper was calling me for another interview. You can read it all today or later at the press page. What happened during my week off ... well let's see: another Sony rootkit related problem appeared, more RTF spam, monster.com was attacked and some other malware problems. Honestly speaking, that has been the reason why I normally did not go on vacation at the end of August the last 10 years. During the past years we've got always a small boost of malware at the end of the summer vacations. And I thought that it all was over this year. I was clearly wrong!
