Another STUPID Anti-Virus Test: I'm really 'untangled'!
Recently I put out a Vodcast about anti-virus and anti-malware testing. I also explained very carefully what not to do ... and this is exactly what has been done by www.untangle.com in their latest anti-virus test.
So the people at untangle.com decide to “test” anti-virus product in an effort to prove their dedication to open source software at least that is what you can read between the lines... I’m not against open source, but if you want to promote it then be honest about it.
So what did they do ...
• 10 anti-virus vendors were tested (ClamAV, FProt, Fortinet, Global Hauri, Kaspersky, McAfee, SonicWall, Sophos, Symantec, Watchguard)
• 35 samples were used (6 EICAR samples, 12 from Untangle, and 17 user-submitted samples)
• It appears they performed an on-demand scan of the sample set.
What did I found out:
- Small Sample Size .. This was the basic of my Vodcast, please use a representative testset.
- Possible Biased Samples .. With just 35 viruses you can bias the complete test, again the testset must be representative.
- Comparing the Wrong Products .. The test compares 5 Linux, 2 Windows, and 3 Gateway products. That's again one of the main mistakes made by 'beginners'.
- Conflict of Interest .. The fact that this test was performed by Untangle who develops, markets, and sells an anti-virus solution with their gateway product is a blatant example of a conflict of interest.
And than the most problematic one in my own eyes ...
was the inclusion of EICAR test files. In their report untangle says “The first set was a basic test set (from eicar.org) that is a universal virus test.” This is completely incorrect. To call the EICAR test file a universal test virus is really showing complete incompetence! The EICAR test set is there to be used to test if the scanner is functioning. EICAR doesn’t tell you anything you can use to conclude that something is excellent in detection.
As EICAR Director Press & Information I can assure you that this is nearly unbelievable and really misleading to customers!
Finally, these people who cannot competently test software, and who run blatantly biased and incompetent tests are putting viruses up on their web site for anyone to download. By offering a link to these live viruses on their company’s public website, they are in violation of the Computer Fraud & Abuse Act which prohibits the distribution of computer viruses because it is endangering public safety. There is a reason why only trained security professionals should handle computer viruses.
Conclusion: Please ignore such tests or sites in the future. They are just showing their incompetence of handling security in a good way. However they did one thing correct... everybody knows now untangle.com .
So the people at untangle.com decide to “test” anti-virus product in an effort to prove their dedication to open source software at least that is what you can read between the lines... I’m not against open source, but if you want to promote it then be honest about it.
So what did they do ...
• 10 anti-virus vendors were tested (ClamAV, FProt, Fortinet, Global Hauri, Kaspersky, McAfee, SonicWall, Sophos, Symantec, Watchguard)
• 35 samples were used (6 EICAR samples, 12 from Untangle, and 17 user-submitted samples)
• It appears they performed an on-demand scan of the sample set.
What did I found out:
- Small Sample Size .. This was the basic of my Vodcast, please use a representative testset.
- Possible Biased Samples .. With just 35 viruses you can bias the complete test, again the testset must be representative.
- Comparing the Wrong Products .. The test compares 5 Linux, 2 Windows, and 3 Gateway products. That's again one of the main mistakes made by 'beginners'.
- Conflict of Interest .. The fact that this test was performed by Untangle who develops, markets, and sells an anti-virus solution with their gateway product is a blatant example of a conflict of interest.
And than the most problematic one in my own eyes ...
was the inclusion of EICAR test files. In their report untangle says “The first set was a basic test set (from eicar.org) that is a universal virus test.” This is completely incorrect. To call the EICAR test file a universal test virus is really showing complete incompetence! The EICAR test set is there to be used to test if the scanner is functioning. EICAR doesn’t tell you anything you can use to conclude that something is excellent in detection.
As EICAR Director Press & Information I can assure you that this is nearly unbelievable and really misleading to customers!
Finally, these people who cannot competently test software, and who run blatantly biased and incompetent tests are putting viruses up on their web site for anyone to download. By offering a link to these live viruses on their company’s public website, they are in violation of the Computer Fraud & Abuse Act which prohibits the distribution of computer viruses because it is endangering public safety. There is a reason why only trained security professionals should handle computer viruses.
Conclusion: Please ignore such tests or sites in the future. They are just showing their incompetence of handling security in a good way. However they did one thing correct... everybody knows now untangle.com .
posted by Eddy Willems at 11:45 am
<< Home