We Will Find the Way …
Article by: Dr Anton Zajac – VP, Eset Software
Courtesy of Virus Bulletin 
When Hannibal led his troops through the freezing peaks of the snow-covered Alps, he declared: ‘We will find the way … and if not we will make one!’. The fatalism and determination of Hannibal’s famous statement has inspired many and will continue to do so. However, when a frozen death is not the unspoken alternative, I believe that we must take care to ensure a noble cause is not used to justify unethical behaviour.
When I read consumer computing and technology publication CNet’s latest online reviews of a number of anti-virus products, I realized the reviewers had fallen into ‘Hannibal’s trap’ of obsessing over a chosen path despite its obvious folly. The goal was to provide readers with valid information on the quality of the products being tested. They failed – the question is why.
To the less well informed reader, this comparative review of anti-virus products appears to be based upon a solid, perhaps even scientific, testing methodology. To test the detection rate of the products, the reviewers used ‘synthesized viruses’ generated by the Rosenthal Utilities (RU). However, the use of RU was discredited by the AV community several years ago. Almost two years ago, well-known anti-virus researcher Joe Wells wrote an open letter to CNet, pointing out the flaws in their testing methodology. The letter was co-signed by 19 prominent members of the anti-virus community (see VB, November 2000, p.3) . However, CNet continued to carry out their tests using RU and reporting consequently flawed results on their website. Perhaps they honestly thought the industry experts were wrong; perhaps they thought anti-virus products should detect RU generated synthesized viruses. Perhaps they were just obsessed with not admitting an error.
I contacted CNet’s Chief Editor but received no answer to my question regarding CNet’s definition of a computer virus, nor to the question: ‘Is the attachment [generated by RU] a virus?’ which I posed specifically to CNet’s Technical Editor and to the CEO. However, I was assured repeatedly that CNet was ‘phasing out further use of the RU for anti-virus testing’. I could accept CNet’s diplomatic silence more easily had the flawed test results been removed from CNet’s websites and an appropriate apology been made to all affected parties (including their misled readers).
Without an answer, my search for the truth continued (I had to ‘find the way’). Analysis of all the simulated COM and EXE files revealed a striking simplicity of all (roughly 2000) generated RU samples. All the samples have similar structure and one thing in common: none of them is a virus nor a virus-infected file. A virus alarm triggered by any of the RU samples is a false alarm.
RU are distributed with documentation defining their scope of applicability, which states ‘the simulators all produce safe and controlled dummy test virus samples – these samples contain the signatures (only) from real viruses. The programs themselves are not really infected with anything – The simulators’ ability to actually test products is limited.’ The documentation goes on to state: ‘These test virus simulators are not intended to replace the comprehensive collection of real virus samples.’ In spite of the striking clarity of RU’s disclosure, CNet used samples ‘not really infected with anything’ to perform tests whose results were to be presented to public as real and relevant – in my view this was irresponsible.
The laws of quantum mechanics have never ceased to amaze me. When a test is performed on a set of identically prepared systems (e.g. electrons) the test results differ as determined by the probabilistic nature of those laws. CNet’s ‘laws’ for testing anti-virus products are more peculiar still. Regardless of what set of products is tested, and regardless of what feature is tested, the outcome is always the same: there is only one perfect product. CNet will always ‘find its way’. Perhaps times are changing – it took CNet two years to discontinue its flawed testing; we can only guess the time it will take CNet to stop misleading its readers.
VB would like to hear your views. Send us your thoughts: comments@virusbtn.com.
Article Copyright 2002 Virus Bulletin Ltd (www.virusbtn.com). Permission is granted to Help Net Security to re-print the article.