Today Cyveillance posted a press release titled “Cyveillance testing finds AV vendors detect on average less than 19% of malware attacks”. I have to say that this report was clearly put together by the person who cleans the toilets. Even an intern could do a better job. Lets examine why I think this has the dubious honor of being the Worst Report Ever (just imagine comic book guy saying it).
First of all the title, now if that isn’t FUD then I don’t know what is. Lets examine the release in detail.
The report reveals that traditional antivirus (AV) vendors continue to lag behind online criminals when it comes to detecting and protecting against new and quickly evolving threats on the Internet.
Duh! Of course they are. I wonder how many analysts it took to figure this out?
Cyveillance testing1 shows that even the most popular AV signature-based solutions detect on average less than 19% of malware threats. That detection rate increases only to 61.7% after 30 days.
Tell me something that I couldn’t have figured out myself. Every self respecting security professional knows that signature-based solutions just don’t cut the mustard in todays security landscape. End users won;t know this of course but I couldn’t figure out who the report was aimed at so I used my experience as a baseline. I recently attended a presentation by one of the AV vendors mentioned and they presented a pretty bleak landscape for signature-based solutions. They mentioned that one particular piece of malware had over 1000 variants. Now show me a product that can cope with signatures for all the circa 1 million pieces of malware out there? There isn’t one and that’s why AV vendors rely on other technologies like heuristics and reputation etc. But the report is on signature-based detection so did they mean they will only test this functionality or did someone forget that AV is not about signatures any more? If they intended just to report on one function of AV then this is stupid as it doesn’t give a complete picture about the level of protection. So maybe they meant just malware detection using any technique the vendor can use, if this is the case they don’t say “using signature-based detection” because that’s just wrong.
Now the biggest bugbear I have with this report is its use of stats. One of my favorite quotes is
There are three kinds of lies: Lies, Damn Lies, and Statistics.
Now anyone can manipulate statistics but I am not saying that they manipulated the data in anyway. It’s all to do with how it’s presented. First of all Cyveillance states that they used 1,708 malware samples. What samples did they use and where did they get them from? Wikipedia states
Sampling is that part of statistical practice concerned with the selection of an unbiased or random subset
Incorrect sampling will certainly reduce a skew in the results. Some vendors will likely class one piece of software as malware whilst others may not. So what if this happens? Does the vendor get scored down because they didn’t detect the malware? Who knows?
Additionally, I have concern with regards to the use of the average across all the products. Some of the products they used I have never heard and clearly bring the over all results down. I question the use of these products in the test. If for example I was going to report on the overall speed of cars and I use the following data set:
Bugatti Veyron: 253 mph
Koenigsegg CCX: 245 mph
Ferrari Enzo: 217 mph
Ford Focus: 133 mph
Renault Clio: 108 mph
Smart Car: 80 mph
Average 172.7 mph
Now does this mean that the average speed of all cars is 172.7 mph? Clearly not.
Also, I suspect that some of the vendors do not have the money and resources to put into sampling malware and defining signatures. But don’t detract from the fact that using the average creates FUD in this case. Should you be worried? So only 61.7% of their sample of malware was detected after 30 days, what I am worried about is does the product I am using detect the malware after 30 days? Well if your a Kaspersky or AVG user then no but if your a Norman or eTrust-Vet then yes. The use of the average just brings the overall results down. This is plain wrong to do this!
Would you be surprised if I told you that if you chose the top 8 vendors from the list the overall detection rate jumps to 79.4%?
Panos Anastassiadis, COO of Cyveillance. “To increase protection, users can’t forget the basics – avoid unknown or disreputable websites, increase security settings on their web browser and leverage supplemental malware block lists to increase security on their devices. ”
Excuse me? The users? Do you expect me to rely on administrative controls to keep my environment secure? I cannot expect users not visit disreputable websites either knowingly or unknowingly, they simply do not have a banner which says “disreputable website”.
Finally,
The only thing which I can agree on with Cyveillance COO Panos Anastassiadis is
Only through both proactive and reactive tools can a solid security platform be achieved.
I do hope though that when he means tools he is referring to controls in general and not just software.
Rant over
