Software Security Rating Systems
Posted by Aaron Massey on 08 Jul 2011.
In my last post, I talked about the inherent challenges in evaluating software for security flaws. One approach to this problem is to rate products on a vastly simplified scale using a rough checklist of the most egregious errors. The DHS and Mitre Corp. have built just such a system:
The Homeland Security Department and consulting firm Mitre Corp. on Monday unveiled a system for rating the protection of software products to help agencies, contractors and consumers ensure they are buying safe technology, in the same way the Energy Star labeling program helps guarantee eco-friendly purchases, DHS officials said.
The scoring reflects the degree to which software offerings defend against the most common programming flaws — which are widespread in agency systems as noted by a recent audit of Internal Revenue Service databases. Last week, the Treasury Department inspector general released a report that found software housing taxpayer information is not always protected against attacks.
I’m not convinced that programs like this will be successful at doing anything other than eliminating the obviously deficient. There’s some real value in eliminating the obviously deficient, but software security rating systems almost always imply that if a product has achieved the ‘best’ rating, then it’s ‘secure.’
Establishing that something is 100% secure is extremely difficult because of the nature of security, and any software security rating system needs to be clear about its limitations. This proposed system provides a metric-based rating from one to 100, which looks too much like a ‘percentage’ for security to me. I would prefer something different, perhaps even a binary rating to indicate that a product either has known security concerns or doesn’t have them.
Despite all their potential pitfalls, I suspect that software security ratings systems are the closest thing we’re likely to see as a way to quickly evaluate whether a product has serious security concerns.