Over the past few months I’ve found myself thinking about change control a surprising amount. It’s not an exaggeration to say that change control is the only reason we can drive a reliable car, get to space, or take a commercial flight.
After prototypes have been tested and confirmed to work, change control is about building exactly the same thing, again and again. Any changes, including the tiniest tweaks, are carefully analyzed and tested. This approach has proven extremely successful and has become enshrined as one of the key tools in Engineering. Our society has even produced regulatory frameworks that rely on change control.
This was great in the past, but enter the current software environment. Software stacks are changing at a dizzying rate, with dozens of security issues being discovered and fixed every day. Security fixes in off-the-shelf software often prioritize security over other characteristics, such as performance and usability.
This leaves engineers who care about change control in a very difficult spot. They can’t simply ignore change control processes — nor can they ignore the security issues. The natural result is triaging by CVSS score to help rapidly sorting through the dozens of new CVEs that arrive every day.
I think it’s safe to say that the only long term answer to this problem is entirely automated testing, but saying that is too little too late for devices whose original design team didn’t have the insight to invest in automated testing.
My conclusions — though underwhelming — are this: (1) Automated testing is insanely important (2) As an industry, we need to improve CPEs. Without better CPEs, triaging criteria is likely to begin with ignoring any CVE with a CVSS score less than 7.0. (3) Better data from OS vendors on the performance impacts of patches — via a standardized method — would be amazingly helpful for device manufacturers.