Mozilla blames ‘interlocking complex systems’ and confusion for Firefox’s May add-on outage

Mozilla blames ‘interlocking complex systems’ and confusion for Firefox’s May add-on outage

Mozilla has issued multiple after-action reports analyzing the major mix-up in May that crippled most Firefox add-ons. The reports also made recommendations for preventing similar incidents in the future.

The fiasco started just after 8 p.m. ET on Friday, May 3, when a certificate used to digitally sign Firefox extensions expired. Because Mozilla had neglected to renew the certificate, Firefox assumed add-ons could not be trusted – that they were potentially malicious – and disabled any already installed. Add-ons could not be added to the browser for the same reason.

Mozilla rushed a stop-gap fix to the browser via its Studies system, infrastructure normally responsible for pushing test code to small groups or collecting data on reactions to sponsored content. Because the Studies approach did not reach everyone, on May 5 and May 7 Mozilla shipped two Firefox updates – 66.0.4 and 66.0.5 – that addressed the certificate mess.

“The first question that everyone asks is, “How did you let this happen?'” wrote Firefox’s CTO Eric Rescorla in a post to a company blog. “At a high level, the story seems simple: we let the certificate expire. This seems like a simple failure of planning.”

Rescorla disputed that characterization, however. Saying that the situation was “more complicated” than that, he said the responsible team knew the certificate was expiring but assumed that the browser would ignore the expiration date because in an earlier incident certificate checking had been disabled. “This led to confusion about the status of intermediate certificate checking. Moreover, the Firefox QA plan didn’t incorporate testing for certificate expiration and therefore the problem wasn’t detected. This seems to have been a fundamental oversight in our test plan.”

Others covered the crisis from different angles in separate postmortems, including an incident report and a technical report.

Copyright © 2019 IDG Communications, Inc.


Leave a Reply

Your email address will not be published.