The room is called the lab on the org chart and the cupboard by everyone who works in it. Four metres by three, second floor of a converted cutlery works in Sheffield, containing forty-one phones in a steel rack, each wired to a powered hub, each labelled with a strip of masking tape and a chipset code. The extraction fan runs constantly. It smells of warm plastic and, for reasons nobody has ever explained, cinnamon.
This is where Corvid’s operating system updates go to be disbelieved. The company sells about nine million handsets a year across four device families, which sounds manageable until you count the variants: six chipsets across the supported range, three modem firmware branches, and fourteen carrier configurations that each insist on their own dialler behaviour. Every quarterly release has to survive all of them.
Nadine Ferris has run releases here for five years and describes the job as professional pessimism. Her team’s build passes continuous integration in forty minutes and then spends three weeks failing to convince the cupboard. The automated suite covers perhaps sixty per cent of what matters. The rest is a rota: two engineers a day working through a printed checklist, making actual telephone calls to a landline in the kitchen.
A staged rollout is a confession
When the build finally ships, it does not ship. It goes to one per cent of eligible devices, weighted towards the newest hardware and the two carriers with the best telemetry. Forty-eight hours later, if nothing has moved, it goes to five. Then twenty-five, then the rest, with a hold over any public holiday, because an incident is a great deal worse when half the team is on a train.
The staged rollout exists because we know we have missed something. Anyone who tells you their release process catches everything has never been in the room when the modem team takes a phone call at six in the morning.
The halt criteria are written down, which is more than can be said at some far larger outfits, and they are deliberately blunt. Anything that trips them stops the rollout automatically, without a meeting. The meeting happens afterwards.
- Crash rate more than 0.4 percentage points above the outgoing build, on any single device family.
- Any regression in emergency calling, on any variant, at any rate above zero.
- Idle battery drain up more than eight per cent against the seven-day baseline.
- Two or more carriers independently reporting attach failures inside the same twelve-hour window.
That last criterion earned its place in November. A release reached four per cent before two operators in different countries filed near-identical reports of handsets dropping to 2G and staying there. The cause was a timer change three layers below anything Ferris’s team had touched, in a modem firmware branch that only shipped on one chipset. It never reproduced in the cupboard, because the cupboard sits on a single network and the fault needed a handover between two.
They bought a second SIM plan the following week. The rack now carries a small cell unit bolted to the underside of the top shelf, which lets the team force handovers on demand, and which Ferris calls the most useful two thousand pounds anyone here has ever spent. Forty-one phones, one fan, and a fault they will catch next time, along with, presumably, several dozen they still will not.
Leave a Reply