A staged firmware rollout should move to the next wave only when the devices on the new release look like the devices still on the old one: they updated and confirmed, none went silent, resets and crashes per device-day did not rise, power use did not climb, and the product still did its job. Write those thresholds, the soak time and the name of the person who may halt before the first wave goes out. Small first waves catch releases that break everywhere; rarer faults show up only when larger waves are compared with the previous release over the same days. If any device cannot boot, or cannot take the next update, stop.
What a staged firmware rollout is for, and what it cannot do
A staged rollout limits how many devices a bad release can reach before you notice, and it gives you a group of devices to compare against. It does not find faults you are not measuring, and it does not replace testing before release.
The mechanics are simple. You offer a release to a small part of the fleet, wait, check what happened, then offer it to more. Google’s Site Reliability Workbook defines canarying as “a partial and time-limited deployment of a change in a service and its evaluation”. The last three words are the easiest part to skip. Percentages and schedules are easy to write down; the evaluation is where the work is.
Devices make that evaluation matter more than it does for a web service. A server release that misbehaves is replaced from a terminal. A device that stops talking after an update may need a site visit, and one that can no longer accept updates may stay broken for good.
AWS’s Well-Architected IoT Lens says this in its change management guidance. It asks that over-the-air updates be “controlled and reversible”, and it calls it “a good practice to test new firmware on a small group of devices” before the entire fleet is updated. If the parts underneath a rollout (bootloader, signing, two image slots, a tested revert) are not in place yet, start with what it takes to ship firmware you can safely update and come back here.
The terms this article uses:
| Term | Meaning here |
|---|---|
| Wave | A group of devices offered the release at the same time |
| Soak | The time you watch a wave before offering the release to the next one |
| Gate | The written check between waves, with three outcomes: go, hold or halt |
| Halt | Stop offering the release to devices that have not started it |
| Revert | The device’s bootloader returns to the previous image on its own |
| Control cohort | Devices still on the previous release, watched over the same days |
| Device-day | One device running one version for one day; the unit for rates |
Three limits deserve saying early. A staged rollout cannot catch a fault that takes longer to appear than your soak. It cannot see a signal your devices do not report. And it cannot make up for a release never tested against interrupted downloads, power loss during the swap and a forced self-test failure; those tests belong on the firmware release checklist.
Why rollout percentages fail a small fleet
Percentage ladders such as 1, 10, 50 and 100 percent suit fleets large enough that 1 percent is a useful sample. In a pilot fleet of 200 devices, 1 percent is two units, and two clean units tell you almost nothing about a fault that hits one device in a hundred.
The arithmetic is short. Suppose a release fails on a share p of devices, independently of each other. The chance that a wave of n devices shows at least one failure is 1 - (1 - p)^n. Figure 1 shows what that means for wave sizes a startup actually uses.
A first wave of 4 bench units has about a 1 in 5 chance of showing a fault that hits 5 percent of devices, and almost none of showing a 0.2 percent fault. A wave of 120 catches the 1 percent fault about 70 percent of the time. Only near 1,200 devices does a 0.2 percent fault show up reliably.
Turn the question around and the limits get sharper. If a wave of n devices shows zero failures, the highest failure rate still consistent with that result at 95 percent confidence is 1 - 0.05^(1/n). For larger waves this is close to 3/n, because the natural logarithm of 0.05 is about minus 3.
| A clean wave of | Failure rate it cannot rule out (95% confidence) |
|---|---|
| 4 devices | Up to about 53% |
| 12 devices | Up to about 22% |
| 24 devices | Up to about 12% |
| 40 devices | Up to about 7.2% |
| 120 devices | Up to about 2.5% |
| 300 devices | Up to about 1.0% |
So a clean wave of four bench probes proves only that the release is not broken everywhere. That gives each wave a different job:
- Early waves are smoke tests. They catch faults that hit every device of a kind: boot loops, an image built for the wrong hardware revision, a radio that never reconnects, an update agent that no longer works. Judge them on single devices. One bricked or unexplained silent unit is a halt.
- Later waves are measurements. They gather enough device-days to show rare faults and slow drifts. Judge them on rates compared with the control cohort.
The SRE workbook makes the same split: a small first stage relies on the clearest signs of trouble, such as crashes, and larger stages carry subtler metrics (Google SRE Workbook, chapter 16).
The update funnel: from offered to healthy
Count every device in a wave at each step between “offered” and “healthy”. Update servers report the early steps. The later steps, which decide whether the release is good, come from your own telemetry.
| Step | What it means | What typically goes wrong | Where the count comes from |
|---|---|---|---|
| Offered | The server has the release ready for this device | Device is offline or asleep | Update server |
| Downloaded | The whole image sits in the secondary slot | Interrupted transfer, retries | Device report |
| Verified | Signature, hash, version and hardware revision checks passed | Wrong key, wrong hardware revision | Device report |
| Installed | The bootloader copied or swapped the image | Power lost during the swap | First report after reboot |
| Booted | The new version is running | Crash at start-up, so the bootloader reverts at the next reset | Version reported after reboot |
| Confirmed | The image passed its self-test and marked itself good | Self-test fails, so the image is never confirmed | Device report |
| Reporting | The device checks in on its normal schedule | Device goes silent: radio, power or a hang | Telemetry |
| Healthy | Signals over the soak match the control cohort | Resets, power use or data quality drift | Telemetry |
| Updatable | The new version can accept the next release | The update agent itself is broken | A test update on bench units |
Build one view per release that shows this table per wave, plus each silent device with its last-seen time. A gate that needs numbers assembled by hand gets skipped when it matters most. Two steps deserve a closer look.
Confirmation is your first automatic gate
MCUboot, a secure bootloader for 32-bit microcontrollers that works with Zephyr and several other operating systems, can install a new image as a test. Its design documentation says that if a device crashes immediately after booting a bad image, “MCUboot will revert to the old (working) image at the next device reset”. That, in its words, “allows device firmware to make test swaps permanent only after performing a self-test routine”. In Zephyr, the application marks its image good by calling boot_write_img_confirmed() (Zephyr MCUboot API).
When the image confirms itself decides what the bootloader can catch for you. Confirm too early, straight after boot, and a release that hangs ten minutes later stays installed. Confirm too late, say after a full day, and an unrelated brownout during that day reverts a good release. A middle course suits most products: confirm after a self-test that reads every sensor, reaches the gateway or cloud, and completes one round trip with the update service.
The same document allows the first swap to be made permanent directly, in which case MCUboot “will never attempt to revert the images on the next reset”. For staged rollouts, keep the test swap. It turns every start-up crash into a counted revert instead of a dead device.
Updatable again is the gate people forget
The worst release breaks the update path itself, because no later release can reach the devices to fix it. Check it on the first wave: give the bench units a throwaway build with only the build number raised, and confirm they install it and mark it good. AWS’s IoT Lens recommends “a known-safe default firmware version” to fall back to; it helps only if the new version can still fetch it.
Five health signals to watch during each wave
Watch five families of signals, each compared with devices still on the previous release: update outcomes, silent devices, resets and crashes by cause, power use, and whether the product still does its job. Add the support queue once customers hold the devices.
The SRE workbook advises choosing the few metrics that best indicate real problems, “perhaps no more than a dozen”, because each extra metric needs its own definition of acceptable behavior. Five families with one or two numbers each stays well inside that.
1. Update outcomes
Count failures and reverts at each funnel step, per wave. A download that needed retries is not a failure. A rising retry count on one gateway model or one carrier, though, tells you where the next wave will struggle.
A verification failure on a device that should have accepted the image is a halt. It usually means a signing or packaging mistake, and every device will receive the same image. A revert is a halt in early waves and a rate in later ones.
2. Silent devices
A silent device is one that started the update and has not reported since, inside a window you set in advance. Set the window from the device’s normal reporting interval plus the expected transfer, swap and reboot time. For example, a device that reports every 15 minutes might be called silent after 60; a device that checks in once a day needs a window of days.
Platforms catch part of this. As of November 2026, AWS IoT Jobs lets you set an in-progress timer between 1 minute and 7 days, after which an unfinished execution moves to TIMED_OUT (AWS IoT Jobs configuration). That catches a device stuck mid-install. It does not catch one that installed, reported success and then went quiet.
Silence is the signal a revert cannot give you. A release that hangs after confirming, or breaks the radio after the first hour, never reverts and never reports. Count silent devices apart from reverts, and treat one in an early wave as a halt until someone has explained it.
3. Resets and crashes, by cause
Count resets per 1,000 device-days, split by cause, and compare with the control cohort. Zephyr’s hardware information API reads the chip’s reset cause flags, which tell why the device reset, “for example due to a watchdog timeout or due to power cycling”; each chip supports a different subset. Send the cause with the first message after every boot.
Splitting by cause is what makes the number useful. A rise in watchdog or fault-handler resets points at the release. A rise in brownout or power-on resets across both cohorts points at the weather, the batteries or the installer. What each device should capture when it crashes (fault registers, a short log, a coredump) is covered in field crash diagnostics; for gating, the count and the cause are enough.
4. Power use
A release that raises average current shortens battery life, and a two-day soak will not show it in a battery voltage reading. The arithmetic: if a release adds 6 microamps to the average current, it draws an extra 6 µA × 48 h = 288 µAh over a 48-hour soak. That is about 0.3 mAh, roughly 0.01 percent of a 3,000 mAh battery.
It still matters. The battery life calculator default example averages 24.0 µA and estimates 12.1 years. At 30 µA, the same arithmetic gives about 9.7 years, a fifth less, from a change nobody would see in two days of voltage data.
Watch the causes of power use instead. Have the firmware count milliseconds awake, radio transmissions and time connected per hour, report them with its telemetry, and compare each device with its own numbers on the previous release. On the bench units in wave 1, measure the current directly with a power analyzer across at least one full duty cycle.
5. Product function and people
Check that the product still does its job, and that the fix works. Look for data gaps, readings outside physical limits, readings that jump at the moment of the update, and whatever the release was meant to change.
Then read the support queue, and on a pilot, ask the site contacts. People notice what telemetry misses: a wrong LED pattern, slower pairing, a warm case.
| Signal | Measure it as | Compare with | Earliest it can show |
|---|---|---|---|
| Update outcomes | Failures and reverts per funnel step | Zero for never-events | Download and first boot |
| Silent devices | No report inside the set window | Zero in early waves | One reporting window |
| Resets and crashes | Resets per 1,000 device-days, by cause | Control cohort, same days | Once there are enough device-days |
| Power use | Awake time and radio events per hour | Same device, previous release | One full duty cycle |
| Product function | Data gaps, out-of-range readings | Control cohort, physical limits | One reporting cycle to weeks |
| People | Support tickets, site reports | Rate before the release | Next time users handle it |
Compare each wave with the previous release, not a fixed target
Judge each wave against devices still on the previous release over the same days, rather than against a fixed target or against the same devices last week. The control cohort shares the weather, the network and the cloud, so a difference between the cohorts is more likely to come from the release.
The SRE workbook is blunt about the alternative. Before-and-after evaluation is risky, “because time is one of the biggest sources of change in observed metrics”. Device fleets carry plenty of time-driven noise. A carrier’s maintenance night, a cloud deployment, a cold snap that sags battery voltage, or a gateway that loses power at one site changes the numbers for every device in range, whatever it runs.
A fixed target cannot tell those events from a regression. A control cohort can, because the event shows up on both sides. To compare cohorts of different sizes, normalize by exposure: count events per 1,000 device-days. A wave of 48 devices watched for two days gives 96 device-days.
Example (fictional). Ridgeline Meters, a fictional startup, has 320 cellular water meters in a pilot. Wave 1 was 8 of its own meters. Wave 2 offers release 2.3.0 to 48 more, while 264 meters stay on 2.2.4 as the control cohort. After 48 hours the team compares 96 device-days on 2.3.0 with 528 device-days on 2.2.4.
Modem and brownout resets: 5 on 2.3.0 (52 per 1,000 device-days) and 26 on 2.2.4 (49 per 1,000). A carrier maintenance window hit both cohorts on the second night. Compared only with the same 48 meters over their previous two days (1 reset, about 10 per 1,000), the release would have looked five times worse. Against the control it looks the same.
Watchdog resets: 3 on 2.3.0 (31 per 1,000) and 4 on 2.2.4 (7.6 per 1,000). At the control rate, 96 device-days should show about 0.7 watchdog resets. A Poisson calculation says 3 or more would happen by chance about 4 percent of the time. That is not proof of a regression, and it is not noise to wave away.
The verdict is hold: wave 3 waits, wave 2 soaks another 48 hours, and the firmware lead pulls the reset logs. If the resets trace to one code path in 2.3.0, the team halts and prepares 2.3.1.
The example shows why a gate needs three outcomes, not two. Go means the evidence is clean. Halt means a written rule was broken. Hold means the evidence is not clear yet, and the cheapest way to get more is to keep the current wave soaking without adding devices.
Two kinds of rule work together:
- Never-events, judged on single devices. A device that cannot boot, a silent device nobody can explain, a verification failure on an image that should have passed, a device that cannot take the next update. One is enough to halt, at any wave size.
- Relative rules, judged on rates. Resets by cause, power counters, data gaps and download retries, each compared with the control cohort over the same days, with a minimum exposure before the rule applies.
The workbook adds a caution that fits devices well: canary and control share infrastructure, so keep absolute measures alongside the comparison. If both cohorts get worse together, something else changed, and that deserves its own investigation.
For a fleet of a few dozen devices, cohort rates are too noisy to compare. Use each device as its own baseline instead. Compare its last seven days on the old release with its days on the new one, then check the control cohort over the same days to see whether the environment moved.
How long to soak each wave
Soak each wave until most of its devices have confirmed the release and then run through every cycle the product repeats. For most products that means at least a full day, often 48 hours. Measure the soak from when devices updated, not from when they were offered the release.
Four things set the minimum:
- Check-in interval. Polling devices take a release when they next contact the server. Mender’s documentation says the time a device receives a deployment is set by “the concrete moment the device contacts the server following its polling cycle”, not by the server. A fleet that checks in once a day needs a day before most of a wave has even started.
- The product’s own cycles. A solar gateway that charges by day and runs on battery at night, a probe that uploads a daily summary, a meter that calibrates weekly: the soak should cover at least one of each cycle the release touches. The SRE workbook gives the same rule for batch systems, where the canary should at minimum span the duration of a single work unit.
- Metric windows. If a metric is computed hourly or daily, the soak must be at least that long. The workbook asks that metric intervals be “either the same as or less than your canary duration”.
- People. Someone has to read the data and sign the gate. A soak ending on Saturday ends on Monday in practice, so start waves early in the week.
A worked rule, with hypothetical numbers: suppose 90 percent of a wave confirms within 12 hours of the offer because devices check in every few hours, and the longest cycle the release touches is a daily upload. The minimum soak is 12 + 24 = 36 hours. Round it to 48 hours so a person reads the results during a working morning.
What no soak will catch
Some faults take longer than any reasonable soak. A 32-bit millisecond counter wraps after 2^32 milliseconds, about 49.7 days. Flash wear from a logging change appears only after many erase cycles. A fix that proves itself after 30 days in wet clay cannot be judged in 48 hours.
These need different tools: bench units with the clock or event rate sped up, deliberate wrap-around and wear tests before release, and watching the fleet after the last wave. The rollout ends when the whole fleet has been offered the release; the watching does not.
Also run one rollout at a time. The SRE workbook strongly advises running only one canary deployment at a time, because overlapping canaries mix their signals. For firmware, that means no new release starts its first wave while the previous one is still soaking.
Designing waves for a pilot fleet
Size early waves by coverage, not by percentage. The first wave should contain at least one device of every hardware revision, connection path and starting firmware version in the fleet, ideally devices you can reach by hand. Later waves can grow three to five times each.
Coverage matters because a fault tied to one hardware revision fails every device on that revision and none of the others. If wave 1 holds no Rev C boards, it cannot show a Rev C fault, however clean it looks.
A coverage list for wave 1:
- Every hardware revision still in the field.
- Every connection path: each gateway model, each phone operating system the app supports, each cellular carrier or modem firmware.
- Every starting version the release must update from, including the oldest still in the fleet. An update from two releases back is a separate path with its own risks.
- The hard cases you can reach: a device on weak signal, one with a low battery, one at the edge of its temperature range.
- Devices you can physically reach within a day, in case one stops responding.
Example (fictional). Loamwick’s 40 LW-1 probes take release 1.1.0, which fixes moisture readings that drifted upward after 30 days in wet clay. The team writes the plan in its update policy before wave 1:
- Wave 1: Loamwick’s own 4 bench probes, covering Rev C and Rev D boards, updating from 1.0.2. Day 0.
- Wave 2: 12 trial probes, chosen to include every gateway and the plot with the weakest signal. Day 2.
- Wave 3: the remaining 24 probes. Day 4.
- Soak: 48 hours between waves, longer if a weekend falls inside.
- Halt: any probe that rolls back or goes silent.
The probes take a reading every 5 minutes, so a probe counts as silent if no reading arrives within 30 minutes after its update starts; the Bluetooth LE update time calculator puts transfer, verify, swap and reboot at 36.6 seconds. The bench probes also take a throwaway build with only the build number raised, to prove 1.1.0 can install the next release. Four clean bench probes rule out a release that breaks everything, not a fault in one probe of ten, so wave 2 still soaks for 48 hours.
The example project follows this release from the first review to the release report. The staged rollout planner does the wave arithmetic for larger fleets, and the Bluetooth LE update time calculator gives the transfer time the silent window is built on.
| Fleet | Wave 1 | Wave 2 | Wave 3 | Wave 4 | Judge early waves by |
|---|---|---|---|---|---|
| 40 probes (Loamwick, fictional) | 4 bench units | 12 | 24 | None | Single devices |
| 320 meters (Ridgeline, fictional) | 8 own units | 48 | 264 | None | Single devices, then rates against control |
| 12,000 devices (planner default, fictional) | 120 (1%) | 1,080 (to 10%) | 4,800 (to 50%) | 6,000 (to 100%) | Rates against control from wave 2 |
In the planner’s 12,000-device example, the halt line for wave 1 is more than 1 rollback among 120 devices, from a 1 percent threshold. That is a rate rule applied early, and Figure 1 shows its limit: at a true failure rate of 0.2 percent, a 120-device wave shows any failure only about 21 percent of the time. Pair it with the never-event rules.
Choosing devices for later waves
After wave 1, choose devices at random within each hardware revision and connection path, so each wave looks like the fleet. Do not fill the early waves with only friendly, well-connected sites. A clean result from the best sites says little about the worst ones.
Gateways and phones in the path
Many small products update through a relay: a gateway that receives the image over cellular and passes it to sensors over Bluetooth LE, or a phone app that carries it. When the gateway’s own firmware changes too, put whole gateways in a wave with all the devices they serve, so a gateway fault and a device fault can be told apart.
With a relay, the update server sees the gateway or the phone, not the end device, so the funnel counts must come from the relay’s reports. When a phone carries the update, the phone decides when the transfer happens, and the soak clock starts per device, not per wave.
Write the gates down before wave 1
Write a gate sheet for every release before the first wave: each signal, its go, hold and halt thresholds, the minimum exposure before it is judged, and who decides. Rules written in advance are applied the same way at 4 p.m. on a Tuesday and at 11 p.m. on a Friday.
| Signal | Go | Hold: keep soaking, add no devices | Halt: stop offering the release |
|---|---|---|---|
| Boot and confirm | Every updated device confirmed | A device still unconfirmed inside its window | Any device that cannot boot, or any revert in waves 1 and 2 |
| Silent devices | None | A device late, but inside twice its window | Any device past twice its window, unexplained |
| Image verification | No unexpected failures | Not used | Any failure on an image the device should accept |
| Watchdog and fault resets | Within 1.5 times the control rate | 1.5 to 3 times the control rate, or under 100 device-days | Over 3 times the control rate with at least 100 device-days |
| Awake time and radio events | Within 10% of the same devices on the previous release | 10% to 25% higher | More than 25% higher on bench units or on the wave average |
| Next update | Bench units installed and confirmed the test build | Not yet tried | Test build failed to install or confirm |
Three habits make a gate sheet work:
- Asymmetric authority. Anyone on the release rota may halt. Only the named owner may resume, and only with a written cause. Halting is cheap; resuming after a wrong call is expensive. The update policy worksheet has a row for exactly this decision.
- Minimum exposure. Every rate rule names the device-days it needs before it applies. Without that, the first reset in a 12-device wave reads as an 8 percent reset rate.
- A record. Each gate decision is written down with its numbers. Those records become the release report and the starting point for the next release’s thresholds.
Let the platform enforce a backstop
Update platforms such as AWS IoT Jobs can stop a rollout on their own when install failures cross a threshold. As of November 2026, the AWS IoT Jobs abort configuration takes a failure type (FAILED, REJECTED, TIMED_OUT or ALL), a threshold percentage, and a minimum number of devices that must be notified before the job can abort (AbortCriteria API reference). Set that minimum too high and the abort never fires in a small wave.
AWS’s rollout configuration can cap notifications per minute, or start at a base rate and multiply it as devices are notified or succeed; the documentation’s example goes from 50 per minute to 100, 200 and 400 (AWS IoT Jobs configuration). Microsoft’s Device Update for IoT Hub describes “gradual update rollout through device grouping and update scheduling controls”.
Use these features as a backstop, and know what they see. A rollout rate spreads notifications over minutes; it is not a soak. Abort criteria count job outcomes, not watchdog resets on day two or a 20 percent rise in radio time. To get waves with a soak between them, create one job or deployment per wave, aimed at a device group, and run your own gate in between.
What halting and rolling back actually do
Halting stops the release from reaching more devices; it does not undo anything on devices that already have it. What happens to those devices depends on whether they confirmed the new image, and whether the release raised the anti-rollback counter.
Be exact about what halt means on your platform. In AWS IoT Jobs, when abort criteria are met, “only QUEUED job executions are canceled”; executions already in progress are not canceled. Devices that were mid-download when you halted will finish, and they belong to the wave you are investigating.
| Situation | What the device does | What you do |
|---|---|---|
| Image never confirmed (start-up crash, failed self-test) | MCUboot reverts at the next reset | Count the revert; halt in early waves |
| Image confirmed, then misbehaves | Keeps running the new image | Halt; ship a fix or the fallback build |
| Release raised the security counter | Refuses images with a lower counter | Go forward only |
| Update agent broken on the new release | Cannot take any further update | Physical recovery |
| Fault outside the release (network, cloud) | Nothing changes | Hold, fix the cause, resume |
After confirmation, going back is a new release
Once a device has confirmed the new image, the bootloader treats it as the good one. Going back means sending another update. If the bootloader enforces downgrade prevention, that update must carry a version or security counter the device will accept.
MCUboot’s design notes that its security counter “does not need to increase with each software release”, and that an older image with the same counter value is accepted. That supports a simple policy: raise the counter only in releases that fix a vulnerability, and you keep the option of returning to any release since the last such fix. The document also notes that its other method, comparing version numbers, works only with overwrite-based updates. The full trade-off between recovering from a bad update and blocking downgrades is in rollback versus anti-rollback.
After confirmation, “roll back” therefore means one of two releases: the fix, or the previous release’s source rebuilt with a higher version number and signed again. Build that fallback before wave 1, run it through the same tests, and keep it ready. Writing and signing it at midnight during an incident is how a second mistake follows the first.
After a halt
- Freeze. No new waves or releases until the cause is known.
- Collect. Reset causes, logs and coredumps from affected devices, plus the same numbers from the control cohort.
- Classify. A release fault, an outside cause, or a testing gap.
- Decide. Resume, ship the fix, or ship the fallback build, and record who decided.
- Write it down in the release report: what happened, how many devices, and what changed in the gates or tests.
A rollout checklist for your next release
Work through these steps before the next release leaves your build server. The telemetry takes longest, so start there.
- List the fleet by hardware revision, connection path and firmware version: the coverage list for wave 1.
- Check the telemetry: version, confirmation state, reset cause, awake time, radio events and a heartbeat from every device.
- Set the confirm point after a self-test that reaches the update service.
- Pick wave 1 for coverage, and add the test build that proves the release can take the next one.
- Size later waves and the soak with the staged rollout planner.
- Write the gate sheet: never-events, relative rules with minimum exposure, and go, hold and halt.
- Name the owners in the update policy worksheet: who may halt, and who may resume.
- Build the fallback release and test it like the release itself.
- Run the firmware release checklist, including the interrupted update and the forced self-test failure.
- Keep watching after the last wave, and write the release report.
Fumbar is a Denver firm that writes embedded firmware for hardware startups and builds the over-the-air update path that keeps it improving after devices ship. In its sustaining releases work, every release goes out in waves with a soak and a written halt rule, and the release report records which devices got it in which wave and how many updated, reverted or went silent. The over-the-air updates service sets up the bootloader, signing and rollout plan underneath. If you want a second reader for your rollout plan, a firmware review covers the update plan along with the code and the flash layout.
