Firmware tests that run on every commit
A test that runs only when someone remembers is not much of a test. We set up suites that run on every commit, on real boards, and gate every release.
Key characteristics
- Runs
- On every commit, in your CI account
- Levels
- Unit, on-target and hardware-in-the-loop tests
- Update tests
- Power cuts, bad signatures, forced rollbacks
- You receive
- CI configuration, test suites, rig design, reports
Firmware bugs are expensive in a way web bugs are not. The code runs on devices you cannot reach, and the fix has to travel through the same update path the bug may have broken. Test automation is how a small team keeps shipping without dreading the next release. We build it in three layers, wire it into your CI, and make the update path the thing it tests hardest.
Three layers of tests
| Layer | Runs on | Catches | Typical time |
|---|---|---|---|
| Unit tests | A host computer or simulator | Logic errors in parsers, state machines, calculations and data migrations | Seconds |
| On-target tests | Your microcontroller on a development board | Driver behavior, timing, memory limits, RTOS configuration | Minutes |
| Hardware-in-the-loop | Your own board in a rig that can power it, feed its inputs and watch its outputs | System behavior, power, radio links and updates | Minutes to hours |
On Zephyr, the Ztest framework covers both unit and integration tests, and the Twister test runner builds them and runs them on emulators, simulators and real boards connected to a host. Twister has a mode intended for CI, and a hardware map lets one host drive several boards. On FreeRTOS or bare metal we set up a host-side C test framework and a small on-target harness that do the same job. Most of the value comes from the unit layer, because it is fast enough to run on every commit; the hardware layers catch what the host cannot see.
Hardware-in-the-loop rigs
A hardware-in-the-loop (HIL) rig is your board with a computer around it: a switchable power supply to cut power on command, a debug probe to flash and read it, a way to fake its sensors, a current measurement for sleep-current checks, and the link it normally uses, whether that is a phone, a gateway or a test access point. We design the rig with you, specify parts within your budget, write the scripts and document how to repair it. Rigs are hardware too, and flaky rigs teach teams to ignore failures, so we keep them as simple as the tests allow.
Firmware CI
Every commit builds every board variant from a clean checkout, runs the unit tests and reports the image size against the flash budget, so a slot that is filling up is visible months before it overflows. Merges to the main branch also run on-target and HIL tests. Release candidates run the full suite, including the update tests below, and no image reaches the signing step without passing. The configuration lives in your CI account, under your control, and keeps working after we leave.
Interrupted-update and rollback tests
The update path gets the hardest tests because it is the one feature that has to work when everything else has failed. Before a release reaches its first rollout wave, the rig runs:
- Power cuts at many points during the download, during the image swap and during the first boot of the new image. The device must come back on a working version every time.
- A deliberately broken image that fails its self-test, to confirm the bootloader reverts to the previous version. MCUboot’s test-then-confirm step makes this behavior testable.
- Wrong key and wrong version. An image signed with another key and an image with an older version number must both be refused.
- Updates from the oldest version still in the field, not only from the last release.
- A dropping link, to confirm the download resumes or restarts cleanly.
These match the test items in our free firmware release checklist, and they run for every release, not only the first. The update design they exercise is described in over-the-air updates.
Example (fictional). Loamwick’s requirement FW-003 says the LW-1 probe must accept signed updates from the gateway and roll back on a failed self-test. It is verified by an interrupted-update test that pulls power at 10 points in the transfer. Requirement FW-004, refusing images not signed with the production key, is verified by trying an image signed with a test key. Both run on every release candidate. The 1.1.0 release notes record that the probe now refuses images signed with the retired development key. See the example project.
Tests that reflect the field
Field failures usually come from conditions a desk never sees: cold batteries, weak radio links, gateways that restart, flash that is nearly full, a clock that drifts. We add a test for each field failure we fix, so it stays fixed, and the rig measures sleep current on every release candidate so a change that drains batteries is caught before it ships (see low-power design). Where a requirement has a written verification method, as in the firmware requirements worksheet, we automate that method wherever a rig can run it.
There is a regulatory reason too. For products in its scope, the EU Cyber Resilience Act asks manufacturers to apply effective and regular tests and reviews of a product’s security, with its main obligations applying from December 11, 2027. Automated tests with stored results are the simplest way to show that happens. This is not legal advice; see firmware security for the wider picture.
What you receive
- CI configuration in your account, building every board variant on every commit.
- Unit, on-target and hardware-in-the-loop test suites in your repository.
- The HIL rig design, parts list and scripts, with notes on maintaining it.
- A test report for every release candidate, kept with the release.
Where to start
If your team tests by hand before each release, start with a firmware review: we look at your code, your build and your release process and tell you in writing which tests would catch the most for the least effort. Request a firmware review, or read more in the prototype to production section of Field Update.
Send us your board and your plan.
A firmware review reads your schematics, code, flash layout and update plan, and ends with a written plan: what to build first, what could go wrong in the field and how updates will reach your devices.
Or email sales@fumbar.com, or call (720) 675-4202.