Set up automated firmware testing in this order: host-side unit tests for your logic, a CI build with static checks on every commit, simulation where it is cheap, then one self-hosted CI runner wired to real boards, and finally a fuller rig for release candidates. A hardware-in-the-loop (HIL) rig is worth building as soon as you have an update path to protect, more than one hardware revision in use, or a battery life you have promised someone. Those failures only show on real boards. For a product that updates over the air, that point arrives before the first field trial, because the trial is where the first field update ships.
This article is for the first firmware engineer or CTO who tests by hand today: flash a board, poke at it, watch the log, ship. It gives each stage in order, what the stage catches, what it misses, the tools that do the job on Zephyr, FreeRTOS and bare metal, and a rough idea of the effort. Tool versions and documentation were checked in November 2026.
What each kind of firmware test catches, and what it cannot
Each layer of testing sees a different slice of the product, so the question is never “unit tests or HIL” but which bugs each one can see. Cheap layers run on every commit and catch logic errors; expensive layers run less often and catch the failures that need real silicon, real power and real radio.
Four terms, defined once:
- A unit test calls one module of your code with chosen inputs and checks the outputs. It usually runs on a host computer, not the microcontroller.
- An on-target test runs test code on the real microcontroller, often on a development kit, and reports results over a serial port or debug probe.
- A hardware-in-the-loop (HIL) test runs your shipping firmware on your own board while a computer controls the board’s surroundings: it switches power, feeds sensor inputs, plays the other end of the radio link and measures current.
- A self-hosted runner is a computer you own that runs jobs for your CI service. It is how a cloud CI pipeline reaches boards on a bench.
The table below maps common firmware bug classes to the stage that can catch them. Read it from left to right: if an earlier stage can catch a bug, catch it there, because it will be faster and cheaper on every commit.
| Bug class | 1 Unit tests | 2 Build and static | 3 Simulation | 4 Real boards in CI | 5 Release rig |
|---|---|---|---|---|---|
| Wrong result from a parser, state machine or calculation | Yes | Partly | Yes | Yes | Yes |
| Build breaks for another board variant | No | Yes | No | Only boards on the bench | Only boards on the bench |
| Image grows past its slot | No | Yes | No | Yes, late | Yes, late |
| Null dereference, leak, undefined behavior | Partly, with sanitizers | Partly | Partly | Rarely visible | Rarely visible |
| Driver writes the wrong register or misreads a sensor | No | No | If the peripheral is modeled | Yes | Yes |
| Stack overflow, priority or timing bug under load | No | Partly | Partly | Yes | Yes |
| Sleep current higher than last release | No | No | No | With a current meter | Yes |
| Radio link drops, reconnect fails | Logic only | No | Partly | Yes, on one bench | Yes, with stress |
| Flash wears out from too many erases | Arithmetic only | No | No | No | With a soak test |
| Power cut during an update leaves a dead device | No | No | Bootloader logic only | Yes | Yes, at many points |
| Wrong key or older image accepted | No | No | Partly | Yes | Yes |
Two patterns stand out. Logic bugs can be caught early and cheaply, so there is no reason to wait for a rig before testing them. Power, radio, flash wear and update failures need hardware, so no amount of host testing replaces the rig for them.
Stage 1: host-side unit tests for the logic you can separate
Start with unit tests that compile and run on a laptop and on any CI machine, because they cost seconds per run and cover the code most likely to be wrong in subtle ways. A small team can usually have the first test running in a day, as a rough planning guess, if the code to test is not tangled with hardware access.
Which code to test first
Not all firmware is worth unit testing. Test the code where a wrong answer is plausible and hard to see on the bench:
- Parsers and encoders: messages from the gateway or phone, configuration blobs, sensor frames, anything with a length field.
- State machines: connection handling, provisioning flows, the update agent’s decisions about when to download, verify and reboot.
- Calculations: sensor calibration and compensation, battery estimates, scheduling arithmetic, unit conversions.
- Data migrations: code that reads settings written by an older firmware version.
Drivers that mostly move bytes to registers are poor unit test targets. They belong to on-target and HIL tests, where the register is real.
Tools by platform
For C on any platform, Unity is a common choice. Its authors describe it as written in “100% pure ANSI C”, delivered as “a single C file and a pair of header files”, and portable from small 8-bit parts to 64-bit hosts. It is MIT licensed, and its latest GitHub release as of November 2026 is v2.7.0, published July 16, 2026.
CMock “generates pure C code from regular C headers”: it reads a header and writes a fake module, so your test can check which functions your code calls and with what arguments. CMock uses Ruby scripts to generate those modules. Ceedling ties Unity and CMock together, generates the test runners and, per the Unity site, installs Unity automatically.
On Zephyr, the Ztest framework “can be used in two ways, either as a generic framework for integration testing, or for unit testing specific modules”. Its documentation also covers mocking with FFF (the Fake Function Framework). Zephyr’s native_sim board compiles your application “together with the Zephyr kernel, and libraries, creating a normal Linux executable”. The board “does not intend to simulate any particular HW”, but its documentation describes runs with AddressSanitizer, UndefinedBehaviorSanitizer and coverage reports, which turn silent memory bugs into failing tests.
On FreeRTOS, the kernel repository includes a POSIX port in which “each task has a pthread” and the tick interrupt is driven by a Unix signal. That lets you run task-level logic on a Linux host, with the caveat that timing on a host bears little relation to timing on the chip.
Make the code testable without rewriting it
The usual obstacle is code that reads a register in the middle of a calculation. The fix is a thin seam: a function pointer, a small interface header or a link-time substitute that the test can replace. Keep seams few and placed at hardware boundaries; portable firmware architecture covers where those seams pay for themselves beyond testing.
Example (fictional). Loamwick’s LW-1 probe took a reading every 5 minutes, and version 1.1.0 fixed moisture readings that drifted upward after 30 days in wet clay. The fix lived in one compensation function, so the team pinned it with a host test. They replayed a recorded 30-day trace (30 days at 12 readings an hour is 8,640 samples) through the function and asserted that the drift stayed within a set bound. The test runs in well under a second on every commit, so a later change that reintroduces the drift fails before anyone flashes a board.
void test_drift_stays_small_over_30_days_in_wet_clay(void)
{
const trace_t *t = trace_wet_clay_30d(); /* 8,640 samples */
float drift = moisture_drift_pct(t);
TEST_ASSERT_FLOAT_WITHIN(0.5f, 0.0f, drift);
}
Add a test for every field bug
The most valuable unit tests are written the day a bug is found. Before fixing it, write the test that fails, then make it pass. Over a year this produces a suite shaped by your product’s real failures rather than by a coverage target.
Stage 2: build and static checks on every commit
The second stage builds every firmware variant from a clean checkout on every commit, runs the unit tests and fails the commit on warnings, analysis findings or an image that no longer fits. It needs no boards, so it can run on the hosted runners your CI service provides; as a rough planning guess, a first version takes a day or two.
What the build job should do
- Build every board and hardware revision you support, not only the one on your desk. A change that compiles for Rev D and breaks Rev C should fail here.
- Treat compiler warnings as errors for your own code. Vendor SDK code may need its own exceptions, written down.
- Run the unit tests from stage 1 and publish results in JUnit XML, the format most CI services display.
- Report image size against the slot. Print the image size, the slot size and the headroom on every build, and fail when headroom drops below a limit you choose.
- Keep the artifacts: the image, the map file and the exact toolchain version, so any later failure can be tied to one build.
The headroom check matters more than it looks. In the flash budget calculator’s default example, a 412 KB image fills 92.8% of a 448 KB slot, and the largest image that fits is 444 KB. At that point every feature has to be weighed in kilobytes, and a CI line that prints the number on every commit is the cheapest warning you can get.
Static analysis that a small team will read
GCC’s -fanalyzer option “enables an static analysis of program flow” and warns about problems on paths through the code. GCC’s own documentation notes it is “much more expensive than other GCC warnings”, so many teams run it in a separate job rather than on every local build.
Zephyr’s build system selects an analysis tool with the ZEPHYR_SCA_VARIANT setting. Its static code analysis documentation lists native support for tools including the Clang static analyzer, CodeChecker, Coverity, GCC static analysis and Sparse.
The rule that keeps static analysis useful is simple: start with few checks and a clean baseline, then fail the build on new findings only. A report with 400 old warnings that nobody reads catches nothing.
Zephyr’s runner already speaks CI
If you use Zephyr, its test runner, Twister, has an integration mode that its documentation describes as “used in continuous integration (CI) and other automated environments used to give developers fast feedback on changes”. Twister writes twister.xml, a “JUnit XML summary suitable for CI systems”, and with --footprint-report a ROM and RAM footprint report. The same tool returns at stage 4 to drive real boards.
Hosted runners have limits worth knowing. On GitHub Actions, each job on a GitHub-hosted runner can run for up to 6 hours, according to GitHub’s Actions limits page. That is ample for builds and unit tests, but not for soak tests.
Stage 3: simulation where it is cheap
Simulation runs your firmware, or a close build of it, on a model of the chip or board inside a CI job, so you can test boot, scheduling and some peripheral behavior without hardware. Use it where a model of your part already exists; skip it where you would have to write the model yourself.
The options
- Zephyr native_sim runs the whole application, kernel included, as a Linux program. It is the cheapest simulation for Zephyr projects and the natural home for integration tests of tasks, queues and protocol logic.
- BabbleSim boards in Zephyr simulate Nordic radio parts. The nrf52_bsim target simulates an nRF52833, using BabbleSim “to simulate the radio activity” between simulated devices, with models of peripherals such as the radio, timers, flash controller and real-time counter.
- Renode, an open source simulation framework from Antmicro, runs unmodified binaries on modeled boards and is “integrated with the Robot Framework testing suite”. One command,
renode-test, starts Renode, runs a test file and writes a machine-readable result file plus HTML logs. As of November 2026 its latest GitHub release is 1.17.0, published September 7, 2026. - QEMU runs some Cortex-M and other targets. Twister can run tests on QEMU platforms, and Espressif’s pytest-embedded has a QEMU service.
What simulation misses
Simulators model what someone chose to model. Renode’s own documentation says its goal is to run binaries “without modification, although of course your specific use case may require extending the provided hardware description / models”. The nrf52_bsim page is blunter: “unlike a real nrf52 device, the nrf52_bsim has unlimited RAM and flash for code.”
That second sentence is the reason simulation sits third, not first or last. A simulated run will not tell you the image no longer fits, that sleep current rose, or that a sensor needs 3 ms more to wake up on the real board. It is very good at telling you that the scheduler, the protocol logic and the boot sequence still work after a change.
When to skip this stage
Skip simulation if your chip has no existing model and your team would spend weeks writing peripheral models. Go straight to stage 4, where a development kit on a bench gives you the real peripheral for the price of the kit. Simulation pays off when a model exists already, when you need to test multi-device radio logic without a room full of boards, or when you want many test runs in parallel.
Stage 4: one self-hosted runner wired to real boards
The fourth stage is the first HIL step: one small computer on a bench, registered as a CI runner, with one or two of your boards attached, flashing and testing every merge to the main branch. As a rough planning guess, building the first station takes one to two weeks of an engineer’s time, most of it on reliable power switching and test scripts rather than on CI configuration.
How the CI side works
GitHub’s documentation defines a self-hosted runner as “a system that you deploy and manage to execute jobs from GitHub Actions on GitHub”. Self-hosted runners “are free to use with GitHub Actions, but you are responsible for the cost of maintaining your runner machines”. GitLab Runner offers the same pattern with self-managed runners.
A runner receives default labels for its operating system and architecture, plus any custom labels you add. Give the bench computer a label such as ls1-rig, and only jobs that ask for that label will run on it.
Because the boards are a shared physical resource, only one job should use them at a time. GitHub’s concurrency setting allows “at most one running job or workflow in a concurrency group at any time”. By default, though, a newer pending job replaces an older pending one. Setting queue: max lets up to 100 jobs wait their turn instead, which is usually what you want for test results on every merge.
hil:
needs: build
runs-on: [self-hosted, linux, ls1-rig]
concurrency:
group: ls1-rig
queue: max
timeout-minutes: 60
steps:
- uses: actions/checkout@v7
- run: west twister --device-testing --hardware-map rig/map.yml -T tests/hil
Security rules for a runner on your network
GitHub’s secure use reference says self-hosted runners “should almost never be used for public repositories”, because anyone can open a pull request and run code on the machine. It also warns that on private repositories, anyone who can fork and open a pull request can compromise the runner and reach its secrets.
For a firmware bench, three rules follow:
- Keep the rig on a private repository, and run hardware jobs only for branches your team pushes, not for outside pull requests.
- Never store production signing keys on the rig. Test images should be signed with a test key that production devices refuse.
- Put the bench computer on its own network segment if it sits on the office network.
How the board side works
On Zephyr, Twister flashes and runs tests on connected boards. Running west twister --generate-hardware-map map.yml lists each attached board with its serial port, probe and flashing tool; you then fill in the platform names. With that map, west twister --device-testing --hardware-map map.yml builds tests for those platforms and runs them on the boards. The documentation notes that hardware map support currently covers boards flashed with pyocd, nrfjprog, J-Link, OpenOCD or dediprog.
The hardware map also records fixtures: extra wiring or instruments attached to a board, such as a loopback cable or a sensor. A test that needs a fixture runs only on a board that has it, and run_with_fixture_only reserves a board for those tests.
For tests that need a script rather than a pass or fail from the device, Zephyr’s pytest integration adds fixtures named dut, shell and mcumgr. The documented mcumgr example uploads a signed image, marks it for a test boot and resets the device: the core of an automated update test.
On Espressif chips, pytest-embedded is “a pytest plugin that has multiple services available for various functionalities”, designed for embedded testing. Its services cover serial ports, ESP-IDF builds that it flashes automatically, JTAG through OpenOCD and GDB, and QEMU or Wokwi in place of a real target. On FreeRTOS or bare metal elsewhere, the same pattern is a small pytest or Robot Framework suite that flashes with the vendor’s command-line tool and reads the serial log.
What goes on the bench
A first station needs fewer parts than vendor HIL systems suggest:
- A runner computer. Any small Linux machine with enough USB ports.
- A debug probe per board to flash, reset and read faults. Use the probe your Twister or vendor tools already support.
- A serial connection for logs and test output.
- A switch on the board’s supply, controlled by the runner, so a test can cut power at an exact moment. A relay or a load switch on a small controller board both work; it must switch the battery input, not only USB.
- The other end of the link: a gateway board, a second development kit acting as a phone, or a test access point.
- Optional at first: a current meter that logs microamps, and a small board that fakes sensor inputs with a DAC or GPIO lines.
Example (fictional). Loamwick’s first station holds one Rev C probe, one Rev D probe and one gateway board, driven by a small Linux computer on the bench. The hardware map lists both probes with the fixtures power_switch and gateway_link; only the Rev D probe has current_meter. On every merge, the station flashes both probes, runs the on-target test suite and checks requirement FW-005 from the firmware requirements worksheet: the gateway restarts 20 times and each probe must reconnect within 10 minutes. The Rev C board is there because Rev C has a known temperature offset above 40 °C, and testing only the newest revision would hide revision-specific bugs.
What only real hardware shows: power, radio, timing and flash wear
Four classes of failure need a real board to appear, and they are the reason a HIL rig exists. Each can be turned into an automated check once the rig has the right instrument attached.
Power: sleep current that creeps up
A peripheral left enabled, a pull-up fighting an input or a debug interface left on can multiply sleep current without changing any visible behavior. Nothing fails until batteries die early in the field. The test is a measurement: put the board to sleep, record current for a fixed window, and fail when the result exceeds the last release by more than a set margin.
The stakes are easy to show with the battery life calculator’s default example: a 3,000 mAh cell, 4 µA sleep current and a 24.0 µA average give an estimated 12.1 years. For example, suppose a change raises sleep current tenfold, to 40 µA, with everything else equal. The average becomes about 59.9 µA and the estimate falls to about 4.9 years. That is a hypothetical calculation, but it shows why sleep current deserves a line in every release report.
Radio: links that drop and do not come back
Simulators model packets; they do not model a gateway that reboots mid-connection, a phone that changes its connection interval or a crowded 2.4 GHz band. The bench version of these tests is crude but effective: restart the other end of the link repeatedly, move it out of range with an attenuator or by switching it off, and count reconnects and the time each takes. A fixture on the hardware map marks which station has the link partner.
Timing: what the clock and the interrupts really do
On a host, every timer is perfect and interrupts never collide. On the board, a crystal has a tolerance, a sensor takes longer to wake when cold, and a radio interrupt can delay a sensor read. Twister’s hardware runs and pytest scripts can check durations from the device’s own timestamps, for example that a 5-minute reading interval stays within tolerance over a day.
Flash wear: erases add up
Flash pages survive a limited number of erases. Zephyr’s NVS documentation uses an nRF51822 as an example, with pages that “can be erased about 20,000 times”, and gives a formula for expected life: the number of sectors times the sector size times the erase limit, divided by the bytes written per minute (data plus 8 bytes of metadata per write).
Example (fictional). The LW-1 keeps settings in 32 KB of flash: eight 4 KB sectors managed by Zephyr’s NVS. Suppose the datasheet rates each page for 10,000 erases (check your own part). Writing a 16-byte record every 5 minutes uses 24 bytes per write, or 4.8 bytes a minute, and the NVS formula gives about 68 million minutes, roughly 130 years. Now suppose a bug writes that record every second instead. The same formula gives about 228,000 minutes, roughly 158 days. A 72-hour soak test that reads an erase counter from the device would see about 190 erases per sector in the buggy build and less than one in the correct build. No unit test would notice; one counter on the rig does.
Stage 5: release-candidate runs on a fuller rig
The last stage runs before a release candidate may be signed with the production key: the full on-target suite on every supported hardware revision, the update-path tests, power and radio stress, and a soak test that runs for hours or days. It is the only stage that answers “is this build safe to send to devices we cannot reach?”
Update-path tests, automated
The update path is the feature that must work when everything else has failed, so it gets the hardest tests. MCUboot’s design documentation explains that “test swaps are supported to provide a rollback mechanism to prevent devices from becoming ‘bricked’ by bad firmware”, and that firmware makes a test swap permanent “only after performing a self-test routine”. It also records swap progress in flash so an interrupted swap can be resumed after a reset. A rig turns those design promises into checks:
- Power cuts during transfer. Start an update and cut power at several points in the download. After each cut, the device must boot the old version and be able to retry.
- Power cuts during the swap and first boot. Cut power while the bootloader is copying images and while the new image runs its self-test. The device must come back on a working version every time.
- Forced revert. Install an image built to fail its self-test and confirm the device reverts at the next reset.
- Refusals. An image signed with a development key, an image with a lower version and an image built for another hardware revision must each be refused.
- Oldest version first. Update from the oldest version still in the field, not only from the last release.
The MCUboot project runs its own simulator, written in Rust, which is “designed to exercise the mcuboot upgrade code, specifically testing untimely reset scenarios”. That tests the bootloader’s logic. It does not test your flash part, your power supply’s brown-out behavior or your update agent, which is why the same scenarios belong on your rig.
Example (fictional). Loamwick’s requirement FW-003 says the probe must accept signed updates from the gateway and roll back on a failed self-test, verified by pulling power at 10 points in the transfer. FW-004 says it must refuse images not signed with the production key, verified with an image signed with a test key. On the rig, both run for every release candidate: the runner splits the transfer into 10 equal points, cuts power at each one in turn, and checks over the serial log which version booted. The LW-1 self-test (sensors in range, gateway connected, watchdog quiet) decides whether the new image confirms itself.
Soak tests and their time limits
A soak test runs the release candidate in normal operation for a long period, watching for crashes, resets, memory growth, flash erases and battery drain. GitHub’s limits allow a job on a self-hosted runner to run for up to 5 days, and a job can wait in the queue for up to 24 hours before it is cancelled. A 72-hour soak fits; a month-long one should run outside CI and report back.
Where testing ends and rollout begins
A release that passes the rig is ready for real devices, not for the whole fleet. The first rollout wave is the next test, with your own bench units first. Staged firmware rollouts covers wave sizes, soak time between waves and the signals that stop a rollout. The firmware release checklist lists the release gates, including the interrupted-update and rollback tests above.
Keeping a test rig trustworthy
A rig that fails for its own reasons teaches the team to ignore red builds, which is worse than having no rig. Treat the rig as a product with its own tests, its own owner and its own repair notes.
Separate rig failures from firmware failures
Start every hardware job with a rig self-test: probe attached, serial port open, power switch toggles, board answers. If that fails, mark the job as an infrastructure error and alert the rig’s owner, not the author of the commit. Report the self-test as its own test case in the JUnit output, so the CI page shows at a glance whether the rig or the firmware failed.
A flaky test policy
A flaky test passes and fails on the same code. Write the policy down before the first one appears:
- A test that fails once and passes on retry is logged as flaky, not ignored.
- Flaky tests move to a quarantine list that still runs but does not block merges, with a named owner and a date.
- Nothing stays quarantined for more than two weeks; it is fixed or deleted.
Keep results with the release
Store the JUnit reports, logs and current measurements with each release candidate. When a field problem appears months later, you can see what the rig measured for that exact build.
There is a regulatory reason too. For products in its scope, the EU Cyber Resilience Act requires manufacturers to “apply effective and regular tests and reviews of the security of the product with digital elements” (Regulation (EU) 2024/2847, Annex I, Part II). The regulation applies from December 11, 2027, with its reporting obligations applying since September 11, 2026. Stored, automated test results for refusals, signatures and rollback are a simple way to show tests happen. This is not legal advice.
When is a full HIL rig worth building?
A single bench station is worth building early, usually during EVT (engineering validation test) boards, once the bootloader and update path exist. A fuller rig, with several stations, every hardware revision and stress instruments, is worth building when the cost of one bad release clearly exceeds the cost of the rig and its upkeep.
Use these triggers. If two or more are true, build the fuller rig before the next release that reaches customers:
| Trigger | Why it matters |
|---|---|
| Devices update over the air and are hard to reach | A failed update can strand devices you cannot recover by hand |
| Two or more hardware revisions in the field | A change can pass on one revision and fail on another |
| A promised battery life or power budget | Sleep current regressions are invisible without measurement |
| Releases more often than once a quarter | Manual regression testing cannot keep pace with the release rate |
| A radio link to a phone or gateway you do not control | Reconnect and interval behavior changes with the other side |
| Customers or regulators ask for test evidence | Stored rig results are evidence; memories of bench sessions are not |
And reasons to wait:
- The hardware will change substantially within weeks. Build one station on a development kit and keep the rig wiring simple.
- The product is a single-purpose device with no field updates and no radio. Unit tests, a build job and a factory test may be enough.
- Nobody will own the rig. An unowned rig decays; assign an owner before you buy parts.
Effort rises at each stage, so the order matters. As rough planning guesses, not measurements: stage 1 is a day to start; stage 2 a day or two; stage 3 a few days if models exist and much more if they do not; stage 4 one to two weeks for the first station; stage 5 grows with each field failure you add a test for.
A 30-day plan for a small firmware team
Do not wait for the rig to start testing. The stages above fit into about a month for one engineer working part time on tests, as a rough guide, and each week leaves something running that keeps catching bugs.
- Week 1: unit tests. Pick Unity and CMock, or Ztest on Zephyr. Write tests for one parser, one state machine and the update agent’s decision logic. Run them locally.
- Week 2: CI build. Build every board variant on every commit, fail on warnings, run the unit tests, and print image size against slot size. Add one static analysis job with a clean baseline.
- Week 3: first bench station. Register a self-hosted runner on a private repository, attach one board, a debug probe, a serial line and a power switch. Flash and run a smoke test on every merge to main.
- Week 4: update tests. Automate power cuts during transfer and swap, a forced revert and refusal of a test-key image. Run them on every release candidate and store the results.
After that, add a current meter, a second hardware revision and a soak test as the product needs them. Write a new test for every field bug, and keep the rig’s repair notes next to its scripts.
If you are moving from prototype code to a product, prototype firmware to production covers the code decisions that come before the test plan. Fumbar’s test automation work sets up these stages in the client’s own CI account, and its firmware development work writes the code to be testable from the start. If you would like a second reader for your current setup, a firmware review looks at the code, the build and the release process and ends with a written plan that names the tests worth automating first.
