Before the first fleet ships, build in five things: a watchdog fed only when every critical task shows progress, the reset cause read and reported at every boot, a small fault record saved before the device resets, a build ID that ties every report to the exact binary, and an upload path that sends those records within the battery and data budget. Add a full coredump where flash and bandwidth allow. With that kit you can count, classify and fix field crashes from a desk, and the same numbers decide whether a staged rollout continues or stops.
Fumbar is a Denver firm that writes embedded firmware and over-the-air update systems for hardware startups. This guide is the minimum diagnostic kit we would ask for in any connected product, described with the features Zephyr RTOS and Espressif’s ESP-IDF provide for each part. Every fact below was checked against the project or vendor documentation in November 2026, and the examples are labeled fictional.
The minimum crash kit to build in before the first fleet
The minimum kit answers three questions for every reset in the field: why the device reset, where the code was when it failed, and which exact build it was running. Five pieces of firmware answer them, plus an optional coredump, and all of them are cheap if designed in before launch.
Register-level tutorials on decoding a Cortex-M HardFault assume an engineer with a debug probe attached to the failing board. A field device has no probe attached, its debug port should be locked, and the failure may happen once a month on one unit out of hundreds. Whatever the firmware does not save before it resets is gone.
| Piece | Question it answers | Typical cost | Zephyr feature | ESP-IDF feature |
|---|---|---|---|---|
| Task-aware watchdog | Did a task stop making progress? | A timer and a few bytes per channel | Task watchdog with hardware fallback | Task watchdog (TWDT) and interrupt watchdog (IWDT) |
| Reset cause at boot | Why did the last reset happen? | One register read | Hardware information API (hwinfo) |
esp_reset_reason() |
| Fault record | Where did the code fail, and in which state? | Tens of bytes of RAM that survive a reset | Custom fatal error handler plus the retention system | Panic handler output, core dump summary |
| Build ID | Which binary produced this report? | 20 to 32 bytes in the image | GNU linker --build-id |
App ELF SHA256 in esp_app_desc_t |
| Upload path | How does the record reach you? | A message per boot, more on request | Your transport, plus the core dump API | Your transport, plus core dump in flash |
| Coredump (optional) | What was in memory? | Kilobytes of flash per dump | Core dump module | Core dump to flash or UART |
One default deserves attention before anything else. Zephyr’s fatal error documentation says the default handler “calls LOG_PANIC() to dump all pending logging messages and then unconditionally halts the system”. On a bench that is helpful. In the field, a halted device sits silent until a watchdog resets it or someone pulls the battery. Decide what your product does after a fatal error, write it into your own handler, and test it.
ESP-IDF chooses differently. Its fatal errors guide lists “Print registers and reboot” as the default panic behavior, which prints the registers and backtrace, then restarts the chip. That output goes to the console unless the core dump feature saves it to flash, so a field build needs one of them configured before launch.
Watchdogs that catch hangs, not just a stuck main loop
A watchdog only detects what its feeding rule can see. Feed it only when every critical task has shown progress within its own deadline, keep a hardware watchdog underneath in case the scheduler itself stops, and record which task missed its deadline before the reset.
The Zephyr task watchdog documentation states the basic rule: once initialized, a hardware watchdog “has to be restarted (‘fed’) in regular intervals to prevent it from timing out”, and if the software stops feeding it, “the corrective action is triggered”. It then names the weakness of a single watchdog: in a system with several tasks, “a single watchdog instance may not be sufficient anymore, as it can be used for only one task.”
Feed on progress, per task
The common mistake is feeding the watchdog from a periodic timer interrupt or from the top of one loop. A timer interrupt keeps running while the radio task is deadlocked, and the loop may keep spinning while the sensor task is stuck. The watchdog stays fed and the device stays useless.
Zephyr’s task watchdog addresses this with channels. Each critical thread calls task_wdt_add() with its own reload period and calls task_wdt_feed() when it completes a unit of real work: a sensor reading stored, a message acknowledged, a state machine step finished. The task watchdog is built on a kernel timer, and “An existing hardware watchdog can be used as an optional fallback if the task watchdog itself or the scheduler has a malfunction.” Each channel can register a callback that runs when that channel expires, or pass NULL “to use system reset handler”, according to the task watchdog API.
Use that callback. Before the reset, write the expired channel number into memory that survives a reset, so that the next boot can report “the radio task missed its 30-second deadline” instead of only “watchdog”.
Example (fictional). A Wi-Fi water-leak sensor runs three threads. The sensor thread reads its probe every 10 seconds and feeds its channel after each stored reading, with a 30-second reload period. The network thread feeds after each message the cloud acknowledges, with a 10-minute period, because a router reboot can legitimately delay a report by several minutes. The update thread adds its channel only while a download is in progress and deletes it afterward. A hardware watchdog with a longer period sits underneath all three. When the network thread deadlocks on a mutex, the sensor thread keeps feeding its own channel, the network channel expires, its callback stores “channel 1”, and the device resets with that number on record.
ESP-IDF offers the same pattern under different names. Its watchdog documentation describes an Interrupt Watchdog Timer that keeps interrupt handlers from being blocked, a Task Watchdog Timer that detects “tasks running without yielding for too long”, and an RTC watchdog used during boot and panic handling. Tasks subscribe with esp_task_wdt_add() and feed with esp_task_wdt_reset(). Note the fatal errors guide’s wording: a task watchdog timeout is “only fatal if CONFIG_ESP_TASK_WDT_PANIC is set”. Decide whether a missed deadline should reset your product, and set the option to match.
Timeouts, sleep and debugging
Three settings trip up first fleets.
- Timeout length. Set each channel’s period from the longest legitimate gap for that task, plus margin, not from a guess. A cellular attach or a flash erase can block a task for seconds; measure the worst case on the bench.
- Sleep. Zephyr’s watchdog API offers
WDT_OPT_PAUSE_IN_SLEEP(“Pause watchdog timer when CPU is in sleep state”), and the task watchdog providestask_wdt_suspend()to “Pause all channels before changing system power mode”. Without a deliberate choice, a device that sleeps longer than the watchdog period resets every time it sleeps. - Debugger halts.
WDT_OPT_PAUSE_HALTED_BY_DBGpauses the watchdog while a debugger has halted the CPU. Turn it on for development builds so breakpoints do not cause resets, and confirm what production builds do.
ESP-IDF’s hardware watchdogs have “four timeout stages”, each of which can raise an interrupt, reset a CPU core or reset the system. The documentation suggests interrupt actions in early stages and a system reset in a later one. That staging is a natural place to save the fault record before the final reset.
Record why the device reset, every time it boots
Read the hardware reset cause at every boot, combine it with your own record of why the firmware reset itself, and send both with the first message after boot. Reset counts split by cause are the cheapest crash signal you can collect, and they work even when nothing else was saved.
Zephyr’s hardware information API reads “reset cause flags”, which “can be used to determine why the device was reset; for example due to a watchdog timeout or due to power cycling.” Each chip supports a subset; hwinfo_get_supported_reset_cause() tells you which. The flags include RESET_PIN, RESET_SOFTWARE, RESET_BROWNOUT, RESET_POR, RESET_WATCHDOG, RESET_CPU_LOCKUP, RESET_LOW_POWER_WAKE and others.
Read the API notes before you trust the value. On some platforms the flags “accumulate between successive resets”, so hwinfo_get_reset_cause() may report every cause since power-on. The documentation’s fix: call hwinfo_clear_reset_cause() after reading, so the next boot sees only the latest reset.
ESP-IDF returns a single value from esp_reset_reason(), documented in its system API reference. The values distinguish more firmware causes than most chips’ hardware flags do: ESP_RST_PANIC (“Software reset due to exception/panic”), ESP_RST_INT_WDT, ESP_RST_TASK_WDT, ESP_RST_WDT (“Reset due to other watchdogs”), ESP_RST_BROWNOUT, ESP_RST_SW for a deliberate esp_restart(), ESP_RST_DEEPSLEEP and ESP_RST_POWERON, among others.
| Reset cause | Zephyr hwinfo flag | ESP-IDF reason | What it usually points to | First action |
|---|---|---|---|---|
| Watchdog | RESET_WATCHDOG | ESP_RST_TASK_WDT, ESP_RST_INT_WDT, ESP_RST_WDT | A task or interrupt stopped making progress | Read the expired channel from the retained record |
| Fault or panic | Your record, then RESET_SOFTWARE from your reboot; RESET_CPU_LOCKUP where supported | ESP_RST_PANIC | A firmware bug: bad pointer, stack overflow, failed assert | Group by build ID and fault address |
| Brownout | RESET_BROWNOUT | ESP_RST_BROWNOUT | Supply dipped: weak battery, cold, radio current peaks | Compare with battery voltage and temperature |
| Planned software reset | RESET_SOFTWARE | ESP_RST_SW | Your own reboot, for an update or a recovery step | Check the reason code your firmware stored |
| Power-on | RESET_POR | ESP_RST_POWERON | Battery change, installation, or a full power loss | Expected at install; investigate repeats |
| Wake from deep sleep | RESET_LOW_POWER_WAKE | ESP_RST_DEEPSLEEP | Normal operation on parts that reset on wake | Exclude from crash counts |
Store your own reason before every deliberate reset
The hardware cannot tell a reboot after a fault from a reboot after an update, because your code requested both. Whenever the firmware resets itself, store a reason code first: update installed, fault, watchdog channel expired, configuration change, recovery attempt. Zephyr’s sys_reboot() performs the reset; the reason has to be written before the call.
The Zephyr retention system exists for this. It reads and writes memory areas “that retain the data while the device is powered”, with an optional magic header and checksum to confirm the data is valid. The documentation’s own example splits “a 64-byte retained data area” into “4 bytes for a boot mode, 16 bytes for a timestamp, 44 bytes for a last log message”. Note the limit it states: the data “should not persist in the event of a power failure”. Anything that must survive a battery pull belongs in flash.
Add a boot counter and an uptime-at-reset value to the same area. A device that resets every 40 minutes tells a different story from one that resets once after 30 days, even when the cause flag is the same.
What to save when the firmware faults
Save a small, fixed-size fault record from inside the fault handler: the reason code, the program counter and link register, the stack pointer, the processor’s fault status registers, the build ID, uptime and boot count. Write it to retained RAM, mark it valid, and reset. Do as little else as possible in that state.
Read the fault registers while you can
On Arm Cortex-M parts with configurable fault handlers, the System Control Block holds the evidence. Arm’s CMSIS header for the Cortex-M4 defines the Configurable Fault Status Register (CFSR, which packs the UsageFault, BusFault and MemManage status fields), the HardFault Status Register (HFSR), and the MemManage and BusFault address registers (MMFAR, BFAR), with a valid bit telling you whether each address register holds a real address. Copy all four into the record before anything overwrites them.
Smaller cores report less. Zephyr’s Cortex-M architecture notes explain that in Baseline Cortex-M, “HardFault is the only available processor fault exception”, while Mainline cores add MemManage, UsageFault and others. On a Cortex-M0+ part, the stacked program counter and the reason code carry most of the information, which is one more reason to capture them.
Use the handler Zephyr gives you, and keep it short
In Zephyr, override k_sys_fatal_error_handler(). It receives the fatal error reason (such as K_ERR_CPU_EXCEPTION, K_ERR_KERNEL_OOPS or K_ERR_STACK_CHK_FAIL) and an exception context with “partial or full register state when the error occurred”, which “May in some cases be NULL”. The API reference also warns that unless the fault occurred in user mode, “the kernel itself may be in an inconsistent state, with API calls to kernel objects possibly exhibiting undefined behavior or triggering another exception.”
That warning sets the rule for the handler: no logging calls that allocate, no flash writes through a file system, no network. Copy registers into a structure in retained RAM, compute a checksum, write the reset reason, and call sys_reboot(). A record like the one below is about 72 bytes:
struct crash_record {
uint32_t magic, boot_count, uptime_s;
uint8_t build_id[20]; /* GNU build ID, sha1 style */
uint32_t reason; /* fatal error reason code */
uint32_t pc, lr, sp; /* from the exception context */
uint32_t cfsr, hfsr, mmfar, bfar;
uint32_t wdt_channel; /* set by the watchdog callback */
uint32_t crc32;
};
On the next boot, the application checks the magic value and checksum, copies the record to flash or queues it for upload, and clears it.
Keep a short trail of recent events
A fault record shows where the code was; it rarely shows how it got there. A breadcrumb trail fills that gap: a small ring buffer in retained RAM that records the last few significant events, such as state machine transitions, radio connects and disconnects, flash writes and update steps, each as a code and a timestamp. Eight to sixteen entries of a few bytes each fit easily beside the fault record, and writing one costs a few instructions, so the trail can stay on in release builds. The Zephyr retention documentation’s own example reserves 44 bytes for “a last log message”, which is the same idea in its simplest form. Send the trail with the fault record.
Decide which assertions stay in release builds
An assertion that fails in the field turns silent data corruption into a clean, reported fault at the line that noticed it. Zephyr’s assertion documentation describes four levels per module: ZASSERT_LEVEL_OFF compiles assertions out; ZASSERT_LEVEL_TERSE checks them but reports only a fixed banner; ZASSERT_LEVEL_NORMAL reports the location (“ASSERTION FAIL @ file:line”); and ZASSERT_LEVEL_VERBOSE adds the condition and message. CONFIG_ASSERT is “the master switch”, and when it is off every assertion “compiles to nothing”. A reasonable release policy is the normal level for modules that handle state, storage and the update path, with the file and line copied into the fault record, and terse or off only where measured flash or timing says so.
Catch stack overflows where they happen
A stack overflow often corrupts memory first and crashes later somewhere unrelated, which makes the eventual fault record misleading. Zephyr documents several detectors: CONFIG_HW_STACK_PROTECTION uses CPU features or guard regions and “will catch stack overflows in supervisor mode”; CONFIG_STACK_SENTINEL is a software check that may detect an overflow “a nontrivial amount of time after the stack actually overflowed”; and CONFIG_STACK_CANARIES uses compiler canaries. Armv8-M parts add hardware stack limit registers, and Zephyr notes that with them “Stack overflows trigger the dedicated UsageFault exception”. Enable the strongest detector your part supports in every build, including release builds, because the cost is small and the alternative is a crash with no useful address.
ESP-IDF lists stack overflow, stack smashing protection, heap integrity checks and failed assertions among the errors its panic handler catches, so on that platform the work is mostly choosing the options and saving the output.
Coredumps: when they are worth the flash
A coredump saves registers plus memory, so an engineer can open the crash in a debugger and inspect variables and other threads. It costs kilobytes of flash per dump and a larger upload. Keep the small fault record on every device; add a coredump when the flash budget has room and you can retrieve dumps on request.
Zephyr’s core dump module
The Zephyr core dump module is “called when a fatal error is encountered” and dumps “the CPU registers and memory content for offline debugging”. Its backends write to the log, to UDP, or to a flash partition defined in devicetree (the documentation’s example partition is 4 KB). Three memory options set the size:
| Option | What is saved | Use when |
|---|---|---|
DEBUG_COREDUMP_MEMORY_DUMP_MIN |
The faulting thread’s stack, its thread structure and minimal data for walking the stack | Flash is tight; you mainly need the call stack |
DEBUG_COREDUMP_MEMORY_DUMP_THREADS |
The structures and stacks of all threads | Deadlocks and interactions between threads |
DEBUG_COREDUMP_MEMORY_DUMP_LINKER_RAM (the default) |
Application RAM, including data, noinit and BSS sections | You can afford a dump roughly the size of used RAM |
DEBUG_COREDUMP_CRC appends a CRC-32 so that “a dump corrupted or truncated in transit or storage can be detected” before analysis. The core dump API lets the application ask whether a stored dump exists (COREDUMP_QUERY_HAS_STORED_DUMP), get its size, copy it out (COREDUMP_CMD_COPY_STORED_DUMP) and erase or invalidate it after upload. Analysis runs on a computer: the dump and “the Zephyr ELF file” go into a GDB server script that ships with Zephyr. Without the exact ELF, the dump is a block of hex.
ESP-IDF core dumps
ESP-IDF’s core dump guide saves dumps to flash or prints them over UART. A dump “contains snapshots of all tasks in the system at the moment of failure”, stored in ELF format with “a SHA256 checksum that verifies the integrity of the file”. By default it holds “CPU registers, tasks data and summary of the panic reason”; CONFIG_ESP_COREDUMP_CAPTURE_DRAM adds data, BSS and heap, which the guide recommends for debugging while noting it makes the file larger.
The guide gives the partition arithmetic: at least “20 + max tasks number x (12 + TCB size + max task stack size) bytes”, and its example partition table reserves 64 KB. If flash encryption is enabled, the core dump partition needs the encrypted flag.
Example (fictional). Loamwick’s LW-1 soil probe has 1 MB of flash: a 48 KB MCUboot partition, two 448 KB image slots, 32 KB of settings and 48 KB of logs, per the flash budget calculator. The 412 KB image already fills 92.8% of its slot, so there is no spare partition for a full-RAM coredump. The team keeps the 72-byte fault record in retained RAM, copies it into the existing 48 KB log area at the next boot, and configures the minimal Zephyr dump for bench units only. When the second slot moves to external flash on Rev E, the freed internal flash can hold a coredump partition for field units.
Tie every report to the exact binary
A crash address means nothing without the binary that produced it. Embed a build ID in every image, report it with every crash record, and archive the ELF file and map file for every build that could reach a device.
The GNU linker’s --build-id option adds a note section whose contents are “unique bits identifying this linked file”. With the default sha1 style it is a 160-bit hash (20 bytes), and the md5, sha1 and xx styles produce “an identifier that is always the same in an identical output file”. The documentation adds that a linked file “may be changed later by other tools, but the build ID bit string identifying the original linked file does not change”, which matters when a signing step appends a header and trailer to the image.
ESP-IDF stores an equivalent in every application image: the app_elf_sha256 field of esp_app_desc_t holds the SHA256 of the ELF file. esp_app_get_elf_sha256_str() returns it as text and “Can be used in panic handler or core dump during when cache is disabled”, so the crash path can include it safely.
On the server, symbolization is a lookup. Given the archived ELF for the reported build ID, GNU addr2line “translates addresses or symbol+offset into file names and line numbers”, with --functions to print function names and --inlines to show the inlined call chain. Run it on the program counter and the link register from each fault record, and on the return addresses a coredump contains, and store the result with the report so that nobody has to repeat the step.
A version string is not enough on its own. Two engineers can build “1.1.0” from different trees, and a debug build and a release build share a version. Report both: the version for people, the build ID for tools. Then make the archive part of the release process. The firmware release checklist already asks for releases “built by CI from a tagged commit, not on a laptop” with the “Version string, build number and commit hash embedded in the image”; add “ELF and map file archived under the build ID” to the same list.
Getting crash data off the device within budget
Send a short crash summary with the first message after every boot, keep full coredumps on the device until someone asks for them, and erase records only after the server acknowledges them. Budget the upload like any other traffic, because on battery and cellular products it is not free.
What to send, and when
- Every boot: reset cause, your stored reason, boot count, uptime at reset, build ID and version. A few dozen bytes.
- After a fault or watchdog reset: the fault record. Under a hundred bytes in the layout above.
- On request: the coredump, for a device or a build an engineer is investigating.
Rate-limit the second and third. A device in a crash loop should not drain its battery reporting the same fault every few minutes: send the first record in full, then a count of repeats.
The arithmetic
Crash data is usually small next to firmware images, but it is worth one line of arithmetic per product. Suppose, hypothetically, that a 16 KB coredump travels over cellular at the $0.40 per MB the OTA data cost calculator uses as its default price. Each dump costs about $0.006. For a hypothetical fleet of 5,000 devices that each sent one dump a month, that is about $375 a year. Sending summaries from every device and full dumps only on request keeps most of that number off the bill.
For a Bluetooth LE product, the cost is radio time and battery rather than money. At the BLE update time calculator’s default link (244-byte payloads, 3 packets per 30 ms connection event, 19.1 KB/s of useful throughput), 16 KB moves in under a second. A phone or gateway that negotiates 20-byte payloads would take more than ten times as long.
Treat crash data as data about people
A RAM dump can contain whatever the device held at the time: readings, identifiers, network credentials. The EU Cyber Resilience Act’s essential requirements include processing only data that is “adequate, relevant and limited to what is necessary” and, for security monitoring, recording “relevant internal activity” “with an opt-out mechanism for the user” (Annex I, EUR-Lex). This article is not legal advice. In practice, prefer the minimal dump in the field, keep credentials out of the RAM regions you dump where you can, send crash data over the same authenticated channel as everything else, and set a retention period on the server.
Turning crash reports into fixes and release decisions
Crash records become useful when they are grouped. Group reports by build ID and fault address (or watchdog channel), count them per 1,000 device-days for each build, and compare the new build with the old one over the same days. That comparison drives two decisions: what to fix next, and whether a rollout may continue.
From records to a ranked bug list
On the server, symbolize each record against the archived ELF for its build ID to turn the program counter and link register into a function and line. Then sort by how many devices are affected, not by how many reports arrived, so that one device in a crash loop does not outrank a fault that hits a tenth of the fleet once. Pull a coredump from one or two affected units only when the fault record does not explain the failure.
Each group becomes a ticket with a count, the first and last build it appeared in, and the devices affected. When a fix ships, the same grouping shows whether that signature disappears from the new build.
Crash signals in a staged rollout
A staged rollout sends a new release to a small wave of devices first and widens only when the wave looks healthy. Crash data is one of its main inputs. The gate thresholds, control cohorts and soak times are covered in staged firmware rollouts; what this article adds is that the counts must exist on the first day of the first wave, which means the kit above has to ship in the version you roll out from, not in the version you roll out to.
The bootloader contributes a signal of its own. With MCUboot test swaps, “If the device crashes immediately upon booting a new (bad) image, MCUboot will revert to the old (working) image at the next device reset”, per the MCUboot design document. The firmware makes the swap permanent “only after performing a self-test routine”. Count reverts as crashes of the new build, and do not confirm a new image until the watchdog has stayed quiet for a set period. The rest of the update chain that this depends on is described in what it takes to ship firmware you can safely update.
Test the crash path before you ship
Diagnostics that have never been triggered usually fail the first time they are needed. Before a release goes to a pilot fleet, cause each kind of failure on purpose on a production-configured unit, and check that the right record arrives at the server with the right build ID.
| Test | How to cause it | What must arrive |
|---|---|---|
| Bad pointer | A test command that reads address 0 or an unmapped region | Fault record with fault status registers and the calling function |
| Stack overflow | A test command that recurses in a small-stack thread | Stack overflow reason, not a random fault elsewhere |
| Failed assertion | A test command that asserts false | Assertion reason with the file and line |
| Task hang | Block one task in a loop | Watchdog reset with the expired channel number |
| Scheduler stall | Disable interrupts in a loop | Hardware watchdog fallback reset |
| Brownout | Lower the bench supply slowly during a radio transmission | Brownout cause, no fault record |
| Crash loop | A build that faults within seconds of boot | One full record, then a repeat count; no battery drain |
| Bad update | Install a test image that faults at start | MCUboot revert, reported as a crash of the new build |
| Power cut during a record write | Remove power while the boot code copies the record to flash | Either the full record or none, never a corrupt one |
Keep the test commands out of shipping builds, or behind the same authentication as other service commands. The firmware release checklist already asks for an interrupted update and a forced rollback before each release; these tests do the same for the diagnostics path, and they belong in the same hardware-in-the-loop run.
If you are choosing a platform now, the availability of these features is a real input: Zephyr and ESP-IDF document most of the kit, as above, while a bare-metal project has to build each piece itself. The trade-offs are covered in Zephyr, FreeRTOS or bare metal.
What to build this sprint
Most of this kit is a few days of work if it goes in before launch and a painful retrofit afterward, because the first fleet runs whatever diagnostics it shipped with until its first update succeeds. In order:
- Replace the default fatal handler. Write a short handler that copies registers into a retained structure and reboots; never leave a field build that halts.
- Add task watchdog channels. One per critical task, each fed on real progress, with a hardware fallback and a callback that records the channel.
- Report the reset cause and your stored reason at every boot. Clear the hardware flags after reading them.
- Embed the build ID. Add
--build-id(or use ESP-IDF’s ELF SHA256), report it everywhere, and archive ELF and map files under it in CI. - Define the upload messages. Summary every boot, record after a fault, coredump on request, with rate limits and acknowledgment before erase.
- Decide on coredumps. Check the flash budget; choose the minimal or thread-level dump if a partition fits.
- Run the crash-path tests. Every row of the table, on a production-configured unit, before the pilot.
- Wire the counts into the rollout gate. Resets per 1,000 device-days by cause and build, compared with the previous release.
The firmware development service covers this kind of groundwork in a new codebase, and the sustaining releases service is where crash groups turn into scheduled maintenance releases with a release report after each one. If your fleet is about to ship without a crash path, a firmware review from Fumbar reads your code, flash layout and update plan and ends in a written plan with the risks and the order of work; you can request a firmware review.
