Here's why the first two categories have to be scored separately, not folded together. Picture a run that produces a clean build: no errors, no warnings, green across the board. Now suppose that same run added a DeviceTree node by re-declaring a peripheral at its base address instead of extending the board's existing label. The compiler most likely won't catch it but the behavior difference will be substantial. The only way to catch it is to check the overlay against the board's ownership rules directly, which is exactly what a build-only benchmark never does. As a result the build may pass but the generated firmware will be wrong. A benchmark that stops at compilation would score that run as a success.

As an example of what I am describing take the following annecdotical scenarios. They represent false positives. A run with missing evidence should not score high because the code looks plausible. A run that writes to protected board overlay paths should be capped even if it builds. A run that silently changes the target board, invents an unsupported sensor should fail outright regardless of how the rest of the run performed.

The design principle is simple: reward engineering discipline. A model should be able to generate a fluent answer that build, explain its assumptions and respect hardware ownership.

How Published Multi-Agent Work Points Toward This Model

The multi-agent Zephyr software generation work previously published and similar contract-driven tools emerging elsewhere in the industry are useful concrete examples, because they treat embedded generation as a workflow rather than a one-shot prompt. The flow begins with project context and target metadata, then moves through structured intent, deterministic generation, validation, build gates, repair classification, and retained run artifacts.

These systems change what the LLM is asked to do. Instead of generating final files directly, the model increasingly describes intent, i.e. what the application needs, what peripherals it uses, what APIs should exist, what constraints apply. Deterministic services then render the known structures such as DeviceTree overlays, Kconfig fragments and traceable manifests from that validated intent and the SoC/HW target pack. Silicon, board, and SDK facts stay inside inspectable engineering data instead of living inside model memory, where they can't be checked.

In my opinion contracts are what make the workflow reviewable. An interface contract can describe files, APIs, ownership boundaries and external dependencies. A system contract can capture target-pack data, generated artifacts, and build evidence. Contracts also make repair safer: a repair loop constrained by a contract can't silently rewrite a public interface or cross an ownership boundary just because that's the fastest way to make a build pass.

Conflating failures of the diverse nature is a common failure mode in early agentic tooling. A system that treats a missing SDK the same way it treats a wrong function call will either retry pointlessly against an environment that can't change, or overcorrect a genuinely fixable mistake into an unnecessary refusal. Agentic AI Tooling workflows need to classify failures so it can decide whether deterministic repair, LLM-mediated repair, a request for user input, or refusal is the right response.

This is the core benchmark insight, and it's the one I want to leave you with: agentic AI for embedded software should be measured by the quality of the full engineering loop. The generated code is one artifact among several. The surrounding evidence... what was assumed, what was checked, what was rejected, what was repaired and why... is what tells an expert whether the result is trustworthy.

A Real Scenario Example: When a Green Build Should Still Fail

Consider a benchmark task with a deceptively simple brief: generate a Zephyr application for a MAX32690 evaluation board, configure an SPI accelerometer, sample it from an RTOS worker, and expose a small application API.

The scenario also carries constraints that are easy to miss if the evaluator looks only at code: preserve the selected board target, extend the existing SPI controller by label, keep generated files out of platform-owned board paths, and complete the task with no more than one repair iteration.

Now compare these two runs for illustration purposes.

An illustrative run evaluation in which the firmware compiles and links, but a protected board-overlay write triggers a hard cap. The raw score remains visible so reviewers can distinguish broad run quality from a release-blocking ownership violation.

Run B is the interesting one. It compiles, links, produces a non-empty ELF, and scores well in almost every weighted category. But the generated output writes directly to `boards/max32690evkit_max32690_m4.overlay`, a path owned by the board and project tooling. The correct implementation would place the generated fragment in a tool-owned overlay and extend the existing controller with a label reference such as `&spi2`.

A build-only benchmark would report success. An evidence-first benchmark records the 92.30 raw score, applies the ownership rule, caps the result at 40 and returns a failure. It prevents strong performance in easier categories from compensating for a defect that can invalidate the hardware configuration or create a future integration conflict.

The comparison also shows why a single number is not enough. Reviewers need the score and the evidence behind it: the resolved board target, contract hashes, generated-file manifest, gate results, ELF and memory evidence, etc. With those artifacts, an embedded engineer can reproduce the decision, challenge the rule, or correct the run without reconstructing what happened from a chat transcript.

The values above are an illustrative execution of a reference scoring model against synthetic retained-run evidence. They are not a cross-model ranking or a claim of validation on physical hardware.

Conclusion

The next generation of embedded AI tools won't be judged on whether they can write a convincing driver in isolation. They'll be judged on whether they can operate inside real constraints such as hardware context, RTOS configuration, build systems and safety requirements. Building the mechanisms to empower human reviews more efficiently and consequently reducing the cognitive load, will empower developers to build maintainable, safer software that ships faster and more frequently..

Benchmarks have to evolve to match. The lessons from general code-generation research still hold: execute the code, strengthen the tests, evaluate at the repository level, rotate held-out tasks, score security separately from correctness. But embedded software needs more than to introduce evidence and auditability of the whole pipeline run, not just its output.

For platform owners, this change dramatically the product needs and expectations. Agentic embedded tooling shouldn't be a black box that emits files on request. It should produce reviewable firmware proposals, validate them against the live toolchain and hardware context, and leave behind enough evidence that an expert engineer can trust the result, reject it, or improve it on their own terms.

If you've tried to build or evaluate an agentic embedded workflow yourself, I'd genuinely like to know where this model breaks. What is missing? Which one sounds right on paper and falls apart the first time it meets real hardware?

This is Part 2 of a two-part series on benchmarking agentic AI for embedded software. Part 1, "Beyond Code Completion: Benchmarking Agentic AI for Embedded Software" lays out the six axes this piece builds on.