When we are working on a git repository and developers branch and merge features, it is very important that our main firmware deliverable stays in a deliverable state at all times.
The problem we face is that we must verify that our decisions about the behavior of the production firmware stay in place as all of this branching and merging is done on a daily basis. We need to run this for every change that is being merged so that we don’t need to go back and solve issues later.
- When a developer has to rebase their merge request: production firmware image should still continue to satisfy all user requirements.
- When a complex merge of bug-fixes is integrated from a release branch into trunk: production firmware image should still continue to satisfy all user requirements (trunk must always be in releasable state!)
- When the development team has to refactor a large nontrivial module: the production firmware generated after that change is merged must still satisfy all user requirements.
And user requirements, when converted into implementation decisions usually result in hundreds of things that need to be verified.
Simulation is the only way we can achieve reasonable preliminary coverage for all of these decisions without resorting to physical hardware tests (which are time consuming and much harder to automate).
Using simulation we can:
- Build software features when necessary hardware is either not yet available.
- Evaluate new firmware before hardware is manufactured.
- Verify complex user scenarios entirely in software - which makes it possible to run all these tests in CI for every single merge request.
Why this is important
Without simulation as an integral part of your embedded software development workflow, you will inevitably hit a wall where you can not run tests on production firmware.
Sure, you can still always build native posix tests to verify parts of your code-base (which is what we do when we write unit and integration tests) but for the final hex file that will actually run on your device in the field, you are at a loss.
What you need is a system that can load this image, including the bootloader and run the whole boot process from the start - including any signature checks on the image and all of the initialization code as well.
You need a system that will run the unmodified image and allow you to script user interactions with your firmware in such a way that they can be verified as tests.
When you have this in place, the circle becomes closed: you can truly apply powerful test driven development concepts to software that is normally so dependent on availability of hardware.
You can do this because you have simulation in place.
Solution to the simulation problem
For a long time QEMU has been the main simulation environment for testing embedded systems.
However QEMU had it’s own issues: it was hard to extend, it did not have a scripting language for automating test flows, it was not easy to add complex device behaviors and verify them effectively in CI.
Then Renode came along and changed that.
With Renode we can now easily simulate multiple devices running different firmware working as a network and we can write tests that verify the behaviors of the whole network.
Truly powerful.
You can even start using renode right now and gain benefit from it.
But the real benefit comes from systematically integrating Renode into your whole CI workflow. This is where Swedish Embedded can truly help.
High precision machine level simulation
When we start integrating renode and implementing tests using Robot Framework to verify your production firmware, several opportunities open up.
Renode supports very precise simulation of multiple devices.
By implementing custom scripts that describe user scenarios involving your devices, we can:
- Verify that firmware timing is correct across several connected units: so that for instance proper connection timeouts can be tested.
- Trace every function call and peripheral access: and verify number of instructions being executed by the processor over a period of time (making it possible to verify excessive CPU usage easily).
- Simulate any on-board and off-board peripheral: by adding plugins to Renode we can integrate a behavioral model of any device that is interacting with your firmware. This allows that device to be controlled from tests as well when implementing user scenarios.
All of this is done with accurate timing thanks to Renode multi-device simulation capabilities where multiple devices can be precisely simulated as part of the same simulation setup.
Build firmware before hardware is ready
Building hardware is a costly process. Prototyping typically requires multiple revisions of the hardware built.
It is even more costly to wait before hardware is ready before being able to develop and test your firmware.
With a simulation setup and support for your future hardware added in simulation you can easily develop software for future devices without actually having the hardware yet.
Better than software in the loop
Traditional software in the loop where you build your firmware application to run natively on Posix and mock parts of the system (such as physics) using software is of course better than not having any simulation at all.
However it is not enough. The main problem with conventional software in the loop setup is that you are not running the same code in simulation as you do in production.
This makes it impossible to say with certainty that you are testing actual behaviors of production software (you should of course always test on real hardware as well but simulation covers 99% of the way there so that live hardware tests only need to cover the remaining 1% of work).
Simulation integrates into your CI process after the build stage is done with compiling and signing your firmware. It takes this binary and runs all required behavioral tests against it.
Eliminates repetitive manual labor in testing
Hardware in-the-loop (HiL) testing is of course also a very valuable way of testing your product and should not be neglected. However, for every day build, debug, merge cycle, hardware testing is too expensive.
The biggest problem with physical hardware testing is that it is prone to physical limitations: it is enough to flash a slightly broken firmware to bring the hardware into a state that requires manual intervention to fix.
Having simulation as part of the development cycle removes this necessity for manual intervention.
We can code simulation actions in a high level test automation language such as using RobotFramework notation and run these on a simulated setup, verifying our software in the same way every single time!
The benefits of this include:
- Time saved: you spend a little bit of time setting up your simulation and save a lot when you run your tests every day as part of your development cycle.
- Direct mapping to user requirements: you can directly take your user requirements and map them to behavioral requirements that production firmware must satisfy to be considered valid.
- Ability to test multiple versions reliably: you can test how different versions of firmware interact together because it’s all done in software.
Provides ability to easily create custom visualizations
Even if scripted testing is extremely valuable, sometimes it is nice to be able to demo something visual.
Swedish Embedded SDK includes a visualization framework that enables development of customized visual representations of simulated device state.
You can use these visualizations to create beautiful demos of your product to show to your customers and also use it in development for quick visual feedback.
Using the visualization extensions you can:
- Use widely available browser frameworks: like jQuery, bootstrap, chart.js etc. to create arbitrarily detailed visualization of simulation state.
- Visualize networks of devices easily: since you can define multiple boards to be simulated at the same time.
- Export silkscreen and fab layer of your PCB: and use it for visualization of simulation.
- Experiment with mock-ups of your products: including ui and multiple view angles.
