Disclaimer: I'm with Mender.io and author of the article.
The Yocto Project is a popular build system for your own embedded Linux distribution.
Mender integrates with the Yocto Project with a layer (https://github.com/mendersoftware/meta-mender), but is a separate project for end-to-end OTA, which includes the client and the management server. Both are licensed under Apache 2.0 thus it is freely available and you're not locked into a hosted-only backend.
I'm with the open source project Mender.io (OTA for embedded Linux) and we think Casync is a very interesting building block and may look into this and evaluate whether it makes sense to incorporate it into our project.
We had looked into OSTree before but given the use case of embedded devices in the wild, we concluded it was too risky as OSTree relies on the filesystem to protect from power failures. And rollback was not built-in and is quite challenging to implement reliably.
>> Anyone know more precisely what he's talking about? The article doesn't specify.
Apologies for the confusion, yes I had meant vendor lock-in. That is usually a blocker for many embedded projects starting out. This is why we've created a open source project with both client and management server licensed under the permissive Apache 2.0.
If you are interested, we are always looking at ways to improve our project and if you'd like to participate in a one hour user test, we conduct those over Google Hangouts:
Hi there, this is Ralph with Mender.io. Thanks for the kind words! We've met a few people from Resin at conferences and they are very nice folks. We very much appreciate your technology's approach and look forward to seeing your open source platform.
This is a valid point. If booting just hangs after the bootloader but before the Mender daemon comes up is actually quite tricky to manage.
We have looked into hardware watchdog for this, but it is in the gray-zone of what an updater should be involved in. This is actually a more generic problem - maybe it hangs even when you did not deploy an update. There is varying support for hardware watchdogs across boards as well, unfortunately.
Most of the time it will not just hang, maybe it will crash or kernel panic and in those cases Mender will rollback. But the indefinite-hanging case is quite tricky and not yet handled.
Hey! Love the product, but I'm out of the embedded game. Thought I'd give my $0.02:
The first step of our boot process was to enable the watchdog. We extend the timeout periodically during the boot process, but generally if userspace isn't reached within 30 seconds or so we reset. Once in userspace, the daemon validates that things look good (this includes things beyond just application of the update -- did services start up correctly? Is the hardware operating as we expect?) before disabling the watchdog and marking the update as a success, at which point rollback isn't possible. At this point we might consider applying new updates, etc.
We also modified our first stage bootloader to be resilient to bootloader update issues, and chainloaded our second stage bootloader from a stub which could rollback.
We also niced the update process to avoid resource contention, allowed the updates to be delayed until the network was quiet, and paused them when it became noisy to make for a good user experience. There was a server-side flag to force updates to apply regardless, with higher priority, as well as one to basically disable all other functionality in the case of a unforeseen serious, perhaps security related, issue.
We actually had a discrete watchdog service which was responsible for petting an always-on watchdog, to rescue the system if it locked up or became unresponsive (if certain processes were not running, or responding, the watchdog would not be pet).
All of this led to effectively 0 failures in the field, a seamless user experience (except for the 30-second reboot when inactive). I wish everything I owned worked this way.
I could talk ad nauseum about this stuff. It's very cool to see the designs of others. I feel this is an under appreciated and under explored problem space.
block level signatures is not possible, which we feel is a requirement for an over-the-air updater
rollback is not built-in and can be quite challenging to implement reliably (bootloader level)
Think OSTree as more of a building block, like Git is for your development process. We might use it in the future but robustness and easy integration are our first priorities.
Signing an archive would probably be good enough for many cases. Block level is a bit simpler (all or nothing) and thus less risk of mixing with unsigned parts (sideloading attacks).
For security-sensitive embedded devices (e.g. payment terminals), block level signatures would allow hardware verification during boot as well (1st stage bootloader verifies 2nd stage, then kernel, etc.) if designed correctly.
