Safe Linux Updater
Hi,
One month ago, I sent a first request for comments about an open source automatic updater for embedded systems using U-boot: the goal of this project was to split a flash memory or hard disk drive in multiple partition (2 or more) and install a new kernel and/or root filesystem on an empty or outdated partition.
After reboot, we count the boot attempts on this new version of the system and if it fails to boot, we switch back to the last working partition. This guarantees we will eventually boot on a correct partition and that there is no chance to have an unsupervised equipment hang at the u-boot prompt.
The u-boot environment would contain some variables to handle name, state and boot attempt count of each partition of the system. I would use CONFIG_ENV_OFFSET_REDUND to make writing on environment powerfail-safe.
The environment contains 3 variables by partitions:
- part_X_flag: handles state of partition X. It can be NONE for empty partition, OK for working system, LOCK for locking a partition to install a new system on it, UPDATED for new system version (we count boot attempts for this partition) and BAD for bad system which doesn't work.
- part_X_count: handles boot attempts on partition X.
- part_X_cmd: U-boot command used to boot on the partition X (e.g. partition address in flash).
Environment contains also another variable: "boot_seq" which handles a list of bootable partition sorted by version.
As suggested in previous emails, I have written two scripts to implement that:
- An u-boot script to interpret all environment variables and boot on correct partition and count boot attempts when system boots on a new version.
- A Linux script to change environment variables when an update is installed.
The U-boot script reads "boot_seq" and boots on first partition of the list. If this partition is flagged OK, system boot normally. If the partition is flagged UPDATED, system try to boot on it and count boot attempts until attempts limit is reached. Then, the script will boot on next partition into boot list, always in counting boot attempts. This operation continue while partitions are available in the list. Partition flagged with NONE, LOCK or BAD are skipped.
The Linux script has all commands necessary to automatically find and lock the best partition to install an update, to mark the locked partition as UPDATED and commands to mark a new version as good with OK or bad with BAD. This script uses fw_printenv and fw_setenv from U-boot tools.
I would like your comments/suggestions on this system to improve it and to make sure this goes well into the general philosophy of U-Boot. Moreover, if it is possible, to upstream these scripts.
Best regards,
Alexandre Dilly
Dear Alexandre
Quoted 3 lines · In message <[email protected]> you wrote:
One month ago, I sent a first request for comments about an open source automatic updater for embedded systems using U-boot: the goal of this project was to split a flash memory or hard disk drive in multiple partition (2 or more) and install a new kern el and/or root filesystem on an empty or outdated partition.
Could you please restrict yourline length to some 70 characters or so? Thanks!!
Quoted 7 lines
The u-boot environment would contain some variables to handle name, state and boot attempt count of each partition of the system. I would use
CONFIG_ENV_OFFSET_REDUNDto make writing on environment powerfail-safe. The environment contains 3 variables by partitions:
-part_X_flag: handles state of partition X. It can be NONE for empty partition, OK for working system, LOCK for locking a partition to install a new system on it, UPDATED for new system version (we count boot attempts for this partition) and BAD for ba d system which doesn't work.
-part_X_count: handles boot attempts on partition X.
-part_X_cmd: U-boot command used to boot on the partition X (e.g. partition address in flash).
Environment contains also another variable: "boot_seq" which handles a list of bootable partition sorted by version.
This sounds like a bad idea, for two reasons:
1) U-Boot already supports the boot count feature (but of course this hardware-specific as you need to find persistent storage for the counter). However, this does not consider which exact boot command has been executed (in your case: what the used boot partition was), it just counts the number of resets after the last power-on. Actually this is all you should need.
Please note that this is a feature standardized for example in the Open Source Development Labs Carrier Grade Linux Requirements Definition, which says something like: "CGL shall provide support for detecting a repeating reboot cycle due to recurring failures and will go to an offline state if this occurs."
2) Defining the boot counter as part of the envrionment requires automatic writes to the environment for each reset / reboot of the board. This is considered a bad idea, as it causes excessive flash wear. Normally you want to avoid all erase / write operations to the boot loader and it's private data structures in the process of a normal reboot / reset.
Best regards,
Wolfgang Denk
Signature · 5 lines
DENX Software Engineering GmbH, MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
Microsoft Multitasking:
several applications can crash at the same time.Dear Wolfgang,
I havn't seen the scripts of Alexandre but it sounds something like what we have already implemented.
Quoted 5 lines · Wolfgang Denk
Please note that this is a feature standardized for example in the Open Source Development Labs Carrier Grade Linux Requirements Definition, which says something like: "CGL shall provide support for detecting a repeating reboot cycle due to recurring failures and will go to an offline state if this occurs."
As I read Alexandre, the aim is to revert to a previous functional image, not to go to an offline state.
Quoted 3 lines
Normally you want to avoid all erase / write operations to
the boot loader and it's private data structures in the process of a normal reboot / reset.
But a failing boot is not a normal boot. This should only occur when an update fails. After a maximum number of failing boots, the old functional image is used and there is no need to update the counter any more.
Best Regards,
Mats
Hi Mats,
Quoted 6 lines · Mats K?rrman
Dear Wolfgang,
I havn't seen the scripts of Alexandre but it sounds something like what we have already implemented.
Wolfgang Denk wrote:
Quoted 8 lines
Quoted 5 lines
Please note that this is a feature standardized for example in the Open Source Development Labs Carrier Grade Linux Requirements Definition, which says something like: "CGL shall provide support for detecting a repeating reboot cycle due to recurring failures and will go to an offline state if this occurs."
As I read Alexandre, the aim is to revert to a previous functional image, not to go to an offline state.
This is already done in u-boot checking the value of the boot counter (in not persistency storage) and calling a script that switch back to the previous copy, if any.
I find that the proposal does not scale well. Having partitions on a disk / SDCARd is a case, but we have several different way to boot. Think about kernel / rootfs into UBI or UBIFS, or saved as raw data in other kind of storages (NOR, SPI,..). Because we are talking about the feature "updating", this should be abstracted from the specific case to be generalized in U-Boot.
Quoted 7 lines
Quoted 4 lines
Normally you want to avoid all erase / write operations to
the boot loader and it's private data structures in the process of a normal reboot / reset.But a failing boot is not a normal boot. This should only occur when an update fails.
There are runtime conditions that can cause the boot to fail, due for example to power-supply. Or a degrading of the resources (flash gets wrong), and so on.
Even if a failure due to a wrong update is a common case to have a failing boot, this is not the only use case.
Best regards,
Stefano Babic
Signature · 5 lines
===================================================================== DENX Software Engineering GmbH, MD: Wolfgang Denk & Detlev Zundel HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: +49-8142-66989-53 Fax: +49-8142-66989-80 Email: sbabic at denx.de =====================================================================
Dear Mats,
Quoted 3 lines · In message <[email protected]> you wrote:
I havn't seen the scripts of Alexandre but it sounds something like what we have already implemented.
I haven'ty seen the scripts either, but this is something that has been implemented many times before, with many variations depending on specific project needs. But as I mentioned before, most users do not push such scripts into mainline...
Quoted 7 lines
Quoted 5 lines
Please note that this is a feature standardized for example in the Open Source Development Labs Carrier Grade Linux Requirements Definition, which says something like: "CGL shall provide support for detecting a repeating reboot cycle due to recurring failures and will go to an offline state if this occurs."
As I read Alexandre, the aim is to revert to a previous functional image,
Yes, of course. And the boot counter is the mechanism that will decide when to do that.
Quoted 1 line
not to go to an offline state.
Indeed. U-Boots boot counter will select an alternative boot command in this case - you can use this to hard hang or power off the board and so implement strictly CGL conformant behaviour, or you can do somthing else like fall back to the previous version.
Quoted 6 lines
Quoted 3 lines
Normally you want to avoid all erase / write operations to
the boot loader and it's private data structures in the process of a normal reboot / reset.But a failing boot is not a normal boot. This should only occur when an update fails. After a maximum number of failing boots, the old functional
Alexandre wrote about a boot counter; he did not mention that he would update this only on failed boot attempts.
Also, in my experience one should be especially careful when something fails, and in such a situation I would all the more restrain from messing with the environment if it can be avoided (and here it's trivial to avoid).
Quoted 1 line
image is used and there is no need to update the counter any more.
Did you have a look at the current implementation of the boot counter for the systems where it is supported? Yes, we even do have a system where the boot counter is stored in an environment variable (due to hardware restrictions and the expectation that the system will normally not need to be rebooted at all), but normally is can be easily avoided to meddle with the environment for this functionality.
Best regards,
Wolfgang Denk
Signature · 6 lines
DENX Software Engineering GmbH, MD: Wolfgang Denk & Detlev Zundel HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de A person who is more than casually interested in computers should be well schooled in machine language, since it is a fundamental part of a computer. -- Donald Knuth
RFC 1/1 Safe Linux Updater
Dear Wolfgang, Mats and Stefano,
Thanks for your answers,
Quoted 12 lines
In message <[email protected]>
Quoted 4 lines · you
I haven't seen the scripts of Alexandre but it sounds something like what
we have already implemented.I haven't seen the scripts either, but this is something that has been implemented many times before, with many variations depending on specific project needs. But as I mentioned before, most users do not push such scripts into mainline...
Yes, this kind of system has been implemented many times before but, we can't found any examples in u-boot source, so the idea was to give a generic example to gain time in updater development...
Quoted 15 lines
Quoted 11 lines
Quoted 8 lines
Please note that this is a feature standardized for example in the
Open Source Development Labs Carrier Grade Linux Requirements Definition, which says something like: "CGL shall provide support
for detecting a repeating reboot cycle due to recurring failures
and will go to an offline state if this occurs."As I read Alexandre, the aim is to revert to a previous functional image,
Yes, of course. And the boot counter is the mechanism that will decide when to do that.
Quoted 7 lines
Quoted 1 line
not to go to an offline state.
Indeed. U-Boots boot counter will select an alternative boot command in this case - you can use this to hard hang or power off the board and so implement strictly CGL conformant behaviour, or you can do something else like fall back to the previous version.
U-boots boot counter is not persistent which implies we can't handle some error cases like power fail. That's why I suggested to use environment to store boot counter with redundant option to be power safe.
Quoted 20 lines
Quoted 9 lines
Quoted 4 lines
Normally you want to avoid all erase / write operations to
the boot loader and it's private data structures in the process of
a normal reboot / reset.But a failing boot is not a normal boot. This should only occur when an
update fails. After a maximum number of failing boots, the old functionalAlexandre wrote about a boot counter; he did not mention that he would
update this only on failed boot attempts.Also, in my experience one should be especially careful when something
fails, and in such a situation I would all the more restrain from messing with the environment if it can be avoided (and here it's trivial to avoid).
Quoted 9 lines
Quoted 1 line
image is used and there is no need to update the counter any more.
Did you have a look at the current implementation of the boot counter for the systems where it is supported? Yes, we even do have a system where the boot counter is stored in an environment variable (due to hardware restrictions and the expectation that the system will normally not need to be rebooted at all), but normally is can be easily avoided to meddle with the environment for this functionality.
In fact, what Mats explain was my idea: the system updates boot counter only when it detects a partition marked as "updated". When Linux has booted, it does some specific verifications and then mark the partition as "good" or as "bad". So, boot counter is not updated any more until a new update is installed on system. So, like said Mats, count update only occurs when an update fails and then it doesn't cause excessive memory wear, because we have a limit like 3 for boot attempts.
Quoted 7 lines
I find that the proposal does not scale well. Having partitions on a disk / SDCARd is a case, but we have several different way to boot. Think about kernel / rootfs into UBI or UBIFS, or saved as raw data in other kind of storages (NOR, SPI,..). Because we are talking about the feature "updating", this should be abstracted from the specific case to be generalized in U-Boot.
About partition, since the script uses an environment variable to store boot command for each partition, it can be a disk partition or a UBI partition or a raw memory area containing kernel or rootfs which will be loaded with boot command.
To illustrate all ideas, I join to this mail the both scripts. These scripts are definitively not final versions!
U-boot script:
uboot_script new file +64 -0
@@ -0,0 +1,64 @@ +setenv tmp_update +if test "z$force_rescue" != "z"; then + echo "Boot on rescue partition..." + run part_r_cmd + echo "System cannot boot!" + exit 1 +fi +if test "z$max_attempts" = "z"; then + echo "Set maximum boot attempts to default (=3)" + setenv max_attempts 3 +fi +for i in $boot_seq; +do + setenv tmp setenv tmp \$part_${i}_flag + run tmp + if test "z$tmp" != "z"; then + if test $tmp = "OK" -o $tmp = "UPDATE"; then + if test $tmp = "UPDATE"; then + setenv tmp_update 1 + echo "Partition $i is an update" + fi + if test "z$tmp_update" != "z"; then + setenv tmp setenv tmp \$part_${i}_count + run tmp + if test "z$tmp" != "z"; then + echo "Try to boot on partition $i..." + else + echo "Reset counter and try to boot on partition $i..." + setenv "part_${i}_count" 0 + setenv tmp 0 + fi + while test $tmp -lt $max_attempts; + do + setexpr tmp $tmp + 1 + echo " -> Attempt $tmp on partition $i" + setenv "part_${i}_count" $tmp + setenv tmp + setenv tmp_update + saveenv + run part_${i}_cmd + echo " -> Boot failed!" + done + if test $tmp -eq $max_attempts; then + setenv "part_${i}_count" 254 + setenv tmp + setenv tmp_update + saveenv + fi + echo "Partition $i failed to boot!" + else + echo "Boot on partition $i..." + run part_${i}_cmd + echo " -> Boot failed on partition $i!" + fi + fi + else + echo "No flag for partition $i" + fi +done +echo "Boot on partitions failed! Boot on rescue partition..." +run part_r_cmd +echo "System cannot boot!" +exit 1 +
Linux script:
smart_boot.sh new file +522 -0
522 lines not on this page · raw message
Best regards,
Alexandre Dilly