Skip to contentSkip to conversation

Failover root devices

DISCUSSION Drew DeVault 2 participants 5 messages Sep 17, 2015 – Sep 18, 2015

lore
Share & tools
lore
#0 lore raw

I would like to see Linux support multiple root devices, so that it can attempt one and move on to the next if it is not present. I've reviewed the relevant code during boot-up and it seems like a good place for me to submit my first patch, but I want to bring it up for discussion here on LKML first.

The design I had in mind is something like this:

root=device;device;device;...

Where 'device' follows the current format (/dev/sdX, UUIDs, and so on, via name_to_dev_t). I would modify prepare_namespace to iterate through each offered root device until one works.

My use-case for this feature is that I would like to be able to change the hardware of my machine and boot up differently based on what's present. In my case, I would like to install my system normally, with /boot on its own partition, and keep a seperate userspace on a flash drive. Then, during boot-up, if the flash drive is present, it would be used as the root device. If it's not present, a partition on disk would be selected.

The only potential roadblock with this feature that comes to mind is figuring out how to handle time-outs between root devices. I think it would be wise to choose a sensible default value, and provide another cmdline parameter to tweak it. The prepare_namespace flow might end up looking something like this:

1. Wait rootdelay seconds
2. Check 1st device, not present
3. Recheck 1st device until rootfailoverdelay seconds has passed
4. Move on to 2nd device, present -> boot

Or:

1. Wait rootdelay seconds
2. Check 1st device, not present
3. Recheck 1st device until rootfailoverdelay seconds has passed
4. Move on to 2nd device, not present
5. Recheck 2st device until rootfailoverdelay seconds has passed
6. GOTO 2

And so on.

I also need to research how the various init systems interact with this part of the boot process. I suspect systemd probably does something silly wrt waiting for the root device. Since this feature would (of course) be backwards compatible, it might be wise to just implement it here and let the init systems add support for the feature themselves.

Advice? Who should I send my patches to when they're ready? Please CC me, I do not subscribe to LKML.

--
Drew DeVault

#1 lore raw
Quoted 14 lines · Drew DeVault

I would like to see Linux support multiple root devices, so that it can attempt one and move on to the next if it is not present. I've reviewed the relevant code during boot-up and it seems like a good place for me to submit my first patch, but I want to bring it up for discussion here on LKML first.

The design I had in mind is something like this:

root=device;device;device;...

Where 'device' follows the current format (/dev/sdX, UUIDs, and so on, via name_to_dev_t). I would modify prepare_namespace to iterate through each offered root device until one works.

Quoted 7 lines

My use-case for this feature is that I would like to be able to change the hardware of my machine and boot up differently based on what's present. In my case, I would like to install my system normally, with /boot on its own partition, and keep a seperate userspace on a flash drive. Then, during boot-up, if the flash drive is present, it would be used as the root device. If it's not present, a partition on disk would be selected.

I think this is an excellent idea, in addition to the above use-case, it would allow for distros to automatically launch a recovery image if the main root device has failed for some reason.

That said, using the term failover for this is probably not the best idea, many people associate it almost exclusively with online failover and high-availability setups, and trying to do something like that with the root file system is just asking for trouble (I'll be happy to go into specifics as to why if someone asks).

Quoted 13 lines

The only potential roadblock with this feature that comes to mind is figuring out how to handle time-outs between root devices. I think it would be wise to choose a sensible default value, and provide another cmdline parameter to tweak it. The prepare_namespace flow might end up looking something like this:

1. Wait rootdelay seconds
2. Check 1st device, not present
3. Recheck 1st device until rootfailoverdelay seconds has passed
4. Move on to 2nd device, present -> boot

Or:

Quoted 8 lines

1. Wait rootdelay seconds
2. Check 1st device, not present
3. Recheck 1st device until rootfailoverdelay seconds has passed
4. Move on to 2nd device, not present
5. Recheck 2st device until rootfailoverdelay seconds has passed
6. GOTO 2

And so on.

As for this, I'd say default to the first method, and then provide an option to switch to the second (both have practical uses).

Quoted 5 lines

I also need to research how the various init systems interact with this part of the boot process. I suspect systemd probably does something silly wrt waiting for the root device. Since this feature would (of course) be backwards compatible, it might be wise to just implement it here and let the init systems add support for the feature themselves.

If you're using an initramfs (which is a requirement from what I understand for using systemd), then this could be done entirely in the initramfs. The issue with that is that there is no standard syntax for doing it, and no way to do it without an initramfs (both of which would be nice to have).

Quoted 2 lines

Advice? Who should I send my patches to when they're ready? Please CC me, I do not subscribe to LKML.

Use scripts/getmaintainer.pl (or just check the MAINTAINERS file directly) to determine this, but make sure to Cc at least LKML for the changes as well.

#2 lore raw
Quoted 5 lines

That said, using the term failover for this is probably not the best idea, many people associate it almost exclusively with online failover and high-availability setups, and trying to do something like that with the root file system is just asking for trouble (I'll be happy to go into specifics as to why if someone asks).

Do you have a suggestion for another name for this feature? Maybe we can just call it "multiple root devices". The issue comes with the associated command line options, like "rootfailoverdelay". Perhaps it could be called "rootcycledelay". "rootdelay" is the obvious one, but it's taken for another feature.

Quoted 14 lines
Quoted 14 lines

1. Wait rootdelay seconds
2. Check 1st device, not present
3. Recheck 1st device until rootfailoverdelay seconds has passed
4. Move on to 2nd device, present -> boot

Or:

1. Wait rootdelay seconds
2. Check 1st device, not present
3. Recheck 1st device until rootfailoverdelay seconds has passed
4. Move on to 2nd device, not present
5. Recheck 2st device until rootfailoverdelay seconds has passed
6. GOTO 2

Quoted 3 lines
Quoted 1 line

And so on.

As for this, I'd say default to the first method, and then provide an option to switch to the second (both have practical uses).

Sorry to cause confusion - these are actually the same method, but handling different scenarios. The first is dealing with the first device being nonexistent, and the second device existing. The second is dealing with both being nonexistent, and cycling between them until one of them shows up. After further thought, though, I think the best solution is a bit different: a new command line option called "rootmultiwait" or similar, which is a maximum amount of time to wait for the user's first choice of root device to become available, then testing all devices until that time runs out or the first choice becomes available.

--
Drew DeVault

#3 lore raw
Quoted 11 lines · Drew DeVault
Quoted 5 lines

That said, using the term failover for this is probably not the best idea, many people associate it almost exclusively with online failover and high-availability setups, and trying to do something like that with the root file system is just asking for trouble (I'll be happy to go into specifics as to why if someone asks).

Do you have a suggestion for another name for this feature? Maybe we can just call it "multiple root devices". The issue comes with the associated command line options, like "rootfailoverdelay". Perhaps it could be called "rootcycledelay". "rootdelay" is the obvious one, but it's taken for another feature.

Possibly 'multirootdelay'?

However, is there any case you can think of for wanting the values to be different between rootdelay and the per-device scan delay other than having the per-device scan delay be 0 and rootdelay be >0?

The way I'd probably write it would be:
1. Wait rootdelay seconds
2. Check for 1st device
3. If first device is not there, check for 2nd
4. If second device is not there, check next one
5. Repeat 4 until all devices are checked.
6. If a device wasn't found, check if we were told to loop until one is found, and if so, start at 1 again. And then add an option to tell it to wait 'rootdelay' seconds between checking each device.

Quoted 19 lines
Quoted 15 lines
Quoted 15 lines

1. Wait rootdelay seconds
2. Check 1st device, not present
3. Recheck 1st device until rootfailoverdelay seconds has passed
4. Move on to 2nd device, present -> boot

Or:

1. Wait rootdelay seconds
2. Check 1st device, not present
3. Recheck 1st device until rootfailoverdelay seconds has passed
4. Move on to 2nd device, not present
5. Recheck 2st device until rootfailoverdelay seconds has passed
6. GOTO 2

Quoted 3 lines
Quoted 1 line

And so on.

As for this, I'd say default to the first method, and then provide an option to switch to the second (both have practical uses).

Quoted 9 lines

Sorry to cause confusion - these are actually the same method, but handling different scenarios. The first is dealing with the first device being nonexistent, and the second device existing. The second is dealing with both being nonexistent, and cycling between them until one of them shows up. After further thought, though, I think the best solution is a bit different: a new command line option called "rootmultiwait" or similar, which is a maximum amount of time to wait for the user's first choice of root device to become available, then testing all devices until that time runs out or the first choice becomes available.

I think there's value in being able to tell it to go through each one exactly once, and halt like it does now if it can't find the filesystem on any of them. That should probably be the default behavior in fact, as it's more similar to what's done now.

Secondarily, I've been thinking more about this, and I think it would be wonderful to have such functionality in the nfsroot code as well (and for that matter, also in any other built-in networked root filesystem support).

#4 lore raw
Quoted 1 line

Possibly 'multirootdelay'?

I had the same thought, but wanted to avoid using any prefix other than root*= since it would break tradition for this part of the kernel.

Quoted 3 lines

However, is there any case you can think of for wanting the values to be different between rootdelay and the per-device scan delay other than having the per-device scan delay be 0 and rootdelay be >0?

rootdelay is not really a part of this flow, to be honest. It's a number of seconds that it just blocks before trying any devices at all. It might not be necessary to make them seperate, though. What's the "per-device scan delay" you're speaking of?

Quoted 10 lines

The way I'd probably write it would be:
1. Wait rootdelay seconds
2. Check for 1st device
3. If first device is not there, check for 2nd
4. If second device is not there, check next one
5. Repeat 4 until all devices are checked.
6. If a device wasn't found, check if we were told to loop until one is found, and if so, start at 1 again.
And then add an option to tell it to wait 'rootdelay' seconds between checking each device.

-snip-

Quoted 4 lines

I think there's value in being able to tell it to go through each one exactly once, and halt like it does now if it can't find the filesystem on any of them. That should probably be the default behavior in fact, as it's more similar to what's done now.

What is the behavior if we weren't told to loop, but reach the end of the list? I don't want to try one device for a while and move on - I'd prefer to try all devices, then loop. With the other strategy, what happens if you try the first device for a while, then move on to the second, and the first device comes up while you're waiting on the second? The way the user would want that to play out is to boot the first device (since it is their preference, after all), but the actual behavior will be to boot the second if it comes up during this time.

Quoted 4 lines

Secondarily, I've been thinking more about this, and I think it would be wonderful to have such functionality in the nfsroot code as well (and for that matter, also in any other built-in networked root filesystem support).

I agree that it would be great, but there be dragons. I'm nervous to attempt that in my first patch (or even my first few).

--
Drew DeVault