Skip to contentSkip to conversation

Dynamic device.map

RFC In review Colin Watson 5 participants 20 messages Dec 7, 2009 – Jan 8, 2010

lore
Share & tools
lore
#0 lore raw

RFC 1/1 Dynamic device.map

As Robert said recently, we're trying to get rid of our reliance on device.map. Right now, it is still necessary to at least have entries in device.map for any disks you wish to use with GRUB, although they don't have to be particularly sensible; any bidirectional mapping will do (at least as long as the 'search' command works).

I'd like to make it possible to run with no device.map at all, since that makes it possible to plug in new disks without having to reconfigure GRUB. Here's a draft patch to do this. I haven't supplied a ChangeLog entry yet since it isn't suitable for commit yet, as noted in TODO comments, but I'd welcome comments on the approach here.

This does slow down (e.g.) grub-probe a little: on my laptop with a single hard disk, the median time for three runs of './grub-probe --target=drive /' is 0.09 seconds, while the median time for three runs of './grub-probe --target=drive -m /boot/grub/device.map.invalid /' is 0.18 seconds. I expect it would be worse on machines with many disks. For the most part, I doubt this is a significant problem (certainly not in e.g. grub-setup), but it will probably make a difference in scripts that call grub-probe many times, such as grub-mkconfig. I suggest that those should check whether the default device.map exists, and, if not, send the output of grub-mkdevicemap to a temporary file and use that as an ephemeral device.map cache.

=== modified file 'conf/any-emu.rmk'

any-emu.rmk +1 -1
@@ -40,7 +40,7 @@ grub_emu_SOURCES = commands/minicmd.c co
 	fs/befs.c fs/befs_be.c fs/tar.c					\
 	\
 	util/console.c util/hostfs.c util/grub-emu.c util/misc.c	\
-	util/hostdisk.c util/getroot.c					\
+	util/hostdisk.c util/getroot.c util/deviceiter.c		\
 	\
 	disk/raid.c disk/raid5_recover.c disk/raid6_recover.c		\
 	disk/mdraid_linux.c disk/dmraid_nvidia.c disk/lvm.c		\

=== modified file 'conf/common.rmk'

common.rmk +1 -0
@@ -17,6 +17,7 @@ sbin_UTILITIES += grub-probe
 util/grub-probe.c_DEPENDENCIES = grub_probe_init.h
 grub_probe_SOURCES = gnulib/progname.c util/grub-probe.c	\
 	util/hostdisk.c	util/misc.c util/getroot.c		\
+	util/deviceiter.c					\
 	kern/device.c kern/disk.c kern/err.c kern/misc.c	\
 	kern/parser.c kern/partition.c kern/file.c		\
 	\

=== modified file 'conf/i386-pc.rmk'

i386-pc.rmk +1 -0
@@ -93,6 +93,7 @@ util/i386/pc/grub-mkimage.c_DEPENDENCIES
 util/i386/pc/grub-setup.c_DEPENDENCIES = grub_setup_init.h
 grub_setup_SOURCES = gnulib/progname.c \
 	util/i386/pc/grub-setup.c util/hostdisk.c	\
+	util/deviceiter.c					\
 	util/misc.c util/getroot.c kern/device.c kern/disk.c	\
 	kern/err.c kern/misc.c kern/parser.c kern/partition.c	\
 	kern/file.c kern/fs.c kern/env.c fs/fshelp.c		\

=== modified file 'conf/sparc64-ieee1275.rmk'

sparc64-ieee1275.rmk +1 -0
@@ -68,6 +68,7 @@ grub_mkimage_SOURCES = util/sparc64/ieee
 # For grub-setup.
 util/sparc64/ieee1275/grub-setup.c_DEPENDENCIES = grub_setup_init.h
 grub_setup_SOURCES = util/sparc64/ieee1275/grub-setup.c util/hostdisk.c	\
+	util/deviceiter.c					\
 	util/misc.c util/getroot.c kern/device.c kern/disk.c	\
 	kern/err.c kern/misc.c kern/parser.c kern/partition.c	\
 	kern/file.c kern/fs.c kern/env.c fs/fshelp.c		\

=== modified file 'util/hostdisk.c'

hostdisk.c +55 -0

73 lines not on this page · raw message

Thanks,

Signature · 1 line
Colin Watson                                       [[email protected]]
#1 lore raw
Quoted 11 lines · Colin Watson

As Robert said recently, we're trying to get rid of our reliance on device.map. Right now, it is still necessary to at least have entries in device.map for any disks you wish to use with GRUB, although they don't have to be particularly sensible; any bidirectional mapping will do (at least as long as the 'search' command works).

I'd like to make it possible to run with no device.map at all, since that makes it possible to plug in new disks without having to reconfigure GRUB. Here's a draft patch to do this. I haven't supplied a ChangeLog entry yet since it isn't suitable for commit yet, as noted in TODO comments, but I'd welcome comments on the approach here.

Ah, I hadn't noticed r1870, sorry. Am I right in believing that my patch is obsolete, then?

Signature · 1 line
Colin Watson                                       [[email protected]]
#2 lore raw
Quoted 7 lines · Colin Watson
Quoted 6 lines · Colin Watson

As Robert said recently, we're trying to get rid of our reliance on device.map. Right now, it is still necessary to at least have entries in device.map for any disks you wish to use with GRUB, although they don't have to be particularly sensible; any bidirectional mapping will do (at least as long as the 'search' command works).

Quoted 8 lines
Quoted 5 lines

I'd like to make it possible to run with no device.map at all, since that makes it possible to plug in new disks without having to reconfigure GRUB. Here's a draft patch to do this. I haven't supplied a ChangeLog entry yet since it isn't suitable for commit yet, as noted in TODO comments, but I'd welcome comments on the approach here.

Ah, I hadn't noticed r1870, sorry. Am I right in believing that my patch is obsolete, then?

I'm not completely sure, but it seems so. Sorry about that :-(

Anyhow, if you find more issues with systems that have no device.map, help is welcome fixing those of course.

Signature · 5 lines
Robert Millan

  The DRM opt-in fallacy: "Your data belongs to us. We will decide when (and
  how) you may access your data; but nobody's threatening your freedom: we
  still allow you to remove your data and not access it at all."
#3 lore raw
Quoted 5 lines · Robert Millan
Quoted 2 lines · Colin Watson

Ah, I hadn't noticed r1870, sorry. Am I right in believing that my patch is obsolete, then?

I'm not completely sure, but it seems so. Sorry about that :-(

Don't apologise for doing something before I got round to it! :-)

Quoted 2 lines

Anyhow, if you find more issues with systems that have no device.map, help is welcome fixing those of course.

I'm trying to figure out how to make Debian's grub-installer operate without a device.map; it has various legacy bits and pieces that need conversion, and I'm working on these.

Along the way, though, I noticed that grub-install still unconditionally creates a device.map. This seems likely to become confusing if devices are changing around a lot, since it's never updated. I'd like to get to the point where it doesn't do this by default. How about we add support for an option to disable this, and make grub-installer and the Debian maintainer scripts pass it once they're ready? Some time later, we can flip the default value.

Signature · 1 line
Colin Watson                                       [[email protected]]
#4 lore raw
Quoted 5 lines · Colin Watson

I'm trying to figure out how to make Debian's grub-installer operate without a device.map; it has various legacy bits and pieces that need conversion, and I'm working on these.

Quoted 7 lines

Along the way, though, I noticed that grub-install still unconditionally creates a device.map. This seems likely to become confusing if devices are changing around a lot, since it's never updated. I'd like to get to the point where it doesn't do this by default. How about we add support for an option to disable this, and make grub-installer and the Debian maintainer scripts pass it once they're ready? Some time later, we can flip the default value.

Why is this option necessary? If we removed automated generation of device.map, user can still call grub-mkdevicemap manually.

But first we'd need to figure out what we do with the "set root=xxx" backward compatibility hack. Has it been a while long enough that we can remove support for GRUB installs that didn't come with UUID support?

If we don't do anything on grub-mkconfig side and remove device.map, we'll end up with things like "set root=(/dev/sda1)" which is not just useless but also utterly confusing.

Signature · 5 lines
Robert Millan

  The DRM opt-in fallacy: "Your data belongs to us. We will decide when (and
  how) you may access your data; but nobody's threatening your freedom: we
  still allow you to remove your data and not access it at all."
#5 lore raw
Quoted 13 lines · Robert Millan
Quoted 11 lines · Colin Watson

I'm trying to figure out how to make Debian's grub-installer operate without a device.map; it has various legacy bits and pieces that need conversion, and I'm working on these.

Along the way, though, I noticed that grub-install still unconditionally creates a device.map. This seems likely to become confusing if devices are changing around a lot, since it's never updated. I'd like to get to the point where it doesn't do this by default. How about we add support for an option to disable this, and make grub-installer and the Debian maintainer scripts pass it once they're ready? Some time later, we can flip the default value.

Quoted 2 lines

Why is this option necessary? If we removed automated generation of device.map, user can still call grub-mkdevicemap manually.

Certainly, but I would like to be able to deliver a system that does not generate a device.map by default.

Quoted 8 lines

But first we'd need to figure out what we do with the "set root=xxx" backward compatibility hack. Has it been a while long enough that we can remove support for GRUB installs that didn't come with UUID support?

If we don't do anything on grub-mkconfig side and remove device.map, we'll end up with things like "set root=(/dev/sda1)" which is not just useless but also utterly confusing.

No, I don't think we will. grub-probe is perfectly capable of mapping /dev/sda1 to (hd0,1) even without device.map; it's just that that mapping will be solely dependent on device iteration order at the time you run grub-mkconfig. (As opposed to whatever device iteration order happened to be the first time you installed GRUB, for the nearly-everyone who never edited device.map by hand ...)

Signature · 1 line
Colin Watson                                       [[email protected]]
#6 lore raw
Quoted 14 lines · Colin Watson
Quoted 13 lines · Robert Millan
Quoted 11 lines · Colin Watson

I'm trying to figure out how to make Debian's grub-installer operate without a device.map; it has various legacy bits and pieces that need conversion, and I'm working on these.

Along the way, though, I noticed that grub-install still unconditionally creates a device.map. This seems likely to become confusing if devices are changing around a lot, since it's never updated. I'd like to get to the point where it doesn't do this by default. How about we add support for an option to disable this, and make grub-installer and the Debian maintainer scripts pass it once they're ready? Some time later, we can flip the default value.

Quoted 5 lines
Quoted 2 lines

Why is this option necessary? If we removed automated generation of device.map, user can still call grub-mkdevicemap manually.

Certainly, but I would like to be able to deliver a system that does not generate a device.map by default.

We don't need a new option for that. We'd just not generate device.map at all, and provide grub-mkdevicemap for users who want to mess with it.

Quoted 2 lines

No, I don't think we will. grub-probe is perfectly capable of mapping /dev/sda1 to (hd0,1) even without device.map;

I wasn't very fond of this because it's BIOS-specific. But I guess we can live with it.

Signature · 3 lines
Robert Millan

  "Be the change you want to see in the world" -- Gandhi
#7 lore raw
Quoted 5 lines · Robert Millan
Quoted 2 lines

No, I don't think we will. grub-probe is perfectly capable of mapping /dev/sda1 to (hd0,1) even without device.map;

I wasn't very fond of this because it's BIOS-specific. But I guess we can live with it.

On second thought, I'm really not convinced we need this kind of logic. It seems a lot like a workaround.

Just leave it with (/dev/foo). If you remove device.map generation from grub-install, you can remove the backward compatibility hack while at it. Then we rely only on UUIDs.

In the future, we can rely on the initial value of $root as well, but this would require some rework (disk / partition split).

Signature · 3 lines
Robert Millan

  "Be the change you want to see in the world" -- Gandhi
#8 lore raw

From: Robert Millan <[email protected]>
Date: Sat, 26 Dec 2009 14:25:51 +0100

Quoted 1 line

Then we rely only on UUIDs.

This is exactly what I suggested we should avoid.

On OpenFirmware we scan every device alias, %80 of them point to non-existing devices. The aliases are just an enumeration of the set of devices that could be behind a controller, not what has actually been detected by the firmware or anything like that.

So every scan for a UUID can eat up an enormous amount of time, poking devices that simply are not there.

#9 lore raw
Quoted 7 lines · David Miller

From: Robert Millan <[email protected]>
Date: Sat, 26 Dec 2009 14:25:51 +0100

Quoted 1 line

Then we rely only on UUIDs.

This is exactly what I suggested we should avoid.

Quoted 8 lines

On OpenFirmware we scan every device alias, %80 of them point to non-existing devices. The aliases are just an enumeration of the set of devices that could be behind a controller, not what has actually been detected by the firmware or anything like that.

So every scan for a UUID can eat up an enormous amount of time, poking devices that simply are not there.

I understand. It's not a good idea to use UUIDs exclussively even

Quoted 2 lines · outside the context of OpenFirmware. This is why I also

In the future, we can rely on the initial value of $root as well, but this would require some rework (disk / partition split).

I'm fine with delaying removal of devices.map untill this replacement is ready.

Signature · 3 lines
Robert Millan

  "Be the change you want to see in the world" -- Gandhi
#10 lore raw
In reply to Robert Millan
Quoted 5 lines · Robert Millan
Quoted 4 lines · Robert Millan
Quoted 2 lines · Colin Watson

No, I don't think we will. grub-probe is perfectly capable of mapping /dev/sda1 to (hd0,1) even without device.map;

Quoted 7 lines
Quoted 2 lines

I wasn't very fond of this because it's BIOS-specific. But I guess we can live with it.

On second thought, I'm really not convinced we need this kind of logic. It seems a lot like a workaround.

Just leave it with (/dev/foo).

You mean literally with the parentheses? I don't understand, since /dev/ names will be unintelligible to GRUB when running outside an operating system.

Quoted 2 lines

If you remove device.map generation from grub-install, you can remove the backward compatibility hack while at it.

Which backward compatibility hack?

Quoted 2 lines

In the future, we can rely on the initial value of $root as well, but this would require some rework (disk / partition split).

Would you mind elaborating on this? I would like to get this sorted out and am willing to work on it.

Sorry to be stupid,

Signature · 1 line
Colin Watson                                       [[email protected]]
#11 lore raw
Quoted 5 lines · Colin Watson
Quoted 1 line

Just leave it with (/dev/foo).

You mean literally with the parentheses? I don't understand, since /dev/ names will be unintelligible to GRUB when running outside an operating system.

Yes. This just means we'd have "set root=(/dev/foo)" statements in grub.cfg, but those are just meant as a backward compatibility hack for pre-UUID GRUB installs.

Which I guess addresses your next question:

Quoted 4 lines
Quoted 2 lines

If you remove device.map generation from grub-install, you can remove the backward compatibility hack while at it.

Which backward compatibility hack?

:-)

However, notice that after David Miller's remark about Sparc support, it's probably better to:

- Keep the backward compatibility hack.

- Not remove device.map on Sparc yet (untill a better solution is available)

Quoted 5 lines
Quoted 2 lines

In the future, we can rely on the initial value of $root as well, but this would require some rework (disk / partition split).

Would you mind elaborating on this? I would like to get this sorted out and am willing to work on it.

Uhm actually I'm not completely sure this would work, as multi-disk setups have multiple values for initial $root.

Better just ignore this part ;-)

Quoted 1 line

Sorry to be stupid,

Not at all!

Signature · 3 lines
Robert Millan

  "Be the change you want to see in the world" -- Gandhi
#12 lore raw
Quoted 9 lines · Robert Millan
Quoted 4 lines · Colin Watson
Quoted 1 line

Just leave it with (/dev/foo).

You mean literally with the parentheses? I don't understand, since /dev/ names will be unintelligible to GRUB when running outside an operating system.

Yes. This just means we'd have "set root=(/dev/foo)" statements in grub.cfg, but those are just meant as a backward compatibility hack for pre-UUID GRUB installs.

Are you are implying that UUID will be the only way? I don't use initrd's on my systems so I need root=(hd0,x) or root=(/dev/foo). AFAIK initrd is the only way to load with UUIDs.

-- Bruce

#13 lore raw
Quoted 7 lines · Bruce Dubbs
Quoted 6 lines · Robert Millan
Quoted 4 lines · Colin Watson
Quoted 1 line

Just leave it with (/dev/foo).

You mean literally with the parentheses? I don't understand, since /dev/ names will be unintelligible to GRUB when running outside an operating system.

Quoted 7 lines
Quoted 3 lines

Yes. This just means we'd have "set root=(/dev/foo)" statements in grub.cfg, but those are just meant as a backward compatibility hack for pre-UUID GRUB installs.

Are you are implying that UUID will be the only way? I don't use initrd's on my systems so I need root=(hd0,x) or root=(/dev/foo). AFAIK initrd is the only way to load with UUIDs.

I think you're mixing up two different things. There are two 'root' variables involved:

1) GRUB's 'root' variable, its base for filesystem operations
2) The root= parameter passed to the Linux kernel, which identifies the desired root filesystem

Robert is talking about 1), but whether you use an initrd/initramfs is only relevant to 2).

Signature · 1 line
Colin Watson                                       [[email protected]]
#14 lore raw
Quoted 16 lines · Colin Watson
Quoted 11 lines · Bruce Dubbs
Quoted 8 lines · Robert Millan
Quoted 4 lines · Colin Watson
Quoted 1 line

Just leave it with (/dev/foo).

You mean literally with the parentheses? I don't understand, since /dev/ names will be unintelligible to GRUB when running outside an operating system.

Yes. This just means we'd have "set root=(/dev/foo)" statements in grub.cfg, but those are just meant as a backward compatibility hack for pre-UUID GRUB installs.

Are you are implying that UUID will be the only way? I don't use initrd's on my systems so I need root=(hd0,x) or root=(/dev/foo). AFAIK initrd is the only way to load with UUIDs.

I think you're mixing up two different things. There are two 'root' variables involved:

Quoted 6 lines

1) GRUB's 'root' variable, its base for filesystem operations
2) The root= parameter passed to the Linux kernel, which identifies the desired root filesystem

Robert is talking about 1), but whether you use an initrd/initramfs is only relevant to 2).

OK, I didn't realize set root was capable of using UUIDs. I did know that the two root entries were different. I got that mixed up with the search command combined with the root=UUID=... which I think needs initrd.

Do I have it right now?

Should 'set root' be renamed to 'set grubroot'? I think something like that would prevent some confusion.

-- Bruce

#15 lore raw

Am Donnerstag, den 07.01.2010, 23:27 -0600 schrieb Bruce Dubbs:

Quoted 31 lines
Quoted 5 lines · Colin Watson
Quoted 4 lines · Bruce Dubbs
Quoted 3 lines · Robert Millan
Quoted 2 lines · Colin Watson
Quoted 1 line

Just leave it with (/dev/foo).

You mean literally with the parentheses? I don't understand,

since /dev/

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

names will be unintelligible to GRUB when running outside an

operating

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 1 line

system.

Yes. This just means we'd have "set root=(/dev/foo)" statements

in grub.cfg,

Quoted 1 line
Quoted 1 line
Quoted 1 line

but those are just meant as a backward compatibility hack for

pre-UUID GRUB

Quoted 10 lines
Quoted 4 lines
Quoted 1 line

installs.

Are you are implying that UUID will be the only way? I don't use initrd's on my systems so I need root=(hd0,x) or root=(/dev/foo). AFAIK initrd is the only way to load with UUIDs.

I think you're mixing up two different things. There are two 'root' variables involved:

1) GRUB's 'root' variable, its base for filesystem operations
2) The root= parameter passed to the Linux kernel, which

identifies

Quoted 3 lines

the desired root filesystem

Robert is talking about 1), but whether you use an initrd/initramfs

is

Quoted 1 line

only relevant to 2).

Quoted 7 lines

OK, I didn't realize set root was capable of using UUIDs. I did know that the two root entries were different. I got that mixed up with the
search command combined with the root=UUID=... which I think needs initrd.

Do I have it right now?

set root only supports plain GRUB devices.
search --set sets $root (for GRUB, not for Linux) to the first device found with that UUID, LABEL or file. Depending on the other argument to search command.

Quoted 5 lines

Should 'set root' be renamed to 'set grubroot'? I think something like
that would prevent some confusion.

-- Bruce

I don't get why there's a confusion at all.
It should be clear that the value specified with the linux command is for it and not for GRUB.

Signature · 2 lines
Felix Zielcke
Proud Debian Maintainer and GNU GRUB developer
#16 lore raw
Quoted 13 lines · Felix Zielcke

Am Donnerstag, den 07.01.2010, 23:27 -0600 schrieb Bruce Dubbs:

Quoted 5 lines

OK, I didn't realize set root was capable of using UUIDs. I did know that the two root entries were different. I got that mixed up with the search command combined with the root=UUID=... which I think needs
initrd.

Do I have it right now?

set root only supports plain GRUB devices.
search --set sets $root (for GRUB, not for Linux) to the first device found with that UUID, LABEL or file. Depending on the other argument to search command.

Quoted 6 lines
Quoted 2 lines

Should 'set root' be renamed to 'set grubroot'? I think something like that would prevent some confusion.

I don't get why there's a confusion at all.
It should be clear that the value specified with the linux command is for it and not for GRUB.

It's not confusing if you've been around GRUB for a while, but it is confusing for new users. The term 'root' is overloaded and only distinguished by context. The two contexts do have similar meanings so the differences are really fairly subtle.

-- Bruce

#17 lore raw
In reply to Robert Millan

Am Donnerstag, den 10.12.2009, 01:55 +0100 schrieb Robert Millan:

Quoted 4 lines

But first we'd need to figure out what we do with the "set root=xxx" backward compatibility hack. Has it been a while long enough that we can remove support for GRUB installs that didn't come with UUID support?

Well we still have Arthur Marsh' bug open that search --fs-uuid fails with right UUID even though ls (hdx,y) shows it.

https://savannah.gnu.org/bugs/?26834
Also on Debian BTS and Launchpad.

The good thing though is that only a grub_errno = GRUB_ERR_NONE; needs to be commited to have this fixed.
Unfortunately he attached the full kern/device.c to the report not the patch Vladimir made for him.

Signature · 2 lines
Felix Zielcke
Proud Debian Maintainer and GNU GRUB developer
#18 lore raw
Quoted 8 lines · Felix Zielcke

Am Donnerstag, den 10.12.2009, 01:55 +0100 schrieb Robert Millan:

Quoted 4 lines

But first we'd need to figure out what we do with the "set root=xxx" backward compatibility hack. Has it been a while long enough that we can remove support for GRUB installs that didn't come with UUID support?

Well we still have Arthur Marsh' bug open that search --fs-uuid fails with right UUID even though ls (hdx,y) shows it.

Uhm ISTR Vladimir fixed this one.

Signature · 3 lines
Robert Millan

  "Be the change you want to see in the world" -- Gandhi
#19 lore raw

Am Donnerstag, den 24.12.2009, 22:17 +0100 schrieb Robert Millan:

Quoted 8 lines · Felix Zielcke
Quoted 2 lines

Am Donnerstag, den 10.12.2009, 01:55 +0100 schrieb Robert Millan:

Quoted 1 line

But first we'd need to figure out what we do with the "set

root=xxx"

Quoted 4 lines
Quoted 3 lines

backward compatibility hack. Has it been a while long enough that we can remove support for GRUB installs that didn't come with UUID support?

Quoted 5 lines
Quoted 1 line

Well we still have Arthur Marsh' bug open that search --fs-uuid

fails

Quoted 1 line

with right UUID even though ls (hdx,y) shows it.

Uhm ISTR Vladimir fixed this one.

Yes in the meanwhile he commited the fix.

Signature · 2 lines
Felix Zielcke
Proud Debian Maintainer and GNU GRUB developer