Skip to contentSkip to conversation

Linux 2.4.2ac20

DISCUSSION David Balazic 12 participants 21 messages Mar 13, 2001 – Mar 16, 2001

lore
Share & tools
lore
#0 lore raw
Quoted 5 lines · Nathan Walp

Also, sometime between ac7 and ac18 (spring break kept me from testing stuff inbetween), i assume during the new aic7xxx driver merge, the order of detection got changed, and now the ide-scsi virtual host is host0, and my 29160N is host1. Is this on purpose? It messed up a bunch of my stuff as far as /dev and such are concerned.

SCSI adapters are enumerated randomly(*) , relying on certain numbering will get you into trouble, sooner or later.
There is no commonly accepted solution, AFAIK.
The same thing can happent to disk enumeration ( sdb becomes sdc ) or partition enumeration ( hda6 becomes hda5 ).

* - theoreticaly no, but practicaly yes ( most of the time )

Signature · 4 lines
David Balazic
--------------
"Be excellent to each other." - Bill & Ted
- - - - - - - - - - - - - - - - - - - - - -
#1 lore raw
Quoted 15 lines · David Balazic
Quoted 5 lines · Nathan Walp

Also, sometime between ac7 and ac18 (spring break kept me from testing stuff inbetween), i assume during the new aic7xxx driver merge, the order of detection got changed, and now the ide-scsi virtual host is host0, and my 29160N is host1. Is this on purpose? It messed up a bunch of my stuff as far as /dev and such are concerned.

SCSI adapters are enumerated randomly(*) , relying on certain numbering will get you into trouble, sooner or later.
There is no commonly accepted solution, AFAIK.
The same thing can happent to disk enumeration ( sdb becomes sdc ) or partition enumeration ( hda6 becomes hda5 ).

Quoted 7 lines

* - theoreticaly no, but practicaly yes ( most of the time )

--
David Balazic
--------------
"Be excellent to each other." - Bill & Ted
- - - - - - - - - - - - - - - - - - - - - -

SCSI adapters are given host numbers in a random order? Even with no hardware changes? Does this make less than sense to anyone else? Every kernel EVER up till now has had the real scsi cards (in some particular order) then ide-scsi. Have I just been lucky???

Nathan

#2 lore raw
In reply to David Balazic
Quoted 8 lines · Nathan Walp
Quoted 7 lines · David Balazic

SCSI adapters are enumerated randomly(*) , relying on certain numbering will get you into trouble, sooner or later.
There is no commonly accepted solution, AFAIK.
The same thing can happent to disk enumeration ( sdb becomes sdc ) or partition enumeration ( hda6 becomes hda5 ).

Quoted 7 lines
Quoted 1 line

* - theoreticaly no, but practicaly yes ( most of the time )

SCSI adapters are given host numbers in a random order? Even with no hardware changes? Does this make less than sense to anyone else? Every kernel EVER up till now has had the real scsi cards (in some particular order) then ide-scsi. Have I just been lucky???

No, it's not truly random. That would be insane. And, no, if you don't change the kernel or the hardware, then they won't jump around.
But, yes, if you change the hardware, or someone changes the probe order in the kernel, you can suffer from device name slippage. This is a minor problem on a small home system, and a massive PITA on a large server. You can at least mandate the probe order on a 2.4 system (see the scsihosts parameter).

Tim

Signature · 4 lines
Tim Wright - [email protected] or [email protected] or [email protected]
IBM Linux Technology Center, Beaverton, Oregon
Interested in Linux scalability ? Look at http://lse.sourceforge.net/
"Nobody ever said I was charming, they said "Rimmer, you're a git!"" RD VI
#3 lore raw
In reply to David Balazic
Quoted 25 lines · Nathan Walp
Quoted 15 lines · David Balazic
Quoted 5 lines · Nathan Walp

Also, sometime between ac7 and ac18 (spring break kept me from testing stuff inbetween), i assume during the new aic7xxx driver merge, the order of detection got changed, and now the ide-scsi virtual host is host0, and my 29160N is host1. Is this on purpose? It messed up a bunch of my stuff as far as /dev and such are concerned.

SCSI adapters are enumerated randomly(*) , relying on certain numbering will get you into trouble, sooner or later.
There is no commonly accepted solution, AFAIK.
The same thing can happent to disk enumeration ( sdb becomes sdc ) or partition enumeration ( hda6 becomes hda5 ).

Quoted 7 lines

* - theoreticaly no, but practicaly yes ( most of the time )

--
David Balazic
--------------
"Be excellent to each other." - Bill & Ted
- - - - - - - - - - - - - - - - - - - - - -

Quoted 6 lines

SCSI adapters are given host numbers in a random order? Even with no hardware changes? Does this make less than sense to anyone else? Every kernel EVER up till now has had the real scsi cards (in some particular order) then ide-scsi. Have I just been lucky???

Nathan

What I mean that too many factors are affecting the enumeration, so that you can not rely on it :

-  kernel changes
-  driver changes ( partly overlaps with the previous poit )
-  hardware changes
-  something else ?

There is no policy for this enumeration ( AFAIK ) , so there is nothing to rely on , except luck :-)

Signature · 4 lines
David Balazic
--------------
"Be excellent to each other." - Bill & Ted
- - - - - - - - - - - - - - - - - - - - - -
#4 lore raw
In reply to David Balazic
Quoted 42 lines · David Balazic
Quoted 25 lines · Nathan Walp
Quoted 15 lines · David Balazic
Quoted 5 lines · Nathan Walp

Also, sometime between ac7 and ac18 (spring break kept me from testing stuff inbetween), i assume during the new aic7xxx driver merge, the order of detection got changed, and now the ide-scsi virtual host is host0, and my 29160N is host1. Is this on purpose? It messed up a bunch of my stuff as far as /dev and such are concerned.

SCSI adapters are enumerated randomly(*) , relying on certain numbering will get you into trouble, sooner or later.
There is no commonly accepted solution, AFAIK.
The same thing can happent to disk enumeration ( sdb becomes sdc ) or partition enumeration ( hda6 becomes hda5 ).

Quoted 7 lines

* - theoreticaly no, but practicaly yes ( most of the time )

--
David Balazic
--------------
"Be excellent to each other." - Bill & Ted
- - - - - - - - - - - - - - - - - - - - - -

Quoted 6 lines

SCSI adapters are given host numbers in a random order? Even with no hardware changes? Does this make less than sense to anyone else? Every kernel EVER up till now has had the real scsi cards (in some particular order) then ide-scsi. Have I just been lucky???

Nathan

What I mean that too many factors are affecting the enumeration, so that you can not rely on it :

(quoted diff) +0 -4
-  kernel changes
-  driver changes ( partly overlaps with the previous poit )
-  hardware changes
-  something else ?
Quoted 2 lines

There is no policy for this enumeration ( AFAIK ) , so there is nothing to rely on , except luck :-)

See, that all makes sense. You can't depend on hardware to detect in the same order, whether it's SCSI cards, network cards, or anything really. But the software psuedo-device that is ide-scsi, shouldn't that pick a spot before or after the hardware and stay there? That's my point, i guess.

Nathan

#5 lore raw

magic device renumbering was -- Re: Linux 2.4.2ac20

The problem:

drivers change their detection schemes; and changes in the kernel can change the order in which devices are assigned names.

For example, the DAC960(?) drivers changed their order of detecting controllers, and I did _not_ have fun, given that the machine in question had about 40 disks to deal with, spread across two controllers.

This can create a lot of problems for people upgrading large, production quality systems -- as, in the worst case, the system won't complete the boot cycle; or in middle cases, the user/sysadmin is stuck rewriting X amount of files and trying again; or in small cases, you find out that your SMC and Intel ethernet cards are reversed, and have to go fix things
...

Possible solutions(?):

Solaris uses an /etc/path_to_inst file, to keep track of device ordering, et al.

Maybe we should consider something similar, where a physical device to logical device map is kept and used to keep things consistent on kernel/driver changes; device addition/removal, and so forth ...

I am, of course, open to better solutions.

--
-- John E. Jasen ([email protected])
-- In theory, theory and practise are the same. In practise, they aren't.

#6 lore raw

Re: magic device renumbering was -- Re: Linux 2.4.2ac20

Quoted 8 lines · In article <[email protected]>…

The problem:

drivers change their detection schemes; and changes in the kernel can change the order in which devices are assigned names.

For example, the DAC960(?) drivers changed their order of detecting controllers, and I did _not_ have fun, given that the machine in question had about 40 disks to deal with, spread across two controllers.

Put LABEL=<label set with e2label> in you fstab in place of the device name.

Christoph

P.S. UUID= work, too - but I prefer a human-readable label...

Signature · 1 line
Of course it doesn't work. We've performed a software upgrade.
#7 lore raw

Re: magic device renumbering was -- Re: Linux 2.4.2ac20

Quoted 5 lines · Christoph Hellwig

Put LABEL=<label set with e2label> in you fstab in place of the device name.

Christoph

P.S. UUID= work, too - but I prefer a human-readable label...

There are a lot of different devices besides disks, e.g. tape drives etc. I seem to remember from the last round this came up that modern FC fabrics have some dynamic properties that may require more intelligence in the kernel.

Peter
--
Peter Svensson ! Pgp key available by finger, fingerprint:
<[email protected]> ! 8A E9 20 98 C1 FF 43 E3 07 FD B9 0A 80 72 70 AF
<[email protected]> !
------------------------------------------------------------------------
Remember, Luke, your source will be with you... always...

#8 lore raw

Re: magic device renumbering was -- Re: Linux 2.4.2ac20

In reply to Christoph Hellwig
Quoted 1 line

Put LABEL=<label set with e2label> in you fstab in place of the device name.

Which is great, for filesystems that support labels. Unfortunately, this isn't universally available -- for instance, you cannot mount a swap partition by label or uuid, so it is not possible to completely isolate yourself from the problems of disk device renumbering.

-- Lars

Signature · 1 line
Lars Kellogg-Stedman <[email protected]>
#9 lore raw

Re: magic device renumbering was -- Re: Linux 2.4.2ac20

Quoted 6 lines · Lars
Quoted 1 line

Put LABEL=<label set with e2label> in you fstab in place of the device name.

Which is great, for filesystems that support labels. Unfortunately, this isn't universally available -- for instance, you cannot mount a swap partition by label or uuid, so it is not possible to completely isolate yourself from the problems of disk device renumbering.

There is room for a LABEL and/or UUID in the swap superblock, if you would want to implement support for this. I took a look once, and it should be possible to add in a compatible way. Of course, you can always put swap into LVM, which also makes it (along with filesystems other than ext2) immune from the nasty device name changes.

Cheers, Andreas

Signature · 3 lines
Andreas Dilger  \ "If a man ate a pound of pasta and a pound of antipasto,
                 \  would they cancel out, leaving him still hungry?"
http://www-mddsp.enel.ucalgary.ca/People/adilger/               -- Dogbert
#10 lore raw

Re: magic device renumbering was -- Re: Linux 2.4.2ac20

In reply to Lars Kellogg-Stedman
Quoted 12 lines · Lars Kellogg-Stedman
Quoted 1 line

Put LABEL=<label set with e2label> in you fstab in place of the device name.

Which is great, for filesystems that support labels. Unfortunately, this isn't universally available -- for instance, you cannot mount a swap partition by label or uuid, so it is not possible to completely isolate yourself from the problems of disk device renumbering.

-- Lars

Signature · 1 line
Lars Kellogg-Stedman <[email protected]>

When my BIOS finds IDE disks, it starts at the lowest address of the port. It then looks for the first master, then the slave(s), etc. Then it tries the second, etc.

When my SCSI BIOS finds disks, it starts at the first controller, the first LUN, the first drive, etc.

This used to even be the way disks were located by the kernel drivers. Now, these are found in some "random" order.

If whatever is causing the "random" order was fixed, put back like it used to be, etc., we wouldn't have these problems.

Cheers,
Dick Johnson

Penguin : Linux version 2.4.1 on an i686 machine (799.53 BogoMips).

"Memory is like gasoline. You use it up when you are running. Of course you get it all back when you reboot..."; Actual explanation obtained from the Micro$oft help desk.

#11 lore raw

Re: magic device renumbering was -- Re: Linux 2.4.2ac20

Quoted 5 lines · Richard B. Johnson

This used to even be the way disks were located by the kernel drivers. Now, these are found in some "random" order.

If whatever is causing the "random" order was fixed, put back like it used to be, etc., we wouldn't have these problems.

Another alternative to path2inst or a database, I suppose, would be to use bus/pci slot information (like in /proc/pci?) to order multiple devices, so at least there's some consistency.

You might have a serious headache, however, when adding a device, under that scheme.

--
-- John E. Jasen ([email protected])
-- In theory, theory and practise are the same. In practise, they aren't.

#12 lore raw

Re: magic device renumbering was -- Re: Linux 2.4.2ac20

In reply to Christoph Hellwig
Quoted 10 lines · Christoph
Quoted 6 lines · In article <[email protected]>…

drivers change their detection schemes; and changes in the kernel can change the order in which devices are assigned names.

For example, the DAC960(?) drivers changed their order of detecting controllers, and I did _not_ have fun, given that the machine in question had about 40 disks to deal with, spread across two controllers.

Put LABEL=<label set with e2label> in you fstab in place of the device name. P.S. UUID= work, too - but I prefer a human-readable label...

Works OK for ext2 only. I'm still waiting on the reiserfs folks to add a UUID and LABEL to their superblock.

However, for raw partitions, you will need to use LVM to get rename-safe device labels. You probably want LVM anyways, if you have 40 disks...

Cheers, Andreas

Signature · 3 lines
Andreas Dilger  \ "If a man ate a pound of pasta and a pound of antipasto,
                 \  would they cancel out, leaving him still hungry?"
http://www-mddsp.enel.ucalgary.ca/People/adilger/               -- Dogbert
#13 lore raw

Re: magic device renumbering was -- Re: Linux 2.4.2ac20

In reply to Christoph Hellwig
Quoted 6 lines · Lars Kellogg-Stedman
Quoted 1 line

Put LABEL=<label set with e2label> in you fstab in place of the device name.

Which is great, for filesystems that support labels. Unfortunately, this isn't universally available -- for instance, you cannot mount a swap partition by label or uuid, so it is not possible to completely isolate yourself from the problems of disk device renumbering.

True. Let's mark for 2.5 ToDO list: magic number for swap...

Just because it does not work universally it doesn't have to be a bad idea...

Christoph

Signature · 1 line
Of course it doesn't work. We've performed a software upgrade.
#14 lore raw

Re: magic device renumbering was -- Re: Linux 2.4.2ac20

In reply to Christoph Hellwig
Quoted 4 lines · Christoph Hellwig
Quoted 1 line · In article <[email protected]>…

The problem:

Quoted 8 lines
Quoted 6 lines

drivers change their detection schemes; and changes in the kernel can change the order in which devices are assigned names.

For example, the DAC960(?) drivers changed their order of detecting controllers, and I did _not_ have fun, given that the machine in question had about 40 disks to deal with, spread across two controllers.

Put LABEL=<label set with e2label> in you fstab in place of the device name.

It solves the example, but not necessarily the problem.

We're still left with partitions that don't do labels, attached tape devices, scsi controllers, NICs, and so forth.

--
-- John E. Jasen ([email protected])
-- In theory, theory and practise are the same. In practise, they aren't.

#15 lore raw

Re: magic device renumbering was -- Re: Linux 2.4.2ac20

In reply to Christoph Hellwig

Hi,

Quoted 6 lines · Lars Kellogg-Stedman
Quoted 1 line

Put LABEL=<label set with e2label> in you fstab in place of the device name.

Which is great, for filesystems that support labels. Unfortunately, this isn't universally available -- for instance, you cannot mount a swap partition by label or uuid, so it is not possible to completely isolate yourself from the problems of disk device renumbering.

It's not convenient, but it is certainly possible: use a single-partition raid0 logical device with raid autostart, and you get a logical /dev/md* device which corresponds to a single partition and which has a fixed name which is detected by the kernel at runtime and mapped to the correct disk, wherever the disk may be.

The IBM EVMS folks are looking to generalise this sort of probing, but for now there is at least one solution to this problem. LVM works too to some extent, but it currently lacks the automatic boot-time/ device-detect-time kernel probing step that the software raid code has.

Cheers,
Stephen

#16 lore raw

Re: magic device renumbering was -- Re: Linux 2.4.2ac20

In reply to Nathan Walp
Quoted 3 lines · John Jasen

The problem:

[ Device name slippage ]

Quoted 12 lines

Possible solutions(?):

Solaris uses an /etc/path_to_inst file, to keep track of device ordering, et al.

Maybe we should consider something similar, where a physical device to logical device map is kept and used to keep things consistent on kernel/driver changes; device addition/removal, and so forth ...

I am, of course, open to better solutions.

This would currently be massive overkill for Linux, but DYNIX/ptx avoids this problem entirely by keeping a device naming database. This became necessary when we added support for multi-path fibre-channel connected disks. Most device-naming conventions rely on "physical" addresses i.e. this disk at the end of this bus connected to this controller in this PCI slot is /dev/sdd. The Solaris scheme mentioned above is no different in that respect. Unfortunately, it doesn't work with multi-path FC-connected devices.

Very briefly, devices that are "id-able" i.e. already have a unique id are simply entered into the database (SCSI drives have a unique id that you can read at autoconf time). For elements that are not "id-able", we establish a derived id by recording their relation to "id-able" elements. At boot time, we scan (in parallel) the system and compare what we find to the database. That way, you get consistent naming for devices, and, at least in the case of the SCSI (or FC) drives, the name doesn't change, even if you pull a drive from one bus and plug it into a different bus entirely.

As I say, this would be massive overkill for Linux, but it's a rather thorough solution :-)

Tim

Signature · 4 lines
Tim Wright - [email protected] or [email protected] or [email protected]
IBM Linux Technology Center, Beaverton, Oregon
Interested in Linux scalability ? Look at http://lse.sourceforge.net/
"Nobody ever said I was charming, they said "Rimmer, you're a git!"" RD VI
#17 lore raw

Re: magic device renumbering was -- Re: Linux 2.4.2ac20

In reply to Nathan Walp
Quoted 8 lines · Tim Wright

This would currently be massive overkill for Linux, but DYNIX/ptx avoids this problem entirely by keeping a device naming database. This became necessary when we added support for multi-path fibre-channel connected disks. Most device-naming conventions rely on "physical" addresses i.e. this disk at the end of this bus connected to this controller in this PCI slot is /dev/sdd. The Solaris scheme mentioned above is no different in that respect. Unfortunately, it doesn't work with multi-path FC-connected devices.

Quoted 8 lines

Very briefly, devices that are "id-able" i.e. already have a unique id are simply entered into the database (SCSI drives have a unique id that you can read at autoconf time). For elements that are not "id-able", we establish a derived id by recording their relation to "id-able" elements. At boot time, we scan (in parallel) the system and compare what we find to the database. That way, you get consistent naming for devices, and, at least in the case of the SCSI (or FC) drives, the name doesn't change, even if you pull a drive from one bus and plug it into a different bus entirely.

This comes up a lot with regards to USB devices too. One of the usb-serial drivers (the edgeport driver) did something like this by looking at the topology of the USB bus and where a specific device was (it was also helped by unique serial numbers) and allowed the devices to be assigned device nodes based on the topology and a small user space program. I'm going to try to do this for all usb-serial devices in 2.5

I can see a scheme like this being very useful for all USB, FireWire, scsi, etc type of devices.

And no, I don't think that having some type of device naming "database" is overkill and will eventually make it into parts of the kernel (the "database" living outside of the kernel of course...)

greg k-h

Signature · 1 line
greg@(kroah|wirex).com
#18 lore raw

Re: magic device renumbering was -- Re: Linux 2.4.2ac20

In reply to Nathan Walp

On Wed, Mar 14, 2001 at 10:15:26AM -0800, Greg KH wrote:
[My ramblings on device naming database deleted]

Quoted 7 lines

This comes up a lot with regards to USB devices too. One of the usb-serial drivers (the edgeport driver) did something like this by looking at the topology of the USB bus and where a specific device was (it was also helped by unique serial numbers) and allowed the devices to be assigned device nodes based on the topology and a small user space program. I'm going to try to do this for all usb-serial devices in 2.5

Quoted 7 lines

I can see a scheme like this being very useful for all USB, FireWire, scsi, etc type of devices.

And no, I don't think that having some type of device naming "database" is overkill and will eventually make it into parts of the kernel (the "database" living outside of the kernel of course...)

Well, if it sounds useful, I can look into putting up the design documentation (yes, shock, horror, there is some :-). It's pretty thorough and covers most of the issues involved, and hence might be a good talking point, even if we chose to implement quite differently.

Tim

Signature · 4 lines
Tim Wright - [email protected] or [email protected] or [email protected]
IBM Linux Technology Center, Beaverton, Oregon
Interested in Linux scalability ? Look at http://lse.sourceforge.net/
"Nobody ever said I was charming, they said "Rimmer, you're a git!"" RD VI
#19 lore raw

Re: magic device renumbering was -- Re: Linux 2.4.2ac20

In reply to Nathan Walp
Quoted 4 lines · Tim Wright

Well, if it sounds useful, I can look into putting up the design documentation (yes, shock, horror, there is some :-). It's pretty thorough and covers most of the issues involved, and hence might be a good talking point, even if we chose to implement quite differently.

I'd be interested in seeing it, and I'm sure other developers would too. If you need a place to host it, I can offer a spot on the linux-hotplug sourceforge site for it.

thanks,

greg k-h

Signature · 1 line
greg@(kroah|wirex).com
#20 lore raw

Re: magic device renumbering was -- Re: Linux 2.4.2ac20

In reply to Nathan Walp

Hi,

The solution is not to go down the path2inst road, that is full of its own traps. You want volume labels via a volume manager (do lvm and raid already do this?) and/or filesystem labels (see e2fslabel). This won't solve all of the ills associated with device instance changes, but it will certainly address the biggest one.

skd

Quoted 38 lines · John Jasen

The problem:

drivers change their detection schemes; and changes in the kernel can change the order in which devices are assigned names.

For example, the DAC960(?) drivers changed their order of detecting controllers, and I did _not_ have fun, given that the machine in question had about 40 disks to deal with, spread across two controllers.

This can create a lot of problems for people upgrading large, production quality systems -- as, in the worst case, the system won't complete the boot cycle; or in middle cases, the user/sysadmin is stuck rewriting X amount of files and trying again; or in small cases, you find out that your SMC and Intel ethernet cards are reversed, and have to go fix things
...

Possible solutions(?):

Solaris uses an /etc/path_to_inst file, to keep track of device ordering, et al.

Maybe we should consider something similar, where a physical device to logical device map is kept and used to keep things consistent on kernel/driver changes; device addition/removal, and so forth ...

I am, of course, open to better solutions.

--
-- John E. Jasen ([email protected])
-- In theory, theory and practise are the same. In practise, they aren't.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/