Skip to contentSkip to conversation

PATCH: Multiprobe sanitizer

PATCH Alan Cox 5 participants 17 messages Aug 16, 2006 – Aug 17, 2006 PCI

lore
Share & tools
lore

Alan Cox

author
#0 lore raw

There are numerous drivers that can use multithreaded probing but having some kind of global flag as the way to control this makes migration to threaded probing hard and since it enables it everywhere and is almost as likely to cause serious pain as holding a clog dance in a minefield.

If we have a pci_driver multithread_probe flag to inherit you can turn it on for one driver at a time.

Quoted 1 line

From playing so far however I think we need a different model at the

device layer which serializes until the called probe function says "ok you can start another one now". That would need some kind of flag and semaphore plus a helper function.

Anyway in the absence of that this is a starting point to usefully play with this stuff

drivers/pci/pci-driver.c +5 -1
diff -u --new-file --recursive --exclude-from /usr/src/exclude linux.vanilla-2.6.18-rc4-mm1/drivers/pci/pci-driver.c linux-2.6.18-rc4-mm1/drivers/pci/pci-driver.c
@@ -432,7 +432,11 @@
 	drv->driver.bus = &pci_bus_type;
 	drv->driver.owner = owner;
 	drv->driver.kobj.ktype = &pci_driver_kobj_type;
-	drv->driver.multithread_probe = pci_multithread_probe;
+	
+	if(pci_multithread_probe)
+		drv->driver.multithread_probe = pci_multithread_probe;
+	else
+		drv->driver.multithread_probe = drv->multithread_probe;
 
 	spin_lock_init(&drv->dynids.lock);
 	INIT_LIST_HEAD(&drv->dynids.list);
include/linux/pci.h +2 -0
diff -u --new-file --recursive --exclude-from /usr/src/exclude linux.vanilla-2.6.18-rc4-mm1/include/linux/pci.h linux-2.6.18-rc4-mm1/include/linux/pci.h
@@ -351,6 +351,8 @@
 	struct pci_error_handlers *err_handler;
 	struct device_driver	driver;
 	struct pci_dynids dynids;
+	
+	int multithread_probe;
 };
 
 #define	to_pci_driver(drv) container_of(drv,struct pci_driver, driver)
#1 lore raw
Quoted 7 lines · Alan Cox

There are numerous drivers that can use multithreaded probing but having some kind of global flag as the way to control this makes migration to threaded probing hard and since it enables it everywhere and is almost as likely to cause serious pain as holding a clog dance in a minefield.

If we have a pci_driver multithread_probe flag to inherit you can turn it on for one driver at a time.

I was thinking about this originally, but didn't want to go and modify every PCI driver to enable it :)

But I do like your patch below that lets the options mix nicely.

Quoted 4 lines

From playing so far however I think we need a different model at the device layer which serializes until the called probe function says "ok you can start another one now". That would need some kind of flag and semaphore plus a helper function.

What would this help out with? Would the PCI layer (for example) handle this "notify the core that it can continue" type logic? Or would the individual drivers need to be able to control it?

I'm guessing that you are thinking of this in relation to the disk drivers, have you found cases where something like this is necessary due to hardware constraints?

thanks,

greg k-h

author
#2 lore raw

Ar Mer, 2006-08-16 am 15:26 -0700, ysgrifennodd Greg KH:

Quoted 7 lines

What would this help out with? Would the PCI layer (for example) handle this "notify the core that it can continue" type logic? Or would the individual drivers need to be able to control it?

I'm guessing that you are thinking of this in relation to the disk drivers, have you found cases where something like this is necessary due to hardware constraints?

Actually it occurs everywhere because what happens is

PCI enumerates in bus order
Threads *usually* run in bus order

so every n'th boot your devices re-order themselves out of bus order, and eth1 becomes eth0 for the day.

If you have a "ok now continue scanning" API then we can do

Grab resources
Register driver
Go parallel
[Slow stuff]

I was thinking if we set multithread = 2 (and define some constants) then the core code would do

	if (multithread == WAIT)
		down(&drv->wait);

and we'd have

	pci_driver_continue_enumerating(struct pci_driver *drv) {
		up(&drv->wait);
	}
#3 lore raw
Quoted 9 lines · Alan Cox

Ar Mer, 2006-08-16 am 15:26 -0700, ysgrifennodd Greg KH:

Quoted 7 lines

What would this help out with? Would the PCI layer (for example) handle this "notify the core that it can continue" type logic? Or would the individual drivers need to be able to control it?

I'm guessing that you are thinking of this in relation to the disk drivers, have you found cases where something like this is necessary due to hardware constraints?

Quoted 7 lines

Actually it occurs everywhere because what happens is

PCI enumerates in bus order
Threads *usually* run in bus order

so every n'th boot your devices re-order themselves out of bus order, and eth1 becomes eth0 for the day.

Devices reorder themselves anyways .... look at my XPC shuttle, it has some usb all-sort-of-card reader built-in and every other day, I get that to be sda instead of the internal SATA... solution: use mounting by label. With USB network type of things etc... same problem. I really don't like the idea of having to add special things in PCI drivers to "work around" the problem (which will only work in some cases and will slightly lower how much parallelism we can do).

In fact, I'm all about making the problem worse by agressively paralellilizing everything to get distros config mecanisms to catch up and stop using the interface name (or use ifrename).

Ben.

#4 lore raw
Quoted 4 lines

In fact, I'm all about making the problem worse by agressively paralellilizing everything to get distros config mecanisms to catch up and stop using the interface name (or use ifrename).

distros do this already based on MAC address ;)
at least some do.

Signature · 1 line
if you want to mail me at work (you don't), use arjan (at) linux.intel.com
author
#5 lore raw
In reply to Benjamin Herrenschmidt

Ar Iau, 2006-08-17 am 08:57 +0200, ysgrifennodd Benjamin Herrenschmidt:

Quoted 3 lines

In fact, I'm all about making the problem worse by agressively paralellilizing everything to get distros config mecanisms to catch up and stop using the interface name (or use ifrename).

I'm so glad I don't have to depend on your code then because I have to actually deal with the real world not try to commit design suicide in pursuit in of elegance.

There are numerous cases where bus stability happens to matter because you cannot identify the different devices. The disk and ethernet cases are relatively managable (disk has some distribution issues with fstab etc on older setups). Things get nasty when you look at say sound or video.

A classic example would be a typical security system with four identical PCI video capture cards. There is no way other than PCI bus ordering to order them, and if you don't order them your system isn't useful as you do different processing on different camera streams.

Quoted 1 line

From a performance perspective the only one we really care about is

probably disks. Even there we need some kind of ordering (or ability to order) so that we can handle unlabelled (eg new) volumes and md components.

Right now lvm/dm/md all depend on real disk names to be useful on large systems because the time to scan for labels is too high. On small systems some tools work ok although not all with labels.

Disk has another awkward problem too - power control and management.

#6 lore raw
Quoted 8 lines · Alan Cox

Ar Iau, 2006-08-17 am 08:57 +0200, ysgrifennodd Benjamin Herrenschmidt:

Quoted 3 lines

In fact, I'm all about making the problem worse by agressively paralellilizing everything to get distros config mecanisms to catch up and stop using the interface name (or use ifrename).

I'm so glad I don't have to depend on your code then because I have to actually deal with the real world not try to commit design suicide in pursuit in of elegance.

Heh, ok, I wasn't 100% serious there :) Though I still think it's a bit sad to have to add that sort of "workarounds" all over the place to get some kind of parallelism. At least disk is getting better on most distros (with, iirc, the notable exception of debian (and maybe ubuntu) which still defaults to not using labels).

Quoted 10 lines

There are numerous cases where bus stability happens to matter because you cannot identify the different devices. The disk and ethernet cases are relatively managable (disk has some distribution issues with fstab etc on older setups). Things get nasty when you look at say sound or video.

A classic example would be a typical security system with four identical PCI video capture cards. There is no way other than PCI bus ordering to order them, and if you don't order them your system isn't useful as you do different processing on different camera streams.

There are ways. Using sysfs, you can match a given card to a given PCI "location", in absence of any other way of identifying, it's still better than relying on driver probe ordering imho.

My main point is if we don't find a way to get some incentive to get it fixed, userland will probably not be fixed. Maybe your approach as a "temporary measure" coupled with a kernel command line argument to force full parallelism is the way to go...

Quoted 10 lines

From a performance perspective the only one we really care about is probably disks. Even there we need some kind of ordering (or ability to order) so that we can handle unlabelled (eg new) volumes and md components.

Right now lvm/dm/md all depend on real disk names to be useful on large systems because the time to scan for labels is too high. On small systems some tools work ok although not all with labels.

Disk has another awkward problem too - power control and management.

I've been thinking about this problem in the past and one solution I found was to have a concept of a "ID" string (sort-of an exension of the current syfs busid, though a sysfs path would probably do the trick for most devices) that allows to uniquely identify a device in the system with good stability. Devices that can support some kind of serial number / uuid would be able to override that with it to provide even better stability. The idea is to provide the stablest possible identifier for a device, slot ID being fairly unperfect but probably the best one can do in absence of better.

Probe ordering is fragile and completely defeated with busses that are already probed asynchronously (like USB or firewire), and things can only get worse. Thus we need to look for generic solutions, the trick of maintaining probe ordering will work around problems today but we'll still hit the wall in an increasing number of cases in the future.

Ben.

#7 lore raw
Quoted 5 lines · Benjamin Herrenschmidt

Probe ordering is fragile and completely defeated with busses that are already probed asynchronously (like USB or firewire), and things can only get worse. Thus we need to look for generic solutions, the trick of maintaining probe ordering will work around problems today but we'll still hit the wall in an increasing number of cases in the future.

That's exactly why udev was created :)

It can handle bus ordering issues already today just fine, and distros use it this way in shipping, "enterprise ready" products.

thanks,

greg k-h

#8 lore raw
Quoted 11 lines · Greg KH
Quoted 5 lines · Benjamin Herrenschmidt

Probe ordering is fragile and completely defeated with busses that are already probed asynchronously (like USB or firewire), and things can only get worse. Thus we need to look for generic solutions, the trick of maintaining probe ordering will work around problems today but we'll still hit the wall in an increasing number of cases in the future.

That's exactly why udev was created :)

It can handle bus ordering issues already today just fine, and distros use it this way in shipping, "enterprise ready" products.

Only up to a certain point and for certain drivers... but yeah. That's probably the right direction to take. Now, I'll let you and Alan argue wether it's sufficient or not to move toward a fully parallel probing :)

Ben.

#9 lore raw
Quoted 8 lines · Benjamin Herrenschmidt
Quoted 7 lines · Greg KH
Quoted 5 lines · Benjamin Herrenschmidt

Probe ordering is fragile and completely defeated with busses that are already probed asynchronously (like USB or firewire), and things can only get worse. Thus we need to look for generic solutions, the trick of maintaining probe ordering will work around problems today but we'll still hit the wall in an increasing number of cases in the future.

Quoted 6 lines
Quoted 4 lines

That's exactly why udev was created :)

It can handle bus ordering issues already today just fine, and distros use it this way in shipping, "enterprise ready" products.

Only up to a certain point and for certain drivers... but yeah.

What drivers are not supported by this? Seriously, have we missed any?

thanks,

greg k-h

#10 lore raw
Quoted 9 lines · Greg KH
Quoted 8 lines · Benjamin Herrenschmidt
Quoted 7 lines · Greg KH
Quoted 5 lines · Benjamin Herrenschmidt

Probe ordering is fragile and completely defeated with busses that are already probed asynchronously (like USB or firewire), and things can only get worse. Thus we need to look for generic solutions, the trick of maintaining probe ordering will work around problems today but we'll still hit the wall in an increasing number of cases in the future.

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines

That's exactly why udev was created :)

It can handle bus ordering issues already today just fine, and distros use it this way in shipping, "enterprise ready" products.

Only up to a certain point and for certain drivers... but yeah.

What drivers are not supported by this? Seriously, have we missed any?

udev will not create stable names for a bunch of things... at least not with the default config that comes with distros. On my shuttle with the built-in USB card reader, whatever config comes up with the box will cause the machine to boot or fail to boot due to sda not beeing what it's expected to be, and udev is of no help because it won't create stable device names. Yes, I'm confident it _can_ be configured to do so. It generally is not.

fbdev's also come to mind etc...

It's mostly a distro problem at this point...

Ben.

#11 lore raw
Quoted 17 lines · Benjamin Herrenschmidt
Quoted 16 lines · Greg KH
Quoted 14 lines · Benjamin Herrenschmidt
Quoted 11 lines · Greg KH
Quoted 5 lines · Benjamin Herrenschmidt

Probe ordering is fragile and completely defeated with busses that are already probed asynchronously (like USB or firewire), and things can only get worse. Thus we need to look for generic solutions, the trick of maintaining probe ordering will work around problems today but we'll still hit the wall in an increasing number of cases in the future.

That's exactly why udev was created :)

It can handle bus ordering issues already today just fine, and distros use it this way in shipping, "enterprise ready" products.

Only up to a certain point and for certain drivers... but yeah.

Quoted 8 lines
Quoted 1 line

What drivers are not supported by this? Seriously, have we missed any?

udev will not create stable names for a bunch of things... at least not with the default config that comes with distros. On my shuttle with the built-in USB card reader, whatever config comes up with the box will cause the machine to boot or fail to boot due to sda not beeing what it's expected to be, and udev is of no help because it won't create stable device names.

that's what mount by label is for though..

(which isn't a udev but a distro thing)

#12 lore raw
Quoted 10 lines · Arjan van de Ven
Quoted 6 lines

udev will not create stable names for a bunch of things... at least not with the default config that comes with distros. On my shuttle with the built-in USB card reader, whatever config comes up with the box will cause the machine to boot or fail to boot due to sda not beeing what it's expected to be, and udev is of no help because it won't create stable device names.

that's what mount by label is for though..

(which isn't a udev but a distro thing)

I know and that's what I use, but it wasn't standard on my distro (debian, ok, my fault...) and the problem is still there for pretty much any other class of device in the system... (and mount by label isn't a udev thing.. won't fix anything but mountable devices which is fairly limitative, no help with usb vs. firewire vs. ata cd-writers etc.... udev can do it but isn't configured to do so by default on distros)

Ben.

#13 lore raw
In reply to Arjan van de Ven
Quoted 27 lines · Arjan van de Ven
Quoted 17 lines · Benjamin Herrenschmidt
Quoted 16 lines · Greg KH
Quoted 14 lines · Benjamin Herrenschmidt
Quoted 11 lines · Greg KH
Quoted 5 lines · Benjamin Herrenschmidt

Probe ordering is fragile and completely defeated with busses that are already probed asynchronously (like USB or firewire), and things can only get worse. Thus we need to look for generic solutions, the trick of maintaining probe ordering will work around problems today but we'll still hit the wall in an increasing number of cases in the future.

That's exactly why udev was created :)

It can handle bus ordering issues already today just fine, and distros use it this way in shipping, "enterprise ready" products.

Only up to a certain point and for certain drivers... but yeah.

Quoted 8 lines
Quoted 1 line

What drivers are not supported by this? Seriously, have we missed any?

udev will not create stable names for a bunch of things... at least not with the default config that comes with distros. On my shuttle with the built-in USB card reader, whatever config comes up with the box will cause the machine to boot or fail to boot due to sda not beeing what it's expected to be, and udev is of no help because it won't create stable device names.

Quoted 3 lines

that's what mount by label is for though..

(which isn't a udev but a distro thing)

No, it's a udev thing too, look in /dev/disk/by-label/ and use that in your fstab.

So yes, udev already handles this for block devices and input devices. It is simple to add new rules for other subsystems as people find that they need them.

thanks,

greg k-h

#14 lore raw
In reply to Greg KH
Quoted 9 lines · Greg KH
Quoted 8 lines · Benjamin Herrenschmidt
Quoted 7 lines · Greg KH
Quoted 5 lines · Benjamin Herrenschmidt

Probe ordering is fragile and completely defeated with busses that are already probed asynchronously (like USB or firewire), and things can only get worse. Thus we need to look for generic solutions, the trick of maintaining probe ordering will work around problems today but we'll still hit the wall in an increasing number of cases in the future.

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines

That's exactly why udev was created :)

It can handle bus ordering issues already today just fine, and distros use it this way in shipping, "enterprise ready" products.

Only up to a certain point and for certain drivers... but yeah.

What drivers are not supported by this? Seriously, have we missed any?

Do serial drivers have a device symlink now, and video drivers?

#15 lore raw
Quoted 12 lines · Olaf Hering
Quoted 11 lines · Greg KH
Quoted 10 lines · Benjamin Herrenschmidt
Quoted 9 lines · Greg KH
Quoted 5 lines · Benjamin Herrenschmidt

Probe ordering is fragile and completely defeated with busses that are already probed asynchronously (like USB or firewire), and things can only get worse. Thus we need to look for generic solutions, the trick of maintaining probe ordering will work around problems today but we'll still hit the wall in an increasing number of cases in the future.

That's exactly why udev was created :)

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

It can handle bus ordering issues already today just fine, and distros use it this way in shipping, "enterprise ready" products.

Only up to a certain point and for certain drivers... but yeah.

What drivers are not supported by this? Seriously, have we missed any?

Do serial drivers have a device symlink now

Yes:

 $ tree /sys/class/tty/ttyS0
/sys/class/tty/ttyS0
|-- dev
|-- device -> ../../../devices/platform/serial8250
|-- subsystem -> ../../../class/tty
`-- uevent
Quoted 1 line

and video drivers?

Properly written ones do, I have heard reports that some do not, but that's a driver bug that should be fixed with a one line addition. If you know of any specific ones, please let me know and I'll make the needed change.

thanks,

greg k-h

#16 lore raw
In reply to Alan Cox
Quoted 9 lines · Alan Cox

Ar Mer, 2006-08-16 am 15:26 -0700, ysgrifennodd Greg KH:

Quoted 7 lines

What would this help out with? Would the PCI layer (for example) handle this "notify the core that it can continue" type logic? Or would the individual drivers need to be able to control it?

I'm guessing that you are thinking of this in relation to the disk drivers, have you found cases where something like this is necessary due to hardware constraints?

Quoted 7 lines

Actually it occurs everywhere because what happens is

PCI enumerates in bus order
Threads *usually* run in bus order

so every n'th boot your devices re-order themselves out of bus order, and eth1 becomes eth0 for the day.

As per the help information for this option, this will happen. Happens all the time on my machines already, that's why I use a tool that always names the devices in the proper way (like almost all distros do these days.)

It's a trade off, if you want a parallel device probe, you have to stop relying on the kernel name for devices and use something that is not going to change. Like PCI device ids, (well, ok, they change all the time on pci hotplug boxes and bios upgrades) or MAC addresses, or serial numbers, or file system labels.

Quoted 6 lines

If you have a "ok now continue scanning" API then we can do

Grab resources
Register driver
Go parallel
[Slow stuff]

It seems that for some types of devices, the "Grab resources" portion takes the longest (SCSI...). And the "register driver" stuff already happens later in the process, in an async manner today. So that's not going to work :(

Quoted 7 lines

I was thinking if we set multithread = 2 (and define some constants) then the core code would do

	if (multithread == WAIT)
		down(&drv->wait);
Quoted 5 lines

and we'd have

	pci_driver_continue_enumerating(struct pci_driver *drv) {
		up(&drv->wait);
	}

Yeah, I played with limiting the number of outstanding threads in a manner like this, but couldn't come up with a real reason to limit it.

If you can think of a place in the PCI core that we could block on in this manner, please let me know.

thanks,

greg k-h