Multi-threaded device probing

This adds the infrastructure for drivers to do a threaded probe.

drivers/base/dd.c +4 -4

-			goto ProbeFailed;
+			goto probe_failed;
 		}
 	} else if (drv->probe) {
 		ret = drv->probe(dev);
 		if (ret) {
 			dev->driver = NULL;
-			goto ProbeFailed;
+			goto probe_failed;
 		}
 	}
 	device_bind_driver(dev);
 	ret = 1;
 	pr_debug("%s: Bound Device %s to Driver %s\n",
 		 drv->bus->name, dev->bus_id, drv->name);
-	goto Done;
+	goto done;
 
- ProbeFailed:
+probe_failed:
 	if (ret == -ENODEV || ret == -ENXIO) {
 		/* Driver matched, but didn't support device
 		 * or device not found.

Quoted 20 lines · Greg KH

This adds the infrastructure for drivers to do a threaded probe.

drivers/base/dd.c +3 -3

-			goto ProbeFailed;
+			goto probe_failed;
 		}
 	} else if (drv->probe) {
 		ret = drv->probe(dev);
 		if (ret) {
 			dev->driver = NULL;
-			goto ProbeFailed;
+			goto probe_failed;
 		}
 	}
 	device_bind_driver(dev);
 	ret = 1;
 	pr_debug("%s: Bound Device %s to Driver %s\n",
 		 drv->bus->name, dev->bus_id, drv->name);
-	goto Done;
+	goto done;

drivers/base/dd.c int driver_probe_device(struct device_dr · line 102

Quoted 11 lines

- ProbeFailed:

drivers/base/dd.c +2 -1

+probe_failed:
 	if (ret == -ENODEV || ret == -ENXIO) {
 		/* Driver matched, but didn't support device
 		 * or device not found.

@@ -110,7 +99,53 @@ int driver_probe_device(struct device_dr
 		       "%s: probe of %s failed with error %d\n",
 		       drv->name, dev->bus_id, ret);
 	}
- Done:
+done:

Removing these changes will make this patch smaller and do one thing. ;-)

During the kernel summit, I was reminded by the wish by some people to do device probing in parallel, so I created the following patch. It offers up the ability for the driver core to create a new thread for every driver<->device probe call. To enable this, the driver needs to have the multithread_probe flag set to 1, otherwise the "traditional" sequencial probe happens.

Note that this patch does not actually enable the threaded probe for any busses, as that's very dangerous at this point in time, without the different bus authors trying it out and verifying that it does work properly.

I did enable this for both USB and PCI and shaved .4 seconds off of the boot time of my tiny little single processor laptop. The savings of my 4-way workstation is much greater, but things start to happen so fast we miss the root disk, as init starts before the disks are finished being initialized. I have some hacks to work around this right now, but I'll hold off on posting them before I make sure they work properly (breaking booting of people's machines isn't the best way to get them to accept new features...)

Anyway, have fun playing around with this if you want, I'll be adding this to the next -mm, but you will have to enable the bit on your own if you want to see any speedups.

During the kernel summit, I was reminded by the wish by some people to do device probing in parallel, so I created the following patch. It offers up the ability for the driver core to create a new thread for every driver<->device probe call. To enable this, the driver needs to have the multithread_probe flag set to 1, otherwise the "traditional" sequencial probe happens.

Quoted 7 lines · Greg KH

During the kernel summit, I was reminded by the wish by some people to do device probing in parallel, so I created the following patch. It offers up the ability for the driver core to create a new thread for every driver<->device probe call. To enable this, the driver needs to have the multithread_probe flag set to 1, otherwise the "traditional" sequencial probe happens.

Another option would be to have probing still serialized within a bus but serviced by a separate thread. The thread can die after let's say 1 minute inactivity timeout and respawned if needed.

During the kernel summit, I was reminded by the wish by some people to do device probing in parallel, so I created the following patch. It offers up the ability for the driver core to create a new thread for every driver<->device probe call. To enable this, the driver needs to have the multithread_probe flag set to 1, otherwise the "traditional" sequencial probe happens.

Note that this patch does not actually enable the threaded probe for any busses, as that's very dangerous at this point in time, without the different bus authors trying it out and verifying that it does work properly.

I did enable this for both USB and PCI and shaved .4 seconds off of the boot time of my tiny little single processor laptop. The savings of my 4-way workstation is much greater, but things start to happen so fast we miss the root disk, as init starts before the disks are finished being initialized. I have some hacks to work around this right now, but I'll hold off on posting them before I make sure they work properly (breaking booting of people's machines isn't the best way to get them to accept new features...)

Anyway, have fun playing around with this if you want, I'll be adding this to the next -mm, but you will have to enable the bit on your own if you want to see any speedups.

thanks,

greg k-h

What happens about the logging?
Surely one would want the output from one probe to be output into the log as a block, and not mix the output from multiple simultaneous probes.

Quoted 3 lines · James Courtier-Dutton

What happens about the logging?
Surely one would want the output from one probe to be output into the log as a block, and not mix the output from multiple simultaneous probes.

Use single-line printks were possible, or mutex-protected multiline blocks where you really can't do without multiple lines of printks that really cannot be separated. (Don't perform time consuming functions within those mutexes; that would defeat the multithreaded probing...)

To adjust printks is only the beginning of what is to be done to adapt single-threaded bus probes to multithreaded ones. There may be hidden assumptions that rely on single-threaded execution.

Quoted 6 lines · Greg KH

During the kernel summit, I was reminded by the wish by some people to do device probing in parallel, so I created the following patch. It offers up the ability for the driver core to create a new thread for every driver<->device probe call. To enable this, the driver needs to have the multithread_probe flag set to 1, otherwise the "traditional" sequencial probe happens.

What happens about the logging?

Surely one would want the output from one probe to be output into the log as a block, and not mix the output from multiple simultaneous probes.

drivers/base/dd.c int driver_probe_device(struct device_dr · line 140

Quoted 5 lines

   +       data = kmalloc(sizeof(*data), GFP_KERNEL);
   +       data->drv = drv;
   +       data->dev = dev;

+

drivers/base/dd.c +1 -0

+	if (drv->multithread_probe) {

            ^^^^^^^^^^^^^^^^^^^^^^
	 if (drv->multithread_probe && !cmdline_mtprobe) {

Also I think providing cmdline option to override the default multithread probe behaviour would be good. Something like above which is useful while debugging the boot issues.

During the kernel summit, I was reminded by the wish by some people to do device probing in parallel, so I created the following patch. It offers up the ability for the driver core to create a new thread for every driver<->device probe call.

Quoted 4 lines · Greg KH

During the kernel summit, I was reminded by the wish by some people to do device probing in parallel, so I created the following patch. It offers up the ability for the driver core to create a new thread for every driver<->device probe call.

...

Just FYI:

1.) SCSI:
There is a patch circulating at linux-scsi which adds parallelized bus scanning to the SCSI subsystem. I believe this cannot be built upon parallelization by driver core. But I am not too familiar with the subsystem facilities which this patch expands on. The patch is from Matthew Wilcox, titled "Asynchronous target discovery".
http://marc.theaimsgroup.com/?t=115349750400001

2.) IEEE 1394:
There was brief preliminary discussion of parallelized probing for the ieee1394 subsystem at linux1394-devel. Using driver core's parallelization would achieve about 1/3rd of what would be desirable.

Background: After each bus reset, the 1394 core (nodemgr) has to download part or all of the configuration ROM of attached devices to determine their identity and capabilities. After that, either a protocol driver probe (generic device hook), a protocol driver remove or suspend routine (generic device hook), or a protocol driver update routine (extra 1394 subsystem hook) is executed; depending on whether nodes were added, removed, or in-use nodes were rediscovered. --- I.e. we better have these subthreads provided by ieee1394/nodemgr itself.

I don't know enough about SCSI to say if this driver core patch will help them out or not. At first glance it does, but the device order gets all messed up from what users are traditionally used to, so perhaps the scsi core will just have to stick with their own changes.

Quoted 4 lines · Greg KH

I don't know enough about SCSI to say if this driver core patch will help them out or not. At first glance it does, but the device order gets all messed up from what users are traditionally used to, so perhaps the scsi core will just have to stick with their own changes.

Right. Networking is in the same boat ... unless they're using udev or some other tool which renames network interfaces. I'm not entirely comfortable with the kernel forcing you to use some other tool in order to maintain stable device names on a static setup.

Perhaps we need either a CONFIG option or a boot option to decide whether to do parallel pci probes.

I still think we need a method of renaming block devices, but haven't looked into it in enough detail yet.

However, almost all distros now use persistant names for network devices due to the PCI Hotplug issue, so it isn't probably as bad as you might think.

Quoted 2 lines

I still think we need a method of renaming block devices, but haven't looked into it in enough detail yet.

That could get "interesting"...

But now that we all are using /dev/disk/ and it has persistant device names for block devices, I really don't think it's that big of a deal.

Quoted 7 lines · Greg KH

Quoted 2 lines

I still think we need a method of renaming block devices, but haven't looked into it in enough detail yet.

That could get "interesting"...

But now that we all are using /dev/disk/ and it has persistant device names for block devices, I really don't think it's that big of a deal.

Actually, that's exactly why it's a big deal. The kernel spits out messages like:

                printk(KERN_DEBUG "%s: Mode Sense: %02x %02x %02x %02x\n",
                       diskname, buffer[0], buffer[1], buffer[2], buffer[3]);

where diskname is something like sda. Now the user has to figure out what sda means in terms of /dev/disk/ and in terms of scsi h:c:t:l and in terms of which sticky label is on which drive. If we let userspace change the gendev's disk_name, that printk can be meaningful to the user in at least one of those senses.

include/linux/device.h +2 -0

@@ -105,6 +105,8 @@ struct device_driver {
         void    (*shutdown)     (struct device * dev);
         int     (*suspend)      (struct device * dev, pm_message_t state);
         int     (*resume)       (struct device * dev);
+
+       unsigned int multithread_probe:1;
  };

Quoted 10 lines · Greg KH

include/linux/device.h +2 -0

@@ -105,6 +105,8 @@ struct device_driver {
 ????????void????(*shutdown)?????(struct device * dev);
 ????????int?????(*suspend)??????(struct device * dev, pm_message_t state);
 ????????int?????(*resume)???????(struct device * dev);
+
+???????unsigned int multithread_probe:1;
 ?};

?

Why use a bit field here? It ends up consuming sizeof(long) anyway and causes more complex code, with no obvious benefit.