driver core: multithreaded probing - more parallelism control
Quoted 2 lines · Ying
This is a new version of multithreaded probing patch, with more parallelism control added.
Thanks. (I'd like to try it out but will probably be busy with other stuff during the next few weeks.)
...
Quoted 1 line
A field named "
probe_queue_no" is added to "struct device", which
I'd call it probe_queue_number or maybe probe_queue_id. The term "no" is ambiguous.
Quoted 2 lines
indicates probing queue No. on which the probing of the device will be done. The subsystem can control the parallelism through this field.
Is the queue number kernel-global or per subsystem?
...
Quoted 3 lines
+ *
schedule_probe- schedule a probing to be done later
+ * @probe_queue_no: probing queue No. on which the probing will be done
+ * @probe: probing infromation include probing function and parameter
^^^^^^^^^^^
typo: information
Signature · 3 lines
Stefan Richter -=====-=-=== -==- =-=-- http://arcgraph.de/sr/
Quoted 1 line
Huang, Ying wrote:
...
Quoted 3 lines
Quoted 1 line
+ * @probe: probing infromation include probing function and parameter
^^^^^^^^^^^typo: information
Also, the meaning of the rest of the sentence is unclear.
Signature · 3 lines
Stefan Richter -=====-=-=== -==- =-=-- http://arcgraph.de/sr/
Quoted 2 lines · Stefan Richter
I'd call it
probe_queue_numberor maybeprobe_queue_id. The term "no" is ambiguous.
Yes, I think probe_queue_id is better.
Quoted 1 line
Is the queue number kernel-global or per subsystem?
The queue number is kernel-global. I think this is easy to be implemented. And the serialization demand between subsystem can be satisfied too.
Quoted 3 lines
Quoted 1 line
+ * @probe: probing infromation include probing function and parameter
^^^^^^^^^^^typo: information
Sorry, I will correct it in the next version.
Best Regards,
Huang Ying
Quoted 3 lines · Ying
Quoted 1 line
Is the queue number kernel-global or per subsystem?
The queue number is kernel-global.
Then there is an API required to allocate and deallocate exclusive queue IDs. This feels strange as a mechanism for (de-)serialization, and might introduce some bulk WRT code and data.
Really, I don't believe there is anything else required from subsystems' point of view than
- the possibility to keep plain serial driver matching/probing,
- to allow unrestricted parallelism,
- to mark devices whose child devices shall be matched/probed serially.
Should there be a subsystem which has more special demands on mixture of parallelism and serialization, it can easily use private means to serialize certain parts of driver probes, for example with the familiar mechanism of mutexes.
Or if need be, such a subsystem can implement its own threading model. The FireWire subsystem for example first fetches so-called configuration ROMs from each node on a bus by means of asynchronous split transactions. The ROMs are scanned for device properties and capabilities, and then drivers are matched/probed. The new FireWire subsystem currently uses workqueue jobs to read the ROMs. The old FireWire subsystem uses one kernel thread per bus. Before the new FireWire subsystem was announced, I planned to let the bus kthread spawn node kthreads which (1) fetch and scan ROMs and (2) match and probe drivers for each unit in a node.
If the old FireWire subsystem had a future, I would most certainly not use your mechanism but implement what you described. I am not sure about the new FireWire subsystem; there isn't much practical experience with it yet.
Signature · 3 lines
Stefan Richter -=====-=-=== -==- =-=-= http://arcgraph.de/sr/
Quoted 5 lines
-----Original Message-----
From: Stefan Richter [mailto:[email protected]]Quoted 1 line
The queue number is kernel-global.
Then there is an API required to allocate and deallocate exclusive
queue
Quoted 2 lines
IDs. This feels strange as a mechanism for (de-)serialization, and might introduce some bulk WRT code and data.
I think the queue IDs of different subsystem need not to be exclusive. The subsystem can allocate queue IDs arbitrarily. If one queue ID is shared between several subsystems, corresponding probing will be serialized. This will slow down the probing unnecessarily, but there will be no race condition.
The benefit of the mechanism is that the maximum parallelism of probing can be control.
Quoted 1 line
Really, I don't believe there is anything else required from
subsystems'
Quoted 5 lines
point of view than
- the possibility to keep plain serial driver matching/probing,
- to allow unrestricted parallelism,
- to mark devices whose child devices shall be matched/probed serially.
Maybe.
Quoted 1 line
Should there be a subsystem which has more special demands on mixture
of
Quoted 6 lines
parallelism and serialization, it can easily use private means to serialize certain parts of driver probes, for example with the familiar mechanism of mutexes.
Or if need be, such a subsystem can implement its own threading model. The FireWire subsystem for example first fetches so-called
configuration
Quoted 6 lines
ROMs from each node on a bus by means of asynchronous split transactions. The ROMs are scanned for device properties and capabilities, and then drivers are matched/probed. The new FireWire subsystem currently uses workqueue jobs to read the ROMs. The old FireWire subsystem uses one kernel thread per bus. Before the new FireWire subsystem was announced, I planned to let the bus kthread
spawn
Quoted 2 lines
node kthreads which (1) fetch and scan ROMs and (2) match and probe drivers for each unit in a node.
I know nothing about FireWire. If I say something nonsense, please just ignore it.
Whether is it possible that a new unit is inserted during probing kthread is running. If it is possible, the probing kthread maybe ignore the new inserted unit. So a kthread for this new unit should be created upon inserting. And there must be some synchronization mechanism provided. So, I think something like probing queue may be better than raw kthread for this. And the bus scanning thread may need to exist during system running, using the probing queue the bus scanning thread can be created and destroyed on demanded too.
Thank you very much for your comment.
Best Regards,
Huang Ying
Quoted 5 lines · Ying
I think the queue IDs of different subsystem need not to be exclusive. The subsystem can allocate queue IDs arbitrarily. If one queue ID is shared between several subsystems, corresponding probing will be serialized. This will slow down the probing unnecessarily, but there will be no race condition.
Parallelism between subsystems may be interesting during boot == "coldplug", /if/ the machine has time-consuming devices to probe on /different/ types of buses. Of course some machines do the really time-consuming stuff on only one type of bus. Granted, parallelism betwen subsystems is not very interesting anymore later after boot == "hotplug".
[...]
Quoted 2 lines
Whether is it possible that a new unit is inserted during probing kthread is running.
Nodes and units on nodes may came and go at arbitrary points in time, and I'm sure similar things can be said about the majority of other bus architectures or network architectures. We take this into account.
(The old FireWire stack will re-enter the main loop of the bus scanning thread sometime after a bus reset event signaled that nodes or units may have appeared or disappeared. The new FireWire stack will schedule respective scanning workqueue jobs after such an event.)
Signature · 3 lines
Stefan Richter -=====-=-=== -==- =-=-= http://arcgraph.de/sr/
Quoted 6 lines · Stefan Richter
Parallelism between subsystems may be interesting during boot == "coldplug", /if/ the machine has time-consuming devices to probe on /different/ types of buses. Of course some machines do the really time-consuming stuff on only one type of bus. Granted, parallelism betwen subsystems is not very interesting anymore later after boot == "hotplug".
Yes. So I think there are two possible solution.
1. Creating one set of probing queues for each subsystem (maybe just the subsystems need it), so the probing queue IDs are local to each subsystem.
2. There is only one set of probing queues in whole system. The probing queue IDs are shared between subsystems. The subsystem can select a random starting queue ID (maybe named as start_queue_id), and allocate the queue IDs from that point on (start_queue_id + private_queue_id). So the probability of queue ID sharing will be reduced.
Quoted 4 lines
(The old FireWire stack will re-enter the main loop of the bus scanning thread sometime after a bus reset event signaled that nodes or units may have appeared or disappeared. The new FireWire stack will schedule respective scanning workqueue jobs after such an event.)
I think the workqueue is better than kernel thread here. With kernel thread, the nodes and units may be needed to be scanned again and again for each unit/node if many units/nodes are appeared at almost the same time, while with workqueue, just schedule the jobs needed.
And a workqueue like the probing queue whose thread can be created/destroyed on demand will save more resources than ordinary workqueue. :)
Best Regards,
Huang Ying
Quoted 10 lines · Ying
Hi,
This is a new version of multithreaded probing patch, with more parallelism control added.
There are more control over which devices and drivers will be probed parallelized or serially. For example, in IEEE1394 subsystem, the different "units" in one "node" can be probed serially while the different "nodes" can be probed parallelized.
Quoted 8 lines
The number of threads can be controlled through a kernel command line parameters.
The patch is against 2.6.22-rc5. The "
wait_for_probes" function in the patch comes from the original multithreaded probing patch. If I need do anything because of it, please let me know.Any comment is welcome.
I'm still not convinced that we need to add this kind of complexity to the driver core, instead of just letting the individual driver subsystems do this, if they want to do it.
Especially as no subsystem wants to do this today :)
thanks,
greg k-h
Quoted 5 lines · Greg KH
I'm still not convinced that we need to add this kind of complexity to the driver core, instead of just letting the individual driver subsystems do this, if they want to do it.
Especially as no subsystem wants to do this today :)
Yes, it should first be shown (with subsystem conversions and runtime tests with, say, at least two different bus architectures) that features like this really are appropriately implemented as a driver core abstraction.
I would lend a hand to put this to test, but I won't do this anymore with the old ieee1394 subsystem, and work on the new firewire subsystem will be focused on stabilization and feature completion in the short to mid term.
Signature · 3 lines
Stefan Richter -=====-=-=== -==- ==--- http://arcgraph.de/sr/
RE: [PATCH] driver core: multithreaded probing - more parallelismcontrol
In reply to Greg KHQuoted 4 lines
From: Greg KH [mailto:[email protected]]
I'm still not convinced that we need to add this kind of complexity to the driver core, instead of just letting the individual driver subsystems do this, if they want to do it.
It may appear not necessary that providing more multithreaded device probing in the driver core, but it seems more necessary that providing more parallel control in the driver core to make some device probing more single-threaded.
There does exist multithreaded device probing in current driver core implementation, supposing two devices are hot-plugged at the same time. But, many device drivers are written without this taken into account. I think it may be better to make default device probing process more single-threaded in the driver core. The single-thread workqueue or some customized version of workqueue like that implemented by my patch can be used for this. The parallel control mechanism can be used to implement multithreaded device probing in needed subsystems too.
Best Regards,
Huang, Ying
Re: [PATCH] driver core: multithreaded probing - more parallelismcontrol
Quoted 5 lines · Ying
Quoted 4 lines
From: Greg KH [mailto:[email protected]]
I'm still not convinced that we need to add this kind of complexity to the driver core, instead of just letting the individual driver subsystems do this, if they want to do it.
Quoted 7 lines
It may appear not necessary that providing more multithreaded device probing in the driver core, but it seems more necessary that providing more parallel control in the driver core to make some device probing more single-threaded.
There does exist multithreaded device probing in current driver core implementation, supposing two devices are hot-plugged at the same time.
No, that is a bus-specific thing, and no bus that I know of supports that at this time.
Quoted 1 line
But, many device drivers are written without this taken into account.
That's why no bus does this :)
Quoted 5 lines
I think it may be better to make default device probing process more single-threaded in the driver core. The single-thread workqueue or some customized version of workqueue like that implemented by my patch can be used for this. The parallel control mechanism can be used to implement multithreaded device probing in needed subsystems too.
But remember, the individual busses already do this all in a single thread anyway, nothing is needed in the driver core to do this.
thanks,
greg k-h
Re: [PATCH] driver core: multithreaded probing - more parallelismcontrol
On Mon, 25 Jun 2007 01:16:24 -0700,
Greg KH <[email protected]> wrote:
[I'm a bit late to the party, but...]
Quoted 7 lines · Ying
Quoted 3 lines
There does exist multithreaded device probing in current driver core implementation, supposing two devices are hot-plugged at the same time.
No, that is a bus-specific thing, and no bus that I know of supports that at this time.
The s390 channel subsystem busses should be fine with any parallelism, especially as the css bus kicks off tons of probes (device recognition) at the same time. Any ccw driver must be able to be handle to be called for many devices in parallel as well (like, when someone attaches their shiny new storage subsystem to the LPAR and some thousands of dasds become available).
Quoted 4 lines
Quoted 2 lines
But, many device drivers are written without this taken into account.
That's why no bus does this :)
It is possible for busses for a small set of device drivers (like the s390 busses; maybe there are others). It looks like a bad idea to try this for PCI :)
Quoted 9 lines
Quoted 6 lines
I think it may be better to make default device probing process more single-threaded in the driver core. The single-thread workqueue or some customized version of workqueue like that implemented by my patch can be used for this. The parallel control mechanism can be used to implement multithreaded device probing in needed subsystems too.
But remember, the individual busses already do this all in a single thread anyway, nothing is needed in the driver core to do this.
I think I could make good use of some more parallelism control (for throttling or so). Not sure if it should really sit at the driver core level, but that would avoid reinventing the wheel.
[Goes reading the original patch]
Cornelia
Quoted 10 lines · Ying
Quoted 6 lines · Stefan Richter
Parallelism between subsystems may be interesting during boot == "coldplug", /if/ the machine has time-consuming devices to probe on /different/ types of buses. Of course some machines do the really time-consuming stuff on only one type of bus. Granted, parallelism betwen subsystems is not very interesting anymore later after boot == "hotplug".
Yes. So I think there are two possible solution.
Quoted 8 lines
1. Creating one set of probing queues for each subsystem (maybe just the subsystems need it), so the probing queue IDs are local to each subsystem.
2. There is only one set of probing queues in whole system. The probing queue IDs are shared between subsystems. The subsystem can select a random starting queue ID (maybe named asstart_queue_id), and allocate the queue IDs from that point on (start_queue_id+private_queue_id). So the probability of queue ID sharing will be reduced.
What should also be considered here is that we may want to have different numbers of queues per subsystem (fewer for those where probing is resource-heavy), but may want to restrict the total number of queues as well. Some throttling mechanism may be helpful here (so that a single subsystem cannot hog all queues while another is stuck with a single queue, or double usage of queues). In fact, throttling may be interesting for any subsystem using parallelism, especially if the number of devices may be huge and/or probing resource hungry.
Cornelia