How to make a bus with heterogeneous devices?
Hello Greg.
Federico Vaga and me are working on the ZIO framework (ohwr.org/projects/zio) and he's currently adding the bus abstraction, so we can support several cards of the same type. Under ZIO, different devices may have different attributes: an SPI ADC chip may allow to choose from two voltage references while a PCI digital-output device may have a device-wide clocking rate.
But the list of attributes associated to a device is only known when the driver is matched. In other workds, the "device" part is only claiming "here I am and I'm called <sth>", then the probe method of the <sth> driver will instantiate the device with its attributes.
However, the device appears in sysfs whether or not there is a driver for it (which we agree is correct), but the list of attributes is only known when the driver is matched -- because the list is known to the driver, not to the device.
Here are some paths we evaluated:
1- /sys/bus/zio/devices may include place-holders, while a successful match and probe will create real stuff is in /sys/zio/devices/<sth>-0/ But this would mean kobjects and you mandated us to only use devices instead.
2- Like above, but falling in /sys/devices/virtual/zio/ ?
3- Attributes may be added after match succeeds, before calling probe. This cam be done with a "raw" call to sysfs_create_group(); then groups cna be added to dev->groups so they will be removed on device_unregister().
4- Attributes may be added to the device before drv->probe is called (or after it succeeds), using FIXMEFEDE. But .... FIXMEFEDE
5- The real attributes may live in another device, which is a child of the one used within the bus abstraction. Here the drawback is using an additional device and directory level for no reason.
6- We may use a fake (almost empty) device for the match function, and then register the real one, with full attributes, when the match succeeds (so match is called again and must be worked around). This would work but is clearly a horrible hack.
We looked at other bus abstractions, and all of them host homogeneous devices (they may have sub-devices, but none of them has different attributes). Even the platform bus has no per-device attribute list instantiated by the driver. In the cases where the attribute list is different for each device, the list is known to the device, before the match function is called.
Any suggestion is welcome. Meanwhile, Federico started implementing 3 above, but it still looks a dirty approach, and possibly not future-proof.
thanks
/alessandro (and federico)
Quoted 9 lines · Alessandro Rubini
Hello Greg.
Federico Vaga and me are working on the ZIO framework (ohwr.org/projects/zio) and he's currently adding the bus abstraction, so we can support several cards of the same type. Under ZIO, different devices may have different attributes: an SPI ADC chip may allow to choose from two voltage references while a PCI digital-output device may have a device-wide clocking rate.
Quoted 4 lines
But the list of attributes associated to a device is only known when the driver is matched. In other workds, the "device" part is only claiming "here I am and I'm called <sth>", then the probe method of the <sth> driver will instantiate the device with its attributes.
That's fine, it is what happens with lots of drivers, you can do this with the default attribute group assigned to that driver, right?
Quoted 22 lines
However, the device appears in sysfs whether or not there is a driver for it (which we agree is correct), but the list of attributes is only known when the driver is matched -- because the list is known to the driver, not to the device.
Here are some paths we evaluated:
1-
/sys/bus/zio/devicesmay include place-holders, while a successful match and probe will create real stuff is in /sys/zio/devices/<sth>-0/ But this would mean kobjects and you mandated us to only use devices instead.2- Like above, but falling in
/sys/devices/virtual/zio/?3- Attributes may be added after match succeeds, before calling probe. This cam be done with a "raw" call to
sysfs_create_group(); then groups cna be added todev->groupsso they will be removed ondevice_unregister().4- Attributes may be added to the device before
drv->probeis called (or after it succeeds), using FIXMEFEDE. But .... FIXMEFEDE
Quoted 8 lines
5- The real attributes may live in another device, which is a child of the one used within the bus abstraction. Here the drawback is using an additional device and directory level for no reason.
6- We may use a fake (almost empty) device for the match function, and then register the real one, with full attributes, when the match succeeds (so match is called again and must be worked around). This would work but is clearly a horrible hack.
Ick, don't do that...
Quoted 6 lines
We looked at other bus abstractions, and all of them host homogeneous devices (they may have sub-devices, but none of them has different attributes). Even the platform bus has no per-device attribute list instantiated by the driver. In the cases where the attribute list is different for each device, the list is known to the device, before the match function is called.
I think you need to look at how the system bus is handled. It just went into 3.3-rc and allows for "different" types of devices to all be on the same bus, with different drivers and attributes. Let me know if how that works does not work out for you.
thanks,
greg k-h
Quoted 4 lines
I think you need to look at how the system bus is handled. It just went into 3.3-rc and allows for "different" types of devices to all be on the same bus, with different drivers and attributes. Let me know if how that works does not work out for you.
Can I have a reference to the patch(s) which introduces this? I taked a look (3.3-rc5) but I didn't find it
Thank you
Signature · 1 line
Federico Vaga
Quoted 7 lines · Federico Vaga
Quoted 4 lines
I think you need to look at how the system bus is handled. It just went into 3.3-rc and allows for "different" types of devices to all be on the same bus, with different drivers and attributes. Let me know if how that works does not work out for you.
Can I have a reference to the patch(s) which introduces this? I taked a look (3.3-rc5) but I didn't find it
Look at ca22e56debc57b47c422b749c93217ba62644be2 and the patches after it.
Hope this helps,
greg k-h
Quoted 2 lines
That's fine, it is what happens with lots of drivers, you can do this with the default attribute group assigned to that driver, right?
We can't because a driver can handle devices with different set of attributes. We can't use device->groups because only if match succeed we know which attributes add to the device.
That's because the same driver handles several similar devices, to avoid proliferation of almost-identical drivers. So the drivers attaches attributes to the device after match succeeds.
The solution that I temporarily implemented is the following:
Quoted 7 lines
Quoted 7 lines
[...]
3- Attributes may be added after match succeeds, before calling probe.
This cam be done with a "raw" call tosysfs_create_group(); then groups cna be added todev->groupsso they will be removed ondevice_unregister().
[...]
(with calling probe I mean calling "my" driver probe)
I know it's not a nice solution and I know it's not nice add groups to device->groups after registration. I can use sysfs_create_group() and sysfs_remove_group() without touching the device.
Quoted 4 lines
I think you need to look at how the system bus is handled. It just went into 3.3-rc and allows for "different" types of devices to all be on the same bus, with different drivers and attributes. Let me know if how that works does not work out for you.
(thank you for the reference to the patches)
I think it doesn't work in our case for the same reason we can't use device_driver->groups: we know too late in the register process which attributes add to a device, practically at the end of the registration. Maybe I'm missing something.
Signature · 1 line
Federico Vaga
Quoted 9 lines · Federico Vaga
Quoted 2 lines
That's fine, it is what happens with lots of drivers, you can do this with the default attribute group assigned to that driver, right?
We can't because a driver can handle devices with different set of attributes. We can't use
device->groupsbecause only if match succeed we know which attributes add to the device.
That's because the same driver handles several similar devices, to avoid proliferation of almost-identical drivers. So the drivers attaches attributes to the device after match succeeds.
Unless you manage the uevents by "hand", you can run into big problems with userspace tools if you add the attributes after the device is matched to a driver. Be very careful here.
Quoted 10 lines
The solution that I temporarily implemented is the following:
Quoted 7 lines
Quoted 7 lines
[...]
3- Attributes may be added after match succeeds, before calling probe.
This cam be done with a "raw" call tosysfs_create_group(); then groups cna be added todev->groupsso they will be removed ondevice_unregister().
[...]
Quoted 5 lines
(with calling probe I mean calling "my" driver probe)
I know it's not a nice solution and I know it's not nice add groups to
device->groupsafter registration. I can usesysfs_create_group()andsysfs_remove_group()without touching the device.
It's not an issue of "touching" the device, it's an issue of "userspace just got missed all of those attributes", so your tools will be all confused. The kernel has no problem with it :)
Quoted 11 lines
Quoted 4 lines
I think you need to look at how the system bus is handled. It just went into 3.3-rc and allows for "different" types of devices to all be on the same bus, with different drivers and attributes. Let me know if how that works does not work out for you.
(thank you for the reference to the patches)
I think it doesn't work in our case for the same reason we can't use
device_driver->groups: we know too late in the register process which attributes add to a device, practically at the end of the registration. Maybe I'm missing something.
I think you are.
Kay, you know this area of the driver core better than I do at the moment, given your most recent changes here. Any reason why Alessandro and Federico can't do the same thing that the sysdev rework did for their devices here?
thanks,
greg k-h
Quoted 10 lines · Greg Kroah-Hartman
Quoted 1 line · Federico Vaga
Maybe I'm missing something.
I think you are.
Kay, you know this area of the driver core better than I do at the moment, given your most recent changes here. Any reason why Alessandro and Federico can't do the same thing that the sysdev rework did for their devices here?
Maybe I miss something but it sounds not that different from USB.
You have a bus that enumerates devices and a "bus driver" creates a generic device on that bus for everything it finds. These devices stay generic, do not have type-specific attributes, but export matches again to bind other device-specific drivers, which create _new_ child devices below the generic ones with the proper attributes matching their individual type of devices. So every "logical" device is always represented by one generic device and one type-specific child device.
All these devices, the generic ones and the type-specific ones can be on the same bus (struct bus_type) when they get a "struct device_type" assigned which carries the device-type specific set of attributes.
It's like USB, where we get a plain "usb_device" for every device connected, then we look at each device to find out how many "usb_interface" childs we need to create, and every interface device offers again matches for more drivers again, like tty, storage, video, ... to bind to as child devices.
Wouldn't that work?
Kay
Quoted 9 lines
Maybe I miss something but it sounds not that different from USB.
You have a bus that enumerates devices and a "bus driver" creates a generic device on that bus for everything it finds. These devices stay generic, do not have type-specific attributes, but export matches again to bind other device-specific drivers, which create _new_ child devices below the generic ones with the proper attributes matching their individual type of devices. So every "logical" device is always represented by one generic device and one type-specific child device.
It sounds like our option 5:
Quoted 3 lines
5- The real attributes may live in another device, which is a child of the one used within the bus abstraction. Here the drawback is using an additional device and directory level for no reason.
- I register the generic (empty) device on the bus
- when the device match with a driver
- I register a new device child of the matched one but without connect it directly on the bus.
The result will be something like (zio is our bus):
/sys/bus/zio/devices/<generic-dev1>/<real-device1>/<attributes> /sys/bus/zio/devices/<generic-dev2>/<real-device2>/<attributes> /sys/bus/zio/devices/<generic-devN>/<real-deviceN>/<attributes>
When there is not a driver:
/sys/bus/zio/devices/<generic-dev1>/<device-stuff> /sys/bus/zio/devices/<generic-devN>/<device-stuff>
Where the generic-dev is an empty device used to make the bus work. Is it correct?
Quoted 3 lines
All these devices, the generic ones and the type-specific ones can be on the same bus (struct
bus_type) when they get a "structdevice_type" assigned which carries the device-type specific set of attributes.
But if all devices can live on the bus (set device->bus for all devices), when I register the "real-device" (type-specific) the bus try to match it with a driver (again); but we already matched the device with its driver. it is better if the type-specific devices are not directly on the bus but only a child of a device on the bus.
Signature · 1 line
Federico Vaga
Quoted 10 lines · Federico Vaga
Quoted 9 lines
Maybe I miss something but it sounds not that different from USB.
You have a bus that enumerates devices and a "bus driver" creates a generic device on that bus for everything it finds. These devices stay generic, do not have type-specific attributes, but export matches again to bind other device-specific drivers, which create _new_ child devices below the generic ones with the proper attributes matching their individual type of devices. So every "logical" device is always represented by one generic device and one type-specific child device.
Quoted 5 lines
Quoted 3 lines
5- The real attributes may live in another device, which is a child of the one used within the bus abstraction. Here the drawback is using an additional device and directory level for no reason.
- I register the generic (empty) device on the bus
You can always add some generic, no-type specific attributes there, if that fits in your model.
Quoted 3 lines
- when the device match with a driver
- I register a new device child of the matched one but without connect it directly on the bus.
You can assign all these devices to the same bus (see details below).
Quoted 6 lines
The result will be something like (zio is our bus):
/sys/bus/zio/devices/<generic-dev1>/<real-device1>/<attributes> /sys/bus/zio/devices/<generic-dev2>/<real-device2>/<attributes> /sys/bus/zio/devices/<generic-devN>/<real-deviceN>/<attributes>
Quoted 7 lines
When there is not a driver:
/sys/bus/zio/devices/<generic-dev1>/<device-stuff> /sys/bus/zio/devices/<generic-devN>/<device-stuff>
Where the generic-dev is an empty device used to make the bus work. Is it correct?
All buses have have a 'subsystem directory' at:
/sys/bus/
This location exposes the 'device tree classification', like the 'collection of devices by subsystem'
The bus "zio" has its directory at:
/sys/bus/zio/
All devices known to the bus "zio" are referenced at:
/sys/bus/zio/devices/*
Which is a bunch of symlinks pointing into the:
/sys/devices/
directory, which is the tree of all devices from all subsystems. This tree exposes the 'device tree hierarchy'.
So your registered devices would all show up in:
/sys/bus/zio/devices/generic0 --> ../../../devices/.../generic0 /sys/bus/zio/devices/generic1 --> ../../../devices/.../generic1
/sys/bus/zio/devices/generic2 --> ../../../devices/.../generic2
/sys/bus/zio/devices/type-a0 --> ../../../devices/.../generic0/type-a0 /sys/bus/zio/devices/type-a1 --> ../../../devices/.../generic1/type-a1 /sys/bus/zio/devices/type-b0 --> ../../../devices/.../generic2/type-b0
They all show up on the same bus, but the type-specific devices are child devices of the generic ones.
Quoted 9 lines
Quoted 3 lines
All these devices, the generic ones and the type-specific ones can be on the same bus (struct
bus_type) when they get a "structdevice_type" assigned which carries the device-type specific set of attributes.But if all devices can live on the bus (set
device->busfor all devices), when I register the "real-device" (type-specific) the bus try to match it with a driver (again); but we already matched the device with its driver. it is better if the type-specific devices are not directly on the bus but only a child of a device on the bus.
You can sort them out by 'struct device_type' in your probe functions, that should not be a problem.
We generally do not like devices which do not belong to a bus or class, they are kind of 'invisible' to userspace, and udev can not use them directly, it sees them only as a parent device of a device with a bus or class.
Kay
We moved to this solution. Thank you
Signature · 1 line
Federico Vaga