USB mini-summit at LinuxCon Vancouver
I'm pleased to announce a USB mini-summit at LinuxCon Vancouver.
What: USB mini-summit
When: Tuesday, August 16th, all day
Where: At the conference venue, room TBD pending confirmation from
Angela Brown.
Proposed topics include USB virtualization, and improved bandwidth APIs between the USB core and drivers (especially webcam drivers). See the detailed topic list below. Anyone is also welcome to propose or show up with a USB related topic. MUSB? USB 3.0 gadget drivers? USB-IP?
The USB mini-summit does overlap with the virtualization mini-summit by a day, but I'm hoping we can schedule talks so some of the virtualization folks can make it to the USB mini-summit. The other option was on Friday during the conference which was not ideal.
Proposed topics:
Topic 1
-------
The KVM folks suggested that it would be good to get USB and virtualization developers together to talk about how to virtualize the xHCI host controller. The xHCI spec architect worked closely with VMWare to get some extra goodies in the spec to help virtualization, and I'd like to see the other virtualization developers take advantage of that. I'd also like us to hash out any issues they have been finding in the USB core or xHCI driver during the virtualization effort.
Topic 2
-------
I'd also like to get the V4L and audio developers who work with USB devices together with the core USB folks to talk about bandwidth management under xHCI.
One of the issues is that since the xHCI hardware does bandwidth management, not the xHCI driver, a schedule that will take too much bandwidth will get rejected much sooner than any USB driver currently expects (during a call to usb_set_interface). This poses issues, since most USB video drivers negotiate the video size and frame rate after they call usb_set_interface, so they don't know whether they can fall back to a less bandwidth-intensive setting. Currently, they just submit URBs with less and less bandwidth until one interval setting gets accepted that won't work under xHCI.
A second issue is that that some drivers need less bandwidth than the device advertises, and the xHCI driver currently uses whatever periodic interval the device advertises in its descriptors. This is not what the video/audio driver wants, especially in the case of buggy high speed devices that advertise the interval in frames, not microframes. There needs to be some way for the drivers to communicate their bandwidth needs to the USB core. We've known about this issue for a while, and I think it's time to get everyone in the same room and hash out an API.
(I will send out an API proposal later this month.)
Hope to see you there!
Sarah Sharp
Quoted 10 lines · Sarah Sharp
Topic 1
-------The KVM folks suggested that it would be good to get USB and virtualization developers together to talk about how to virtualize the xHCI host controller. The xHCI spec architect worked closely with VMWare to get some extra goodies in the spec to help virtualization, and I'd like to see the other virtualization developers take advantage of that. I'd also like us to hash out any issues they have been finding in the USB core or xHCI driver during the virtualization effort.
Do people really want to virtualize the whole xHCI controller, or just specific ports or devices to the guest operating system?
If just specific ports, would something like usbip be better for virtual machines, with the USB traffic going over the network connection between the guest/host?
Quoted 1 line
Hope to see you there!
Thanks for putting this together.
Do we need to sign up somewhere to give the organizers a sense of how many people will be attending?
thanks,
greg k-h
Hi,
Quoted 10 lines
Quoted 7 lines
The KVM folks suggested that it would be good to get USB and virtualization developers together to talk about how to virtualize the xHCI host controller. The xHCI spec architect worked closely with VMWare to get some extra goodies in the spec to help virtualization, and I'd like to see the other virtualization developers take advantage of that. I'd also like us to hash out any issues they have been finding in the USB core or xHCI driver during the virtualization effort.
Do people really want to virtualize the whole xHCI controller, or just specific ports or devices to the guest operating system?
SR/IOV support is an optional xHCI feature. As I understand it you can create a VF which looks like a real xHCI controller. This is partly done in hardware and partly by software. Then you can assign it some ressources (specific ports) and pass it to the guest.
Quoted 3 lines
If just specific ports, would something like usbip be better for virtual machines, with the USB traffic going over the network connection between the guest/host?
There are several ways depending on the use case. Usually the guest sees a (fully software emulated) host adapter with usb devices connected, where the usb devices can be (a) emulated too or (b) real usb devices passed through to the guest. The later is done by passing the guests requests to the real device via usbfs.
One problem with emulating usb fully in software is the polling design of the hardware which makes the emulation quite cpu intensive. Using a xHCI VF should help here alot, but works for the pass through use case only of course.
cheers,
Gerd
Quoted 12 lines · Greg KH
Quoted 10 lines · Sarah Sharp
Topic 1
-------The KVM folks suggested that it would be good to get USB and virtualization developers together to talk about how to virtualize the xHCI host controller. The xHCI spec architect worked closely with VMWare to get some extra goodies in the spec to help virtualization, and I'd like to see the other virtualization developers take advantage of that. I'd also like us to hash out any issues they have been finding in the USB core or xHCI driver during the virtualization effort.
Quoted 2 lines
Do people really want to virtualize the whole xHCI controller, or just specific ports or devices to the guest operating system?
A host OS could chose to virtualize the whole xHCI controller if it wanted to. That's part of the reason why xHCI does all the bandwidth checking in hardware, not in software.
Quoted 3 lines
If just specific ports, would something like usbip be better for virtual machines, with the USB traffic going over the network connection between the guest/host?
It could be done that way too. But that doesn't help if you're trying to run Windows under Linux, right? Only if all the guest OSes use the same USB IP protocol then it would work.
Quoted 6 lines
Quoted 1 line
Hope to see you there!
Thanks for putting this together.
Do we need to sign up somewhere to give the organizers a sense of how many people will be attending?
I private ack by email would be great. Or you can ack by facebook:
https://www.facebook.com/event.php?eid=229532200405657 I could add the event in upcoming.yahoo.com if anyone uses that.
Sarah Sharp
Quoted 13 lines · Sarah Sharp
Quoted 12 lines · Greg KH
Quoted 10 lines · Sarah Sharp
Topic 1
-------The KVM folks suggested that it would be good to get USB and virtualization developers together to talk about how to virtualize the xHCI host controller. The xHCI spec architect worked closely with VMWare to get some extra goodies in the spec to help virtualization, and I'd like to see the other virtualization developers take advantage of that. I'd also like us to hash out any issues they have been finding in the USB core or xHCI driver during the virtualization effort.
Quoted 6 lines
Quoted 2 lines
Do people really want to virtualize the whole xHCI controller, or just specific ports or devices to the guest operating system?
A host OS could chose to virtualize the whole xHCI controller if it wanted to. That's part of the reason why xHCI does all the bandwidth checking in hardware, not in software.
And here I thought it did that so it would be "correct" :)
Quoted 8 lines
Quoted 4 lines
If just specific ports, would something like usbip be better for virtual machines, with the USB traffic going over the network connection between the guest/host?
It could be done that way too. But that doesn't help if you're trying to run Windows under Linux, right? Only if all the guest OSes use the same USB IP protocol then it would work.
usbip works on Windows as well as Linux.
But how could you run Windows with a xHCI controller in a guest, as Windows has no xHCI driver? What would it expect to see?
Quoted 10 lines
Quoted 6 lines
Quoted 1 line
Hope to see you there!
Thanks for putting this together.
Do we need to sign up somewhere to give the organizers a sense of how many people will be attending?
I private ack by email would be great. Or you can ack by facebook:
https://www.facebook.com/event.php?eid=229532200405657 I could add the event in upcoming.yahoo.com if anyone uses that.
Nice try, but I'm still not going to join facebook...
Am 10.06.2011 um 22:50 schrieb Greg KH <[email protected]>:
Quoted 28 lines · Sarah Sharp
Quoted 13 lines · Greg KH
Quoted 12 lines · Sarah Sharp
Quoted 10 lines
Topic 1
-------The KVM folks suggested that it would be good to get USB and virtualization developers together to talk about how to virtualize the xHCI host controller. The xHCI spec architect worked closely with VMWare to get some extra goodies in the spec to help virtualization, and I'd like to see the other virtualization developers take advantage of that. I'd also like us to hash out any issues they have been finding in the USB core or xHCI driver during the virtualization effort.
Quoted 6 lines
Quoted 2 lines
Do people really want to virtualize the whole xHCI controller, or just specific ports or devices to the guest operating system?
A host OS could chose to virtualize the whole xHCI controller if it wanted to. That's part of the reason why xHCI does all the bandwidth checking in hardware, not in software.
And here I thought it did that so it would be "correct" :)
Quoted 5 lines
Quoted 4 lines
If just specific ports, would something like usbip be better for virtual machines, with the USB traffic going over the network connection between the guest/host?
Quoted 5 lines
Quoted 3 lines
It could be done that way too. But that doesn't help if you're trying to run Windows under Linux, right? Only if all the guest OSes use the same USB IP protocol then it would work.
usbip works on Windows as well as Linux.
Do you have a reliable, working usbip solution at hand that work on Windows and Linux and doesn't require real network access, which can be a no-go for some scenarios?
Quoted 3 lines
But how could you run Windows with a xHCI controller in a guest, as Windows has no xHCI driver? What would it expect to see?
There are drivers for xhci adapters on Windows. Also, this whole discussion is pretty much future oriented - which most likely means built-in xhci drivers anywhere.
No, in all seriousity, the main reason to go for FV vs PV is that so far, the best tested and working drivers are built for real hardware. I'm still unsatisfied with the PV driver situation for virtio on Windows. It's just incredibly hard to get Windows drivers right - and open source developers sure are not good at it :)
But sure, let's talk about it during LinuxCon as well - there's a good chance you know more there than me :)
Quoted 1 line
Alex
Quoted 35 lines · Alexander Graf
Am 10.06.2011 um 22:50 schrieb Greg KH <[email protected]>:
Quoted 32 lines · Sarah Sharp
Quoted 13 lines · Greg KH
Quoted 12 lines · Sarah Sharp
Quoted 10 lines
Topic 1
-------The KVM folks suggested that it would be good to get USB and virtualization developers together to talk about how to virtualize the xHCI host controller. The xHCI spec architect worked closely with VMWare to get some extra goodies in the spec to help virtualization, and I'd like to see the other virtualization developers take advantage of that. I'd also like us to hash out any issues they have been finding in the USB core or xHCI driver during the virtualization effort.
Quoted 6 lines
Quoted 2 lines
Do people really want to virtualize the whole xHCI controller, or just specific ports or devices to the guest operating system?
A host OS could chose to virtualize the whole xHCI controller if it wanted to. That's part of the reason why xHCI does all the bandwidth checking in hardware, not in software.
And here I thought it did that so it would be "correct" :)
Quoted 8 lines
Quoted 4 lines
If just specific ports, would something like usbip be better for virtual machines, with the USB traffic going over the network connection between the guest/host?
It could be done that way too. But that doesn't help if you're trying to run Windows under Linux, right? Only if all the guest OSes use the same USB IP protocol then it would work.
Quoted 5 lines
Quoted 1 line
usbip works on Windows as well as Linux.
Do you have a reliable, working usbip solution at hand that work on Windows and Linux and doesn't require real network access, which can be a no-go for some scenarios?
What do you mean by "real network access"? A virtual network connection to the guest and host should be sufficient, right?
Quoted 4 lines
Quoted 2 lines
But how could you run Windows with a xHCI controller in a guest, as Windows has no xHCI driver? What would it expect to see?
There are drivers for xhci adapters on Windows.
Not well-working ones, see the windows driver mailing list as well as the libusb mailing list for problems that people are having due to there not being any "official" Microsoft xhci drivers shipping yet.
Quoted 2 lines
Also, this whole discussion is pretty much future oriented - which most likely means built-in xhci drivers anywhere.
Agreed.
thanks,
greg k-h
Quoted 36 lines · Greg KH
Quoted 35 lines · Alexander Graf
Am 10.06.2011 um 22:50 schrieb Greg KH <[email protected]>:
Quoted 32 lines · Sarah Sharp
Quoted 13 lines · Greg KH
Quoted 12 lines · Sarah Sharp
Quoted 10 lines
Topic 1
-------The KVM folks suggested that it would be good to get USB and virtualization developers together to talk about how to virtualize the xHCI host controller. The xHCI spec architect worked closely with VMWare to get some extra goodies in the spec to help virtualization, and I'd like to see the other virtualization developers take advantage of that. I'd also like us to hash out any issues they have been finding in the USB core or xHCI driver during the virtualization effort.
Quoted 6 lines
Quoted 2 lines
Do people really want to virtualize the whole xHCI controller, or just specific ports or devices to the guest operating system?
A host OS could chose to virtualize the whole xHCI controller if it wanted to. That's part of the reason why xHCI does all the bandwidth checking in hardware, not in software.
And here I thought it did that so it would be "correct" :)
Quoted 8 lines
Quoted 4 lines
If just specific ports, would something like usbip be better for virtual machines, with the USB traffic going over the network connection between the guest/host?
It could be done that way too. But that doesn't help if you're trying to run Windows under Linux, right? Only if all the guest OSes use the same USB IP protocol then it would work.
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
usbip works on Windows as well as Linux.
Do you have a reliable, working usbip solution at hand that work on Windows and Linux and doesn't require real network access, which can be a no-go for some scenarios?
What do you mean by "real network access"? A virtual network connection to the guest and host should be sufficient, right?
Yes, which would either mean we inject a second network adapter into the guest which needs to be configured to an IP address, hence possibly colliding with some other IP range that's in use outside, or reuse some real network connection, which is neither always given, nor always desired. We have a full new PV serial protocol (virtio-serial) just for exactly that reason: To be able to run a daemon inside the guest that does not require IP access to talk to the host.
USB/IP really has its own huge set of issues. Sure, it might be good for certain use cases, but it's definitely not a one-size-fits-all solution. But so far, doing FV emulation right has worked out a lot better for most use cases than inventing PV solutions. Real hardware simply seems to be supported by others, while we can't afford to write drivers for all OSs :). So IMHO, PV should be the last resort, when emulating real hardware utterly fails. In this case, I believe the hardware designers finally got the spec usable for us, so it should be good! :)
Quoted 9 lines
Quoted 5 lines
Quoted 3 lines
But how could you run Windows with a xHCI controller in a guest, as Windows has no xHCI driver? What would it expect to see?
There are drivers for xhci adapters on Windows.
Not well-working ones, see the windows driver mailing list as well as the libusb mailing list for problems that people are having due to there not being any "official" Microsoft xhci drivers shipping yet.
Yes, which is an issue that's hopefully gone when we're finally having xHCI actually properly emulated and all the infrastructure for it laid out. We still have UHCI and OHCI emulation today already to fall back to if really necessary, but installing an xHCI driver inside the guest really isn't more work than installing USB/IP inside the guest.
Alex
Hi,
Quoted 7 lines · Sarah Sharp
I'm pleased to announce a USB mini-summit at LinuxCon Vancouver.
What: USB mini-summit
When: Tuesday, August 16th, all day
Where: At the conference venue, room TBD pending confirmation from
Angela Brown.
Quoted 5 lines
Proposed topics include USB virtualization, and improved bandwidth APIs between the USB core and drivers (especially webcam drivers). See the detailed topic list below. Anyone is also welcome to propose or show up with a USB related topic. MUSB? USB 3.0 gadget drivers? USB-IP?
I would like to give a short presentation on / demo off the usb redirection over the network (USB-IP replacement with a less dumb protocol) I've been working on, followed by a questions and answers session.
For those who want to give this a try now, see:
http://hansdegoede.livejournal.com/9682.html
Sheets from the presentation I gave on this at FOSDEM:
http://people.fedoraproject.org/~jwrdegoede/usb-redir.pdf
Quoted 10 lines
The USB mini-summit does overlap with the virtualization mini-summit by a day, but I'm hoping we can schedule talks so some of the virtualization folks can make it to the USB mini-summit. The other option was on Friday during the conference which was not ideal.
Proposed topics:
Topic 1
-------
Quoted 7 lines
The KVM folks suggested that it would be good to get USB and virtualization developers together to talk about how to virtualize the xHCI host controller. The xHCI spec architect worked closely with VMWare to get some extra goodies in the spec to help virtualization, and I'd like to see the other virtualization developers take advantage of that. I'd also like us to hash out any issues they have been finding in the USB core or xHCI driver during the virtualization effort.
I'm not really happy with how the management / hand over of userspace to kernel space drivers and vice versa works. This is a problem when doing usb-redirection to a virtual machine. I think this is best discussed in a separate thread, and then if needed further discussed during the mini summit. I'll send a mail on this shortly. I'll use the same address list as this mail, except that I'm going to cut linux-kernel from the CC list.
Quoted 17 lines
Topic 2
-------I'd also like to get the V4L and audio developers who work with USB devices together with the core USB folks to talk about bandwidth management under xHCI.
One of the issues is that since the xHCI hardware does bandwidth management, not the xHCI driver, a schedule that will take too much bandwidth will get rejected much sooner than any USB driver currently expects (during a call to
usb_set_interface). This poses issues, since most USB video drivers negotiate the video size and frame rate after they callusb_set_interface, so they don't know whether they can fall back to a less bandwidth-intensive setting. Currently, they just submit URBs with less and less bandwidth until one interval setting gets accepted that won't work under xHCI.
Quoted 8 lines
A second issue is that that some drivers need less bandwidth than the device advertises, and the xHCI driver currently uses whatever periodic interval the device advertises in its descriptors. This is not what the video/audio driver wants, especially in the case of buggy high speed devices that advertise the interval in frames, not microframes. There needs to be some way for the drivers to communicate their bandwidth needs to the USB core. We've known about this issue for a while, and I think it's time to get everyone in the same room and hash out an API.
Interesting, being able to tell the core how much bandwidth a device will actually use (versus what it claims as maxpacketsize in its descriptors) is something which we really need. I know of at least 2 usb1 webcam chipsets (and drivers) which have only 1 altsetting which claim 1023 bytes maxpacketsize. But they also have a register which allows the driver to configure how large the largest (iso) packet it sends will actually be.
Currently these drivers have this "beauty" to be able to tell the linux usb core that they won't be using 1023 as maxpacketsize but something else, and thus function without needing full usb1 bandwidth:
struct usb_host_interface *alt;
alt = &gspca_dev->dev->config->intf_cache[0]->altsetting[1]; alt->endpoint[0].desc.wMaxPacketSize = cpu_to_le16(packet_size); ret = usb_set_interface(gspca_dev->dev, gspca_dev->iface, 1);
Yes they are overwriting the kernels cached descriptors ...
###
Something which I would also like to bring to everyone's attention is that we really need to fix the ehci schedule code wrt scheduling usb1 transfers over usb2 hubs.
The current code is very broken when it comes to periodic transfers, it basically disallows using the last microframe (let alone the crossing of the frame boundary and using the first microframe of the next frame).
This means that trying to submit isoc transfers with a size of 1023 will just plain fail, even if this is the only device on the entire bus. This is one of the reasons I ended up doing the hack above, so that these devices will at least work (be it at a decreased framerate) through a usb2 hub.
Things become even messier when the device has a build in usb audio microphone, often this will just not work.
Finally fixing this has become more important now then ever since sandybridge machines (and maybe generation one core i# machines too, I don't know) no longer have a companion controller, but instead have an integrated usb2 hub.
Regards,
Hans
Improving kernel -> userspace (usbfs) usb device hand off
In reply to Sarah SharpHi all,
The current API for managing kernel -> userspace is a bit rough around the edges, so I would like to discuss extending the API.
First of all an example use case scenarios where the current API falls short.
1) Redirection of USB devices to a virtual machine, qemu, vbox, etc. all have the ability to redirect a USB device to the virtual machine, and they all use usbfs for this. The first thing which will happen here when the user selects a device to redirect is a IOCTL_USBFS_DISCONNECT. This causes the kernel driver to see a device unplug, with no chances for the kernel driver to do anything against this.
Now lets say the user does the following:
-write a file to a usb flash disk
-redirect the flash disk to a vm
Currently this will cause the usb mass storage driver to see a disconnect, and any possible still pending writes are lost ...
This is IMHO unacceptable, but currently there is nothing we can do to avoid this.
2) So called dual mode cameras are (cheap) stillcams often even without an lcdscreen viewfinder, and battery backed sram instead of flash, which double as a webcam. We have drivers for both the stillcam function of these (in libgphoto2, so using usbfs) as well as for the webcam function (v4l2 kernel drivers).
These drivers work well, and are mature. Yet the user experience is rather poor. Under gnome the still-cam contents will be automatically be made available as a "drive" using a gvfs-gphoto2 fuse mount. This however involves sending a disconnect to the v4l2 kernel driver, and thus the /dev/video# node disappearing. So if a user wants to use the device as a webcam he/she needs to first go to nautilus and unmount the gvfs mount. Until that is done the user will simply get a message from an app like cheese that he has no webcam, not even an ebusy error, just that he has no such device.
Again not good.
###
So what do we need to make this situation better:
1) A usb_driver callback alternative to the disconnect callback, I propose to call this soft_disconnect. This serves 2 purposes
a) It will allow the driver to tell the caller that that is not a good idea by returning an error code (think usb mass storage driver and mounted filesystem
b) It will allow for example a v4l2 driver to keep its /dev/video node around
Note that b) means that the normal disconnect handler should still be called after a soft reconnect on a real disconnect.
2) A usb_driver soft_reconnect callback to match the soft_disconnect
3) A mechanism for a usb_driver to signal a usbfs fd owner of the device it would like the device back. So for example the gvfs mount can be automatically unmounted (if not busy).
4) A IOCTL_USBFS_SOFT_DISCONNECT ioctl which will call the drivers soft_disconnect if it has one, and otherwise fall back to the regular disconnect.
5) A method for a usbfs fd owning app to know the device driver would like the device back. I suggest using poll with POLLIN to signal this.
Regards,
Hans
Re: Improving kernel -> userspace (usbfs) usb device hand off
Hi,
Quoted 12 lines · Hans de Goede
Currently this will cause the usb mass storage driver to see a disconnect, and any possible still pending writes are lost ...
This is IMHO unacceptable, but currently there is nothing we can do to avoid this.
2) So called dual mode cameras are (cheap) stillcams often even without an lcdscreen viewfinder, and battery backed sram instead of flash, which double as a webcam. We have drivers for both the stillcam function of these (in libgphoto2, so using usbfs) as well as for the webcam function (v4l2 kernel drivers).
Quoted 9 lines
These drivers work well, and are mature. Yet the user experience is rather poor. Under gnome the still-cam contents will be automatically be made available as a "drive" using a gvfs-gphoto2 fuse mount. This however involves sending a disconnect to the v4l2 kernel driver, and thus the /dev/video# node disappearing. So if a user wants to use the device as a webcam he/she needs to first go to nautilus and unmount the gvfs mount. Until that is done the user will simply get a message from an app like cheese that he has no webcam, not even an ebusy error, just that he has no such device.
that sounds quite weird. Should only happen if still image and video functions are on different configurations or different alt-settings of the same interface. But if they are on same configurations and separate interfaces, you should be able to bind gphoto to the still image interface and v4l2 to the camera interface.
How's the device setup ?
Quoted 6 lines
So what do we need to make this situation better:
1) Ausb_drivercallback alternative to the disconnect callback, I propose to call thissoft_disconnect. This serves 2 purposes
a) It will allow the driver to tell the caller that that is not a good idea by returning an error code (think usb mass storage driver and mounted filesystem
I'm not sure you even need a driver callback for that. Should we leave that to Desktop manager ?
Signature · 1 line
balbi
Re: Improving kernel -> userspace (usbfs) usb device hand off
Hi,
Quoted 25 lines · Felipe Balbi
Hi,
Quoted 12 lines · Hans de Goede
Currently this will cause the usb mass storage driver to see a disconnect, and any possible still pending writes are lost ...
This is IMHO unacceptable, but currently there is nothing we can do to avoid this.
2) So called dual mode cameras are (cheap) stillcams often even without an lcdscreen viewfinder, and battery backed sram instead of flash, which double as a webcam. We have drivers for both the stillcam function of these (in libgphoto2, so using usbfs) as well as for the webcam function (v4l2 kernel drivers).
Quoted 9 lines
These drivers work well, and are mature. Yet the user experience is rather poor. Under gnome the still-cam contents will be automatically be made available as a "drive" using a gvfs-gphoto2 fuse mount. This however involves sending a disconnect to the v4l2 kernel driver, and thus the /dev/video# node disappearing. So if a user wants to use the device as a webcam he/she needs to first go to nautilus and unmount the gvfs mount. Until that is done the user will simply get a message from an app like cheese that he has no webcam, not even an ebusy error, just that he has no such device.
Quoted 8 lines
that sounds quite weird. Should only happen if still image and video functions are on different configurations or different alt-settings of the same interface. But if they are on same configurations and separate interfaces, you should be able to bind gphoto to the still image interface and v4l2 to the camera interface.
How's the device setup ?
These are very cheap devices, and as such poorly designed. There still and webcam functionality is on the same interface. This is likely done this way because the devices cannot handle both functions at the same time.
Quoted 9 lines
Quoted 6 lines
So what do we need to make this situation better:
1) Ausb_drivercallback alternative to the disconnect callback, I propose to call thissoft_disconnect. This serves 2 purposes
a) It will allow the driver to tell the caller that that is not a good idea by returning an error code (think usb mass storage driver and mounted filesystemI'm not sure you even need a driver callback for that. Should we leave that to Desktop manager ?
Not sure what you mean here, but we need for a way for drivers to say no to a software caused disconnection. See my usb mass storage device which is still mounted getting redirected to a vm example. This cannot be reliably done from userspace. Where as it is trivial to do this from kernel space. One could advocate to make the existing disconnect ioctl use the new soft_disconnect usb_driver callback instead of adding a new usbfs ioctl for this, but that means that a driver can block any and all userspace triggered disconnects. Where as having a new ioctl, means that apps which want to play nice can play nice, while keeping the possibility of a hard userspace initiated disconnect.
One could also argue that making the existing disconnect ioctl return -EBUSY in some cases now is an ABI change.
Regards,
Hans
Re: Improving kernel -> userspace (usbfs) usb device hand off
Hi,
Quoted 29 lines · Hans de Goede
Quoted 29 lines · Hans de Goede
Quoted 12 lines
Currently this will cause the usb mass storage driver to see a disconnect, and any possible still pending writes are lost ...
This is IMHO unacceptable, but currently there is nothing we can do to avoid this.
2) So called dual mode cameras are (cheap) stillcams often even without an lcdscreen viewfinder, and battery backed sram instead of flash, which double as a webcam. We have drivers for both the stillcam function of these (in libgphoto2, so using usbfs) as well as for the webcam function (v4l2 kernel drivers).
Quoted 9 lines
These drivers work well, and are mature. Yet the user experience is rather poor. Under gnome the still-cam contents will be automatically be made available as a "drive" using a gvfs-gphoto2 fuse mount. This however involves sending a disconnect to the v4l2 kernel driver, and thus the /dev/video# node disappearing. So if a user wants to use the device as a webcam he/she needs to first go to nautilus and unmount the gvfs mount. Until that is done the user will simply get a message from an app like cheese that he has no webcam, not even an ebusy error, just that he has no such device.
that sounds quite weird. Should only happen if still image and video functions are on different configurations or different alt-settings of the same interface. But if they are on same configurations and separate interfaces, you should be able to bind gphoto to the still image interface and v4l2 to the camera interface.
Quoted 7 lines
Quoted 2 lines
How's the device setup ?
These are very cheap devices, and as such poorly designed. There still and webcam functionality is on the same interface. This is likely done this way because the devices cannot handle both functions at the same time.
ok got it.
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
So what do we need to make this situation better:
1) Ausb_drivercallback alternative to the disconnect callback, I propose to call thissoft_disconnect. This serves 2 purposes
a) It will allow the driver to tell the caller that that is not a good idea by returning an error code (think usb mass storage driver and mounted filesystem
Quoted 6 lines
Quoted 2 lines
I'm not sure you even need a driver callback for that. Should we leave that to Desktop manager ?
Not sure what you mean here, but we need for a way for drivers to say no to a software caused disconnection. See my usb mass storage device which is still mounted getting redirected to a vm example. This cannot
in that case, why don't you just flush all data and continue ? Also, desktop manager knows that a particular device mounted, so it could also ask the user if s/he wants to continue.
I'm not sure preventing a disconnect is a good thing.
Quoted 11 lines
be reliably done from userspace. Where as it is trivial to do this from kernel space. One could advocate to make the existing disconnect ioctl use the new
soft_disconnectusb_drivercallback instead of adding a new usbfs ioctl for this, but that means that a driver can block any and all userspace triggered disconnects. Where as having a new ioctl, means that apps which want to play nice can play nice, while keeping the possibility of a hard userspace initiated disconnect.One could also argue that making the existing disconnect ioctl return
-EBUSYin some cases now is an ABI change.
OTOH, if the user really wants to move the usb device to the guest OS, he has just requested for that, so should we prevent it ? what we need is for the applications to be notified to exit cleanly and release the device because the user has requested to do so. No ?
Signature · 1 line
balbi
Re: Improving kernel -> userspace (usbfs) usb device hand off
Hi,
Quoted 4 lines · Felipe Balbi
Hi,
Quoted 1 line · Hans de Goede
Quoted 1 line · Hans de Goede
<snip>
Quoted 14 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
So what do we need to make this situation better:
1) Ausb_drivercallback alternative to the disconnect callback, I propose to call thissoft_disconnect. This serves 2 purposes
a) It will allow the driver to tell the caller that that is not a good idea by returning an error code (think usb mass storage driver and mounted filesystemQuoted 6 lines
Quoted 2 lines
I'm not sure you even need a driver callback for that. Should we leave that to Desktop manager ?
Not sure what you mean here, but we need for a way for drivers to say no to a software caused disconnection. See my usb mass storage device which is still mounted getting redirected to a vm example. This cannot
Quoted 6 lines
in that case, why don't you just flush all data and continue ? Also, desktop manager knows that a particular device mounted, so it could also ask the user if s/he wants to continue.
I'm not sure preventing a disconnect is a good thing.
I assume you are sure preventing data loss is a good thing? Because in this example the 2 are the same.
Also note I'm not suggestion at always preventing the disconnect, I'm suggesting to add a new try_disconnect ioctl, which apps which want to behave nicely can use instead of the regular disconnect ioctl, and then drivers can prevent the disconnect. Apps using the old ioctl will still get an unconditional disconnect as before.
Quoted 9 lines
Quoted 9 lines
be reliably done from userspace. Where as it is trivial to do this from kernel space. One could advocate to make the existing disconnect ioctl use the new
soft_disconnectusb_drivercallback instead of adding a new usbfs ioctl for this, but that means that a driver can block any and all userspace triggered disconnects. Where as having a new ioctl, means that apps which want to play nice can play nice, while keeping the possibility of a hard userspace initiated disconnect.
Quoted 7 lines
Quoted 2 lines
One could also argue that making the existing disconnect ioctl return
-EBUSYin some cases now is an ABI change.OTOH, if the user really wants to move the usb device to the guest OS, he has just requested for that, so should we prevent it ? what we need is for the applications to be notified to exit cleanly and release the device because the user has requested to do so. No ?
We are talking about a device with a mounted file system on it here, any process could be holding files open on there, and there currently exists no mechanism to notify all apps to exit cleanly and release the files. Even if there were some method for a desktop environment like gnome to ask apps to release those files, and all gnome apps where to be modified to support that mechanism, then there are still 1000-s of non gnome (or kde, xfce, whatever) apps which will not support that.
We already have a mechanism to cleanly close down a filesystem, it is called unmount. And it will fail if apps have files open. All I'm suggesting is forwarding this ebusy failure to the application trying to disconnect the driver from the usb mass storage interface.
Simply removing the filesystem from under apps holding files open will lead to io errors, and very unhappy apps.
Regards,
Hans
Re: Improving kernel -> userspace (usbfs) usb device hand off
Hi,
Quoted 20 lines · Hans de Goede
Quoted 14 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
So what do we need to make this situation better:
1) Ausb_drivercallback alternative to the disconnect callback, I propose to call thissoft_disconnect. This serves 2 purposes
a) It will allow the driver to tell the caller that that is not a good idea by returning an error code (think usb mass storage driver and mounted filesystemQuoted 6 lines
Quoted 2 lines
I'm not sure you even need a driver callback for that. Should we leave that to Desktop manager ?
Not sure what you mean here, but we need for a way for drivers to say no to a software caused disconnection. See my usb mass storage device which is still mounted getting redirected to a vm example. This cannot
Quoted 5 lines
in that case, why don't you just flush all data and continue ? Also, desktop manager knows that a particular device mounted, so it could also ask the user if s/he wants to continue.
I'm not sure preventing a disconnect is a good thing.
Quoted 8 lines
I assume you are sure preventing data loss is a good thing? Because in this example the 2 are the same.
Also note I'm not suggestion at always preventing the disconnect, I'm suggesting to add a new
try_disconnectioctl, which apps which want to behave nicely can use instead of the regular disconnect ioctl, and then drivers can prevent the disconnect. Apps using the old ioctl will still get an unconditional disconnect as before.
I'm just not sure if this should sit in the kernel. You can easily have some sort of device manager (much like udisks for storage) to track this down for ya. So instead of calling usbfs straight, you need to go via the "device manager". This device manager should know that before calling 'disconnect' it must be sure there's nothing pending.
Quoted 8 lines
Quoted 7 lines
Quoted 2 lines
One could also argue that making the existing disconnect ioctl return
-EBUSYin some cases now is an ABI change.OTOH, if the user really wants to move the usb device to the guest OS, he has just requested for that, so should we prevent it ? what we need is for the applications to be notified to exit cleanly and release the device because the user has requested to do so. No ?
Quoted 8 lines
We are talking about a device with a mounted file system on it here, any process could be holding files open on there, and there currently exists no mechanism to notify all apps to exit cleanly and release the files. Even if there were some method for a desktop environment like gnome to ask apps to release those files, and all gnome apps where to be modified to support that mechanism, then there are still 1000-s of non gnome (or kde, xfce, whatever) apps which will not support that.
Still, I'm not sure this is argument good enough to add another sort of conflicting ioctl to the kernel. It won't prevent apps from calling the other disconnect anyway. And besides, if an app can be modified to use the try_disconnect ioctl, they can be modified for the system notification (on userland) too, can't they ?
Quoted 7 lines
We already have a mechanism to cleanly close down a filesystem, it is called unmount. And it will fail if apps have files open. All I'm suggesting is forwarding this ebusy failure to the application trying to disconnect the driver from the usb mass storage interface.
Simply removing the filesystem from under apps holding files open will lead to io errors, and very unhappy apps.
Correct, but does the solution _have_ to sit in kernel ? I'm not sure. If you add the try_disconnect ioctl and all applications one your desktop, except one, are playing nice, you will still have problems. To me, it sounds much nicer to add such things on userland alone, in a way such that it doesn't matter if it's a usb storage, mmc card, or eSATA drive. That should be hidden from apps IMHO.
Signature · 1 line
balbi
Re: Improving kernel -> userspace (usbfs) usb device hand off
In reply to Hans de GoedeQuoted 21 lines · Hans de Goede
Hi all,
The current API for managing kernel -> userspace is a bit rough around the edges, so I would like to discuss extending the API.
First of all an example use case scenarios where the current API falls short.
1) Redirection of USB devices to a virtual machine, qemu, vbox, etc. all have the ability to redirect a USB device to the virtual machine, and they all use usbfs for this. The first thing which will happen here when the user selects a device to redirect is a
IOCTL_USBFS_DISCONNECT. This causes the kernel driver to see a device unplug, with no chances for the kernel driver to do anything against this.Now lets say the user does the following:
(quoted diff) +0
-2-write a file to a usb flash disk -redirect the flash disk to a vm
Quoted 5 lines
Currently this will cause the usb mass storage driver to see a disconnect, and any possible still pending writes are lost ...
This is IMHO unacceptable, but currently there is nothing we can do to avoid this.
You haven't given a proper description of the problem. See below.
Quoted 16 lines
2) So called dual mode cameras are (cheap) stillcams often even without an lcdscreen viewfinder, and battery backed sram instead of flash, which double as a webcam. We have drivers for both the stillcam function of these (in libgphoto2, so using usbfs) as well as for the webcam function (v4l2 kernel drivers).
These drivers work well, and are mature. Yet the user experience is rather poor. Under gnome the still-cam contents will be automatically be made available as a "drive" using a gvfs-gphoto2 fuse mount. This however involves sending a disconnect to the v4l2 kernel driver, and thus the /dev/video# node disappearing. So if a user wants to use the device as a webcam he/she needs to first go to nautilus and unmount the gvfs mount. Until that is done the user will simply get a message from an app like cheese that he has no webcam, not even an ebusy error, just that he has no such device.
Quoted 1 line
Again not good.
As Felipe has mentioned, this sounds like the sort of problem that can better be solved in userspace. A dual-mode device like the one you describe really is either a still-cam or a webcam, never both at the same time. Hence what users need is a utility program to switch modes (by loading/unloading the appropriate programs or drivers). Or maybe a desktop daemon that could accomplish the same result automatically, based on requests from user programs.
Quoted 8 lines
###
So what do we need to make this situation better:
1) Ausb_drivercallback alternative to the disconnect callback, I propose to call thissoft_disconnect. This serves 2 purposes
a) It will allow the driver to tell the caller that that is not a good idea by returning an error code (think usb mass storage driver and mounted filesystem
Not feasible. usb-storage has no idea whether or not a device it controls has a mounted filesystem. (All it does is send SCSI commands to a device and get back the results.) Since that's the main use case you're interested in, this part of the proposal seems destined to fail.
But userspace _does_ know where the mounted filesystems are. Therefore userspace should be responsible for avoiding programs that want to take control of devices holding these filesystems. That's the reason why usbfs device nodes are owned by root and have 0644 mode; there're can be written to only by programs with superuser privileges -- and such programs are supposed to be careful about what they do.
Quoted 4 lines
b) It will allow for example a v4l2 driver to keep its
/dev/videonode around
Note that b) means that the normal disconnect handler should still be called after a soft reconnect on a real disconnect.
In some sense the disconnect callback for usb-storage already _is_ "soft". Although the driver cannot refuse the disconnect, it _can_ continue to communicate with the device until the callback returns.
(As it happens, usb-storage _doesn't_ do any further communication with the device. This is mostly for historical reasons, to compensate for shortcomings in the SCSI stack in earlier kernel versions. On the other hand, there really isn't much that you would want to send to a mass-storage device during a soft disconnect. Perhaps tell it to flush its cache out to the storage medium -- but if there are no open file handles for the device and no mounted filesystems then the cache will already be flushed.)
Quoted 4 lines
2) A
usb_driversoft_reconnectcallback to match thesoft_disconnect
3) A mechanism for ausb_driverto signal a usbfs fd owner of the device it would like the device back. So for example the gvfs mount can be automatically unmounted (if not busy).
This also should be handled in userspace. USB drivers never "want back" a device they are no longer bound to -- in fact, the device model used throughout the kernel makes this whole idea meaningless. Device drivers don't want devices back. Rather, _users_ want to turn control of devices over to specific drivers. That's why this problem needs to be handled in userspace.
Quoted 5 lines
4) A
IOCTL_USBFS_SOFT_DISCONNECTioctl which will call the driverssoft_disconnectif it has one, and otherwise fall back to the regular disconnect.
5) A method for a usbfs fd owning app to know the device driver would like the device back. I suggest using poll with POLLIN to signal this.
It seems as if you're trying to implement some notion of allowing a device to have more than one driver at the same time. This is so far out from the way the kernel behaves now, adopting it would be very difficult if not impossible. Certainly the USB stack isn't the place to start.
Alan Stern
Re: Improving kernel -> userspace (usbfs) usb device hand off
Em 10-06-2011 11:48, Alan Stern escreveu:
Quoted 8 lines · Hans de Goede
As Felipe has mentioned, this sounds like the sort of problem that can better be solved in userspace. A dual-mode device like the one you describe really is either a still-cam or a webcam, never both at the same time. Hence what users need is a utility program to switch modes (by loading/unloading the appropriate programs or drivers).
Unloading a driver in order to access the hardware via userspace? This sounds a very bad idea do me. What happens if another hardware is using the same driver?
Cheers,
Mauro.
Re: Improving kernel -> userspace (usbfs) usb device hand off
Quoted 10 lines · Mauro Carvalho Chehab
Em 10-06-2011 11:48, Alan Stern escreveu:
Quoted 8 lines · Hans de Goede
As Felipe has mentioned, this sounds like the sort of problem that can better be solved in userspace. A dual-mode device like the one you describe really is either a still-cam or a webcam, never both at the same time. Hence what users need is a utility program to switch modes (by loading/unloading the appropriate programs or drivers).
Quoted 3 lines
Unloading a driver in order to access the hardware via userspace? This sounds a very bad idea do me. What happens if another hardware is using the same driver?
A kernel driver wouldn't have to be unloaded. It could simply be unbound from the device via sysfs or usbfs.
Alan STern
Re: Improving kernel -> userspace (usbfs) usb device hand off
In reply to Alan SternHi,
Given the many comments in this thread, I'm just going reply to this one, and try to also answer any other ones in this mail.
As far as the dual mode camera is involved, I agree that that should be fixed in the existing v4l2 drivers + libgphoto. I think that Felipe's solution to also handle the stillcam part in kernel space for dual mode cameras (and add a libgphoto cam driver which knows how to talk the new kernel API for this), is the best solution. Unfortunately this will involve quite a bit of work, but so be it.
Quoted 23 lines · Alan Stern
Quoted 21 lines · Hans de Goede
Hi all,
The current API for managing kernel -> userspace is a bit rough around the edges, so I would like to discuss extending the API.
First of all an example use case scenarios where the current API falls short.
1) Redirection of USB devices to a virtual machine, qemu, vbox, etc. all have the ability to redirect a USB device to the virtual machine, and they all use usbfs for this. The first thing which will happen here when the user selects a device to redirect is a
IOCTL_USBFS_DISCONNECT. This causes the kernel driver to see a device unplug, with no chances for the kernel driver to do anything against this.Now lets say the user does the following:
(quoted diff) +0
-2-write a file to a usb flash disk -redirect the flash disk to a vm
Quoted 7 lines
Quoted 5 lines
Currently this will cause the usb mass storage driver to see a disconnect, and any possible still pending writes are lost ...
This is IMHO unacceptable, but currently there is nothing we can do to avoid this.
You haven't given a proper description of the problem. See below.
I think I've given an excellent description of the problem, drivers can be unbound from devices, and there is no way for drivers to block this. All I'm asking for is for a new usbfs try_disconnect ioctl which the currently bound driver has a chance blocking nothing more nothing less.
<snip dual mode camera stuff>
Quoted 7 lines
Quoted 6 lines
So what do we need to make this situation better:
1) Ausb_drivercallback alternative to the disconnect callback, I propose to call thissoft_disconnect. This serves 2 purposes
a) It will allow the driver to tell the caller that that is not a good idea by returning an error code (think usb mass storage driver and mounted filesystem
Quoted 6 lines
Not feasible. usb-storage has no idea whether or not a device it controls has a mounted filesystem. (All it does is send SCSI commands to a device and get back the results.) Since that's the main use case you're interested in, this part of the proposal seems destined to fail.
This is not completely true, I cannot rmmod usb-storage as long as disks using it are mounted. I know this is done through the global module usage count, so this is not per usb-storage device. But extending the ref counting to be per usb-storage device should not be hard.
All the accounting is already done for this.
Quoted 7 lines
But userspace _does_ know where the mounted filesystems are. Therefore userspace should be responsible for avoiding programs that want to take control of devices holding these filesystems. That's the reason why usbfs device nodes are owned by root and have 0644 mode; there're can be written to only by programs with superuser privileges -- and such programs are supposed to be careful about what they do.
Yes, and what I'm asking for is for an easy way for these programs to be careful. A way for them to ask the kernel, which in general is responsible for things like this and traditionally does resource management and things which come with that like refcounting: "unbind the driver from this device unless the device is currently in use".
Why can't this be done in userspace, simply put:
1) Process a checks if the usb-storage device is not used
2) Process b mounts it after the check
3) Process a unbinds the driver
Yes this can be avoided if all binding/unbinding and all mounting happens under control of a single instance. However this requires completely re-inventing userspace...
Also note that the notion of this can be extended to other devices, want to ubs-redirect a usb printer to a vm better not do it halfway through a printjob being spooled. Want to usb-redirect a usb webcam to a vm, better not do it while something is streaming video from the webcam, etc. etc.
And please don't come with the inevitable "if it hurts do not do that" argument. We want to provide this kind of functionality to non tech savy computer users, and things should just work, including telling the user that the device is currently in use rather then wrecking his printjob or filesystem.
Regards,
Hans
Re: Improving kernel -> userspace (usbfs) usb device hand off
Quoted 6 lines · Hans de Goede
Hi,
Given the many comments in this thread, I'm just going reply to this one, and try to also answer any other ones in this mail.
Quoted 8 lines
As far as the dual mode camera is involved, I agree that that should be fixed in the existing v4l2 drivers + libgphoto. I think that Felipe's solution to also handle the stillcam part in kernel space for dual mode cameras (and add a libgphoto cam driver which knows how to talk the new kernel API for this), is the best solution. Unfortunately this will involve quite a bit of work, but so be it.
Hans,
It appears to me that the solution ought to be at hand, actually.
I was not aware of the recent changes in libusb, which I understand are supposed to allow a kernel driver to be hooked up again.
To review the situation:
1. As of approximately 2 years ago, libusb already was so configured as to suspend the kernel module for a dual-mode device if a userspace-based program tried to claim the device.
2. At this point with the more recent versions of libusb (see the last message from yesterday, from Xiaofan Chen), we are supposed to be able to re-activate the kernel module for the device when it is relinquished by userspace.
This ought to take care of the problems completely, provided that the new capabilities of libusb are actually used and called upon in libgphoto2.
I have checked on what is happening, just now, on my own machine. I have libusb version 1.08 which ought to be recent enough. The advertised abilities did not work, however. Presumably, what is missing is on the other end of the problem, most likely in the functions in libgphoto2 which hook up a camera. That code would presumably need to call upon the new functionality of libusb. My currently installed version of libgphoto2 (from svn, but several months old) clearly does not contain the needed functionality. But it might have been put in recently and I did not notice. I guess that the first thing to do is to update my gphoto tree and then to see what happens. If things still don't work, then something needs to be updated and then things ought to work.
I will try to see that something gets done about this. Thank you for raising the old issue of dual-mode devices yet again, and thanks to Xiaofan Chen for pointing out that the needed missing half of the functionality is supposed to exist now in libusb. That had escaped my attention.
Theodore Kilgore
Re: Improving kernel -> userspace (usbfs) usb device hand off
Hi,
Quoted 52 lines · Theodore Kilgore
Quoted 6 lines · Hans de Goede
Hi,
Given the many comments in this thread, I'm just going reply to this one, and try to also answer any other ones in this mail.
Quoted 8 lines
As far as the dual mode camera is involved, I agree that that should be fixed in the existing v4l2 drivers + libgphoto. I think that Felipe's solution to also handle the stillcam part in kernel space for dual mode cameras (and add a libgphoto cam driver which knows how to talk the new kernel API for this), is the best solution. Unfortunately this will involve quite a bit of work, but so be it.
Hans,
It appears to me that the solution ought to be at hand, actually.
I was not aware of the recent changes in libusb, which I understand are supposed to allow a kernel driver to be hooked up again.
To review the situation:
1. As of approximately 2 years ago, libusb already was so configured as to suspend the kernel module for a dual-mode device if a userspace-based program tried to claim the device.
2. At this point with the more recent versions of libusb (see the last message from yesterday, from Xiaofan Chen), we are supposed to be able to re-activate the kernel module for the device when it is relinquished by userspace.
This ought to take care of the problems completely, provided that the new capabilities of libusb are actually used and called upon in libgphoto2.
I have checked on what is happening, just now, on my own machine. I have libusb version 1.08 which ought to be recent enough. The advertised abilities did not work, however. Presumably, what is missing is on the other end of the problem, most likely in the functions in libgphoto2 which hook up a camera. That code would presumably need to call upon the new functionality of libusb. My currently installed version of libgphoto2 (from svn, but several months old) clearly does not contain the needed functionality. But it might have been put in recently and I did not notice. I guess that the first thing to do is to update my gphoto tree and then to see what happens. If things still don't work, then something needs to be updated and then things ought to work.
Quoted 5 lines
I will try to see that something gets done about this. Thank you for raising the old issue of dual-mode devices yet again, and thanks to Xiaofan Chen for pointing out that the needed missing half of the functionality is supposed to exist now in libusb. That had escaped my attention.
Actually libusb and libgphoto have been using the rebind orginal driver functionality of the code for quite a while now, unfortunately this does not solve the problem, unless we somehow move to 1 central coordinator for the device the user experience will stay subpar.
Example, user downloads pictures from the camera using shotwell, gthumb, fspot or whatever, keeps the app in question open and the app in question keeps the gphoto2 device handle open.
User wants to do some skyping with video chat, skype complains it cannot find the device, since the kernel driver currently is unbound.
-> Poor user experience.
With having both functions in the kernel, the kernel could actually allow skype to use the dual mode cameras as video source, and if the user then were to switch to f-spot and try to import more photo's then he will get an -ebusy in f-spot. If he finishes skyping and then returns to f-spot everything will just continue working.
This is the kind of "seamless" user experience I'm aiming for here.
Regards,
Hans
Re: Improving kernel -> userspace (usbfs) usb device hand off
Quoted 55 lines · Hans de Goede
Hi,
Quoted 52 lines · Theodore Kilgore
Quoted 6 lines · Hans de Goede
Hi,
Given the many comments in this thread, I'm just going reply to this one, and try to also answer any other ones in this mail.
Quoted 8 lines
As far as the dual mode camera is involved, I agree that that should be fixed in the existing v4l2 drivers + libgphoto. I think that Felipe's solution to also handle the stillcam part in kernel space for dual mode cameras (and add a libgphoto cam driver which knows how to talk the new kernel API for this), is the best solution. Unfortunately this will involve quite a bit of work, but so be it.
Hans,
It appears to me that the solution ought to be at hand, actually.
I was not aware of the recent changes in libusb, which I understand are supposed to allow a kernel driver to be hooked up again.
To review the situation:
1. As of approximately 2 years ago, libusb already was so configured as to suspend the kernel module for a dual-mode device if a userspace-based program tried to claim the device.
2. At this point with the more recent versions of libusb (see the last message from yesterday, from Xiaofan Chen), we are supposed to be able to re-activate the kernel module for the device when it is relinquished by userspace.
This ought to take care of the problems completely, provided that the new capabilities of libusb are actually used and called upon in libgphoto2.
I have checked on what is happening, just now, on my own machine. I have libusb version 1.08 which ought to be recent enough. The advertised abilities did not work, however. Presumably, what is missing is on the other end of the problem, most likely in the functions in libgphoto2 which hook up a camera. That code would presumably need to call upon the new functionality of libusb. My currently installed version of libgphoto2 (from svn, but several months old) clearly does not contain the needed functionality. But it might have been put in recently and I did not notice. I guess that the first thing to do is to update my gphoto tree and then to see what happens. If things still don't work, then something needs to be updated and then things ought to work.
Quoted 8 lines
Quoted 5 lines
I will try to see that something gets done about this. Thank you for raising the old issue of dual-mode devices yet again, and thanks to Xiaofan Chen for pointing out that the needed missing half of the functionality is supposed to exist now in libusb. That had escaped my attention.
Actually libusb and libgphoto have been using the rebind orginal driver functionality of the code for quite a while now,
Oh? I can see that libusb is doing that, and I can also see that there is a "public" function for _unbinding_ a kernel driver, namely
int usb_detach_kernel_driver_np()
found in usb.h
and it is used in libgphoto, as well.
I am not sure that there is any corresponding rebind function which is public. Is it perhaps
int usb_get_driver_np()
???
By context (looking at libgphoto2-port/usb/libusb.c) I would think that this function is not the rebind function, but is only checking whether or not there is any potential conflict with a kernel driver. If I am right, then where is the publicly exported rebind function, and where does it currently get used in libgphoto2?
So frankly after my eagerness yesterday I do not see how it can easily be made to work, after all.
unfortunately this
Quoted 11 lines
does not solve the problem, unless we somehow move to 1 central coordinator for the device the user experience will stay subpar.
Example, user downloads pictures from the camera using shotwell, gthumb, fspot or whatever, keeps the app in question open and the app in question keeps the gphoto2 device handle open.
User wants to do some skyping with video chat, skype complains it cannot find the device, since the kernel driver currently is unbound.
-> Poor user experience.
Poor user experience, or merely poor user? The user ought to know better. Of course, I do agree that there are lots of such people, and it is a good idea to try to put up warning signs.
Quoted 7 lines
With having both functions in the kernel, the kernel could actually allow skype to use the dual mode cameras as video source, and if the user then were to switch to f-spot and try to import more photo's then he will get an -ebusy in f-spot. If he finishes skyping and then returns to f-spot everything will just continue working.
Quoted 5 lines
This is the kind of "seamless" user experience I'm aiming for here.
Regards,
Hans
Yes, I can see where you are coming from. But if the camera really will not let you run skype and fspot at the same time, which I do not believe it would allow on _any_ operating system, then each app should give an error message which says it cannot be run unless and until the other app has been closed. If that has to happen at the kernel level, then OK.
Theodore Kilgore
Re: Improving kernel -> userspace (usbfs) usb device hand off
Quoted 18 lines · Theodore Kilgore
Quoted 2 lines · Hans de Goede
Actually libusb and libgphoto have been using the rebind orginal driver functionality of the code for quite a while now,
Oh? I can see that libusb is doing that, and I can also see that there is a "public" function for _unbinding_ a kernel driver, namely
int
usb_detach_kernel_driver_np()found in
usb.hand it is used in libgphoto, as well.
I am not sure that there is any corresponding rebind function which is public. Is it perhaps
int
usb_get_driver_np()
Quoted 7 lines
???
By context (looking at
libgphoto2-port/usb/libusb.c) I would think that this function is not the rebind function, but is only checking whether or not there is any potential conflict with a kernel driver. If I am right, then where is the publicly exported rebind function, and where does it currently get used in libgphoto2?
The rebind happened under the function "static int gp_port_usb_close (GPPort *port)".
Since libgphoto2 is still using libusb-0.1, the unbind is using usbfs IOCTL directly (USBDEVFS_CONNECT).
Quoted 6 lines
So frankly after my eagerness yesterday I do not see how it can easily be made to work, after all.
Quoted 3 lines
unfortunately this
does not solve the problem, unless we somehow move to 1 central coordinator for the device the user experience will stay subpar.
Now I understand what Hans is saying. It will be a lot of work trying to sort out this issue in userspace. What can be the single central coordinator? A device manager applet listing the program or service which hold the device?
Quoted 4 lines
Quoted 4 lines
Example, user downloads pictures from the camera using shotwell, gthumb, fspot or whatever, keeps the app in question open and the app in question keeps the gphoto2 device handle open.
Quoted 8 lines
Quoted 4 lines
User wants to do some skyping with video chat, skype complains it cannot find the device, since the kernel driver currently is unbound.
-> Poor user experience.
Poor user experience, or merely poor user? The user ought to know better. Of course, I do agree that there are lots of such people, and it is a good idea to try to put up warning signs.
It is difficult to call the users "poor users" in this case. Since they may not know that the other open program is holding the device. Some warning message may help, not "I can not find the device" though. It would be better to pinpoint which program is holding the device and then ask the user to close that program. I understand this is easily said than done...
Similar experiences for Windows about the serial port, sometimes it is difficult for the user to know that some program or service are holding the serial port so that the other program or will fail or Windows complain that it is still open when you want to undock the computer.
Quoted 7 lines
Quoted 7 lines
With having both functions in the kernel, the kernel could actually allow skype to use the dual mode cameras as video source, and if the user then were to switch to f-spot and try to import more photo's then he will get an -ebusy in f-spot. If he finishes skyping and then returns to f-spot everything will just continue working.
Quoted 8 lines
Quoted 2 lines
This is the kind of "seamless" user experience I'm aiming for here.
Yes, I can see where you are coming from. But if the camera really will not let you run skype and fspot at the same time, which I do not believe it would allow on _any_ operating system, then each app should give an error message which says it cannot be run unless and until the other app has been closed. If that has to happen at the kernel level, then OK.
Yes. From what I read, to solve it in kernel or to solve it in user space are both a lot of work.
Personally I tend to think to solve it in user space is more feasible.
Signature · 1 line
Xiaofan
Re: [Libusb-devel] Improving kernel -> userspace (usbfs) usb device hand off
Quoted 38 lines · Xiaofan Chen
Quoted 18 lines · Theodore Kilgore
Quoted 2 lines · Hans de Goede
Actually libusb and libgphoto have been using the rebind orginal driver functionality of the code for quite a while now,
Oh? I can see that libusb is doing that, and I can also see that there is a "public" function for _unbinding_ a kernel driver, namely
int
usb_detach_kernel_driver_np()found in
usb.hand it is used in libgphoto, as well.
I am not sure that there is any corresponding rebind function which is public. Is it perhaps
int
usb_get_driver_np()Quoted 7 lines
???
By context (looking at
libgphoto2-port/usb/libusb.c) I would think that this function is not the rebind function, but is only checking whether or not there is any potential conflict with a kernel driver. If I am right, then where is the publicly exported rebind function, and where does it currently get used in libgphoto2?The rebind happened under the function "static int
gp_port_usb_close(GPPort *port)".
Since libgphoto2 is still using libusb-0.1, the unbind is using usbfs IOCTL directly (USBDEVFS_CONNECT).Quoted 3 lines
So frankly after my eagerness yesterday I do not see how it can easily be made to work, after all.
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
unfortunately this
does not solve the problem, unless we somehow move to 1 central coordinator for the device the user experience will stay subpar.Now I understand what Hans is saying. It will be a lot of work trying to sort out this issue in userspace. What can be the single central coordinator? A device manager applet listing the program or service which hold the device?
Something like that. Have a user-space device allocation mechanism so that apps are not constrained by a particular interface semantic (i.e. not required to open /dev/video and have a kernel driver deliver pixels to the apps).
Or hid that behind a library API and let instances of the library coordinate with each other; as long as an app uses the documented public library API to access a webcam, then everyone will play fair with each other.
Look at the mess that this userspace/kernel driver issue has for code in an application like gphoto - that's insane that a photo manipulation app needs to know anything about userspace/kernel switching!
Quoted 42 lines
Quoted 4 lines
Quoted 4 lines
Example, user downloads pictures from the camera using shotwell, gthumb, fspot or whatever, keeps the app in question open and the app in question keeps the gphoto2 device handle open.
Quoted 8 lines
Quoted 4 lines
User wants to do some skyping with video chat, skype complains it cannot find the device, since the kernel driver currently is unbound.
-> Poor user experience.
Poor user experience, or merely poor user? The user ought to know better. Of course, I do agree that there are lots of such people, and it is a good idea to try to put up warning signs.
It is difficult to call the users "poor users" in this case. Since they may not know that the other open program is holding the device. Some warning message may help, not "I can not find the device" though. It would be better to pinpoint which program is holding the device and then ask the user to close that program. I understand this is easily said than done...
Similar experiences for Windows about the serial port, sometimes it is difficult for the user to know that some program or service are holding the serial port so that the other program or will fail or Windows complain that it is still open when you want to undock the computer.
Quoted 7 lines
Quoted 7 lines
With having both functions in the kernel, the kernel could actually allow skype to use the dual mode cameras as video source, and if the user then were to switch to f-spot and try to import more photo's then he will get an -ebusy in f-spot. If he finishes skyping and then returns to f-spot everything will just continue working.
Quoted 8 lines
Quoted 2 lines
This is the kind of "seamless" user experience I'm aiming for here.
Yes, I can see where you are coming from. But if the camera really will not let you run skype and fspot at the same time, which I do not believe it would allow on _any_ operating system, then each app should give an error message which says it cannot be run unless and until the other app has been closed. If that has to happen at the kernel level, then OK.
Quoted 2 lines
Yes. From what I read, to solve it in kernel or to solve it in user space are both a lot of work.
Yes and a kernel-based solution locks you into a kernel-based webcam driver paradigm, or an even uglier loopback driver.
Quoted 1 line
Personally I tend to think to solve it in user space is more feasible.
I agree.
mike
Re: Improving kernel -> userspace (usbfs) usb device hand off
In reply to Hans de GoedeQuoted 14 lines · Hans de Goede
Quoted 7 lines
Quoted 6 lines
So what do we need to make this situation better:
1) Ausb_drivercallback alternative to the disconnect callback, I propose to call thissoft_disconnect. This serves 2 purposes
a) It will allow the driver to tell the caller that that is not a good idea by returning an error code (think usb mass storage driver and mounted filesystemQuoted 6 lines
Not feasible. usb-storage has no idea whether or not a device it controls has a mounted filesystem. (All it does is send SCSI commands to a device and get back the results.) Since that's the main use case you're interested in, this part of the proposal seems destined to fail.
Quoted 6 lines
This is not completely true, I cannot rmmod usb-storage as long as disks using it are mounted. I know this is done through the global module usage count, so this is not per usb-storage device. But extending the ref counting to be per usb-storage device should not be hard.
All the accounting is already done for this.
It would be harder than you think. All the accounting is _not_ already being done. What you're talking about would amount to a significant change in the driver model core and the SCSI core. It isn't just a USB thing.
Quoted 8 lines
Quoted 7 lines
But userspace _does_ know where the mounted filesystems are. Therefore userspace should be responsible for avoiding programs that want to take control of devices holding these filesystems. That's the reason why usbfs device nodes are owned by root and have 0644 mode; there're can be written to only by programs with superuser privileges -- and such programs are supposed to be careful about what they do.
Quoted 5 lines
Yes, and what I'm asking for is for an easy way for these programs to be careful. A way for them to ask the kernel, which in general is responsible for things like this and traditionally does resource management and things which come with that like refcounting: "unbind the driver from this device unless the device is currently in use".
Sure. At the moment the kernel does not keep track of whether a device is currently in use -- at least, not in the way you mean.
I'm not saying this can't be done. But it would be a bigger job than you think, and this isn't the appropriate thread to discuss it.
Alan Stern
Re: Improving kernel -> userspace (usbfs) usb device hand off
In reply to Hans de GoedeQuoted 5 lines · Hans de Goede
Hi all,
The current API for managing kernel -> userspace is a bit rough around the edges, so I would like to discuss extending the API.
[...]
Quoted 16 lines
2) So called dual mode cameras are (cheap) stillcams often even without an lcdscreen viewfinder, and battery backed sram instead of flash, which double as a webcam. We have drivers for both the stillcam function of these (in libgphoto2, so using usbfs) as well as for the webcam function (v4l2 kernel drivers).
These drivers work well, and are mature. Yet the user experience is rather poor. Under gnome the still-cam contents will be automatically be made available as a "drive" using a gvfs-gphoto2 fuse mount. This however involves sending a disconnect to the v4l2 kernel driver, and thus the /dev/video# node disappearing. So if a user wants to use the device as a webcam he/she needs to first go to nautilus and unmount the gvfs mount. Until that is done the user will simply get a message from an app like cheese that he has no webcam, not even an ebusy error, just that he has no such device.
Quoted 1 line
Again not good.
[...]
As I have been involved in writing the drivers (both the kernel and the libgphoto2 drivers) for many of the affected cameras, perhaps I should expand on this problem. There are lots of responses to this original message of Hans. I will try to take some of their comments into account, below. First, some background.
1. All of the cameras in question have only one USB Vendor:Product number. In this sense, they are not "good citizens" which have different Product numbers for the two distinct modes of functioning. Thus, the problems are from that standpoint unavoidable.
2. Until recently in the history of Linux, there was an irreconcilable conflict. If a kernel driver for the video streaming mode was present and installed, it was not possible to use the camera in stillcam mode at all. Thus the only solution to the problem was to blacklist the kernel module so that it would not get loaded automatically and only to install said module by hand if the camera were to be used in video streaming mode, then to rmmod it immediately afterwards. Very cumbersome, obviously.
3. The current state of affairs is an advance on (2) but it is still inelegant. What happens now is, libusb has been revised in such a way that the kernel module is disabled (though still present) if a userspace driver (libgphoto2) wants to access the device (the camera). If it is desired to do video streaming after that, the camera needs to be re-plugged, which then causes the module to be automatically re-loaded.
4. Hans is absolutely correct about the problem with certain Gnome apps (and certain distros which blindly use them), which load up the libgphoto2 driver for the camera as soon as it is detected. The consequence (cf. item
3) is that the camera can never be used as a webcam. The only solution to that problem is to disable the automatic loading of the libgphoto2 driver.
5. It could be said that those who came up with the "user-friendly" "solution" described in (4) were not very clever, and perhaps they ought to fix their own mess. I would strongly agree that they ought to have thought before coding, as the result is not user-friendly in the least.
6. The question has been asked whether the cameras are always using the same interface. Typically, yes. The same altsetting? That depends on the camera. Some of them use isochronous transport for streaming, and some of them rely exclusively upon bulk transport. Those which use bulk transport only are confined to altsetting 0.
Some possible solutions?
Well, first of all it needs to be understood that the problem originates as a bad feature of something good, namely the rigid separation of kernelspace and userspace functionality which is an integral part of the Linux security model. Some other operating systems are not so careful, and thus they do not have a problem about supporting dual-mode hardware.
Second, it appears to me that the problem is most appropriately addressed from the userspace side and perhaps also from the kernelspace side.
In userspace, it would be a really good idea if those who are attempting to write user-friendly apps and to create user-friendly distros would actually learn some of the basics of Linux, such as the rudiments of the security model. Since dual-mode cameras are known to exist, it would have been appropriate, when one is detected, to pop up a dialog window which asks the user to choose a webcam app or a stillcam app. Even this minor innovation would stop a lot of user grief. Frankly, I am mystified that this was not done.
This still leaves the problem (see item 3, above) that a dual-mode camera needs to be re-plugged in order to re-enable the access to /dev/video* if it has been used in stillcam mode. If it is possible to do a re-plug in software, this would help a lot. It does not matter if the re-plug is done in userspace or in kernelspace, so long as it can be done, somehow, after libusb relinquishes the camera. Or, fix things up somehow so that the kernel module automatically re-activates itself when the userspace app (using libgphoto2) is closed down.
Finally, another possible alternative would be to figure out how to do something in the kernel module which permits the camera to be accessed by libusb.
If there is going to be discussion about this problem in Vancouver, I am really sorry that I will probably be able to attend. Classes for the Fall semester begin at Auburn University on August 17. Nevertheless, this is a problem in which I have been interested for a long time.
Theodore Kilgore
Re: Improving kernel -> userspace (usbfs) usb device hand off
Hi,
Quoted 8 lines · Theodore Kilgore
As I have been involved in writing the drivers (both the kernel and the libgphoto2 drivers) for many of the affected cameras, perhaps I should expand on this problem. There are lots of responses to this original message of Hans. I will try to take some of their comments into account, below. First, some background.
1. All of the cameras in question have only one USB Vendor:Product number. In this sense, they are not "good citizens" which have different Product
there's nothing in the USB spec that says you need different product IDs for different modes of operation. No matter if it's still or webcam configuration, the underlying function is the same: capture images using a set of lenses and image sensor.
Quoted 2 lines
numbers for the two distinct modes of functioning. Thus, the problems are from that standpoint unavoidable.
I don't see any problems in this situation. If, for that particular product, webcam and still image functionality are mutually exclusive, then that's how the product (and their drivers) have to work.
If the linux community decided to put webcam functionality in kernel and still image functionality on a completely separate driver, that's entirely our problem.
Quoted 7 lines
2. Until recently in the history of Linux, there was an irreconcilable conflict. If a kernel driver for the video streaming mode was present and installed, it was not possible to use the camera in stillcam mode at all. Thus the only solution to the problem was to blacklist the kernel module so that it would not get loaded automatically and only to install said module by hand if the camera were to be used in video streaming mode, then to rmmod it immediately afterwards. Very cumbersome, obviously.
true... but why couldn't we combine both in kernel or in userspace altogether ? Why do we have this split ? (words from a newbie in V4L2, go easy :-p)
Quoted 6 lines
3. The current state of affairs is an advance on (2) but it is still inelegant. What happens now is, libusb has been revised in such a way that the kernel module is disabled (though still present) if a userspace driver (libgphoto2) wants to access the device (the camera). If it is desired to do video streaming after that, the camera needs to be re-plugged, which then causes the module to be automatically re-loaded.
It's still wrong. This should be just another USB_REQ_SET_CONFIGURATION(). If this is was just one single driver, you could easily do that.
Quoted 5 lines
4. Hans is absolutely correct about the problem with certain Gnome apps (and certain distros which blindly use them), which load up the libgphoto2 driver for the camera as soon as it is detected. The consequence (cf. item
3) is that the camera can never be used as a webcam. The only solution to that problem is to disable the automatic loading of the libgphoto2 driver.
Or, to move the libgphoto2 driver to kernel, combine it in the same driver that handles streaming. No ?
Quoted 4 lines
5. It could be said that those who came up with the "user-friendly" "solution" described in (4) were not very clever, and perhaps they ought to fix their own mess. I would strongly agree that they ought to have thought before coding, as the result is not user-friendly in the least.
I agree here
Quoted 5 lines
6. The question has been asked whether the cameras are always using the same interface. Typically, yes. The same altsetting? That depends on the camera. Some of them use isochronous transport for streaming, and some of them rely exclusively upon bulk transport. Those which use bulk transport only are confined to altsetting 0.
the transfer type or the way the configurations are setup shouldn't matter much for the end functionality.
Quoted 7 lines
Some possible solutions?
Well, first of all it needs to be understood that the problem originates as a bad feature of something good, namely the rigid separation of kernelspace and userspace functionality which is an integral part of the Linux security model. Some other operating systems are not so careful, and thus they do not have a problem about supporting dual-mode hardware.
You'd need to describe this fully. What's the problem in having one driver handle all modes of operation of that particular camera ? Sounds like a decision from V4L2 folks not to take still image cameras in kernel. Am I wrong ?
Quoted 11 lines
Second, it appears to me that the problem is most appropriately addressed from the userspace side and perhaps also from the kernelspace side.
In userspace, it would be a really good idea if those who are attempting to write user-friendly apps and to create user-friendly distros would actually learn some of the basics of Linux, such as the rudiments of the security model. Since dual-mode cameras are known to exist, it would have been appropriate, when one is detected, to pop up a dialog window which asks the user to choose a webcam app or a stillcam app. Even this minor innovation would stop a lot of user grief. Frankly, I am mystified that this was not done.
I think this is still not good. It should be more "fluid". If the camera has separate alternate settings or different configurations, all it takes (from a USB standpoint) is to send the correct SetConfiguration or SetInterface command and the camera should happily change its mode of operation.
Which means that a simple ioctl() to change the mode of operation would suffice should the driver be combined.
Quoted 9 lines
This still leaves the problem (see item 3, above) that a dual-mode camera needs to be re-plugged in order to re-enable the access to
/dev/video*if it has been used in stillcam mode. If it is possible to do a re-plug in software, this would help a lot. It does not matter if the re-plug is done in userspace or in kernelspace, so long as it can be done, somehow, after libusb relinquishes the camera. Or, fix things up somehow so that the kernel module automatically re-activates itself when the userspace app (using libgphoto2) is closed down.
Quoted 3 lines
Finally, another possible alternative would be to figure out how to do something in the kernel module which permits the camera to be accessed by libusb.
I don't see the possibility of combining both drivers and use /dev/videoX for streaming and something like /dev/camX for still image. Now, the only thing you have to be sure in kernel space is that you don't allow both to be accessed together, but there are ways to handle it.
Signature · 1 line
balbi
Re: Improving kernel -> userspace (usbfs) usb device hand off
Quoted 7 lines · Felipe Balbi
I don't see any problems in this situation. If, for that particular product, webcam and still image functionality are mutually exclusive, then that's how the product (and their drivers) have to work.
If the linux community decided to put webcam functionality in kernel and still image functionality on a completely separate driver, that's entirely our problem.
And the problem is how to coordinate the two of them.
Quoted 11 lines
Quoted 7 lines
2. Until recently in the history of Linux, there was an irreconcilable conflict. If a kernel driver for the video streaming mode was present and installed, it was not possible to use the camera in stillcam mode at all. Thus the only solution to the problem was to blacklist the kernel module so that it would not get loaded automatically and only to install said module by hand if the camera were to be used in video streaming mode, then to rmmod it immediately afterwards. Very cumbersome, obviously.
true... but why couldn't we combine both in kernel or in userspace altogether ? Why do we have this split ? (words from a newbie in V4L2, go easy :-p)
I think the problem may be that the PTP protocol used in the still-cam mode isn't suitable for a kernel driver. Or if it is suitable, it would have to be something like a shared-filesystem driver -- nothing like a video driver. You certainly wouldn't want to put it in V4L2.
Quoted 2 lines
Or, to move the libgphoto2 driver to kernel, combine it in the same driver that handles streaming. No ?
No. Something else is needed.
Alan Stern
Re: Improving kernel -> userspace (usbfs) usb device hand off
Hi,
Quoted 8 lines · Alan Stern
Quoted 8 lines
Quoted 7 lines
2. Until recently in the history of Linux, there was an irreconcilable conflict. If a kernel driver for the video streaming mode was present and installed, it was not possible to use the camera in stillcam mode at all. Thus the only solution to the problem was to blacklist the kernel module so that it would not get loaded automatically and only to install said module by hand if the camera were to be used in video streaming mode, then to rmmod it immediately afterwards. Very cumbersome, obviously.
Quoted 8 lines
Quoted 3 lines
true... but why couldn't we combine both in kernel or in userspace altogether ? Why do we have this split ? (words from a newbie in V4L2, go easy :-p)
I think the problem may be that the PTP protocol used in the still-cam mode isn't suitable for a kernel driver. Or if it is suitable, it would have to be something like a shared-filesystem driver -- nothing like a video driver. You certainly wouldn't want to put it in V4L2.
ach true.. Had forgotten that detail. Needs more thinking.
Signature · 1 line
balbi
Re: Improving kernel -> userspace (usbfs) usb device hand off
In reply to Alan SternQuoted 29 lines · Alan Stern
Quoted 7 lines · Felipe Balbi
I don't see any problems in this situation. If, for that particular product, webcam and still image functionality are mutually exclusive, then that's how the product (and their drivers) have to work.
If the linux community decided to put webcam functionality in kernel and still image functionality on a completely separate driver, that's entirely our problem.
And the problem is how to coordinate the two of them.
Quoted 11 lines
Quoted 7 lines
2. Until recently in the history of Linux, there was an irreconcilable conflict. If a kernel driver for the video streaming mode was present and installed, it was not possible to use the camera in stillcam mode at all. Thus the only solution to the problem was to blacklist the kernel module so that it would not get loaded automatically and only to install said module by hand if the camera were to be used in video streaming mode, then to rmmod it immediately afterwards. Very cumbersome, obviously.
true... but why couldn't we combine both in kernel or in userspace altogether ? Why do we have this split ? (words from a newbie in V4L2, go easy :-p)
I think the problem may be that the PTP protocol used in the still-cam mode isn't suitable for a kernel driver. Or if it is suitable, it would have to be something like a shared-filesystem driver -- nothing like a video driver. You certainly wouldn't want to put it in V4L2.
Quoted 7 lines
Quoted 2 lines
Or, to move the libgphoto2 driver to kernel, combine it in the same driver that handles streaming. No ?
No. Something else is needed.
Alan Stern
Agreed. Something else is needed. But what? Also, very good point about PTP.
Theodore Kilgore
Re: Improving kernel -> userspace (usbfs) usb device hand off
In reply to Felipe BalbiQuoted 9 lines · Felipe Balbi
Hi,
Quoted 6 lines · Theodore Kilgore
As I have been involved in writing the drivers (both the kernel and the libgphoto2 drivers) for many of the affected cameras, perhaps I should expand on this problem. There are lots of responses to this original message of Hans. I will try to take some of their comments into account, below. First, some background.
Quoted 7 lines
Quoted 2 lines
1. All of the cameras in question have only one USB Vendor:Product number. In this sense, they are not "good citizens" which have different Product
there's nothing in the USB spec that says you need different product IDs for different modes of operation. No matter if it's still or webcam configuration, the underlying function is the same: capture images using a set of lenses and image sensor.
True, true. But I will add that most of these cameras are Class 255, Subclass 255, Protocol 255 (Proprietary, Proprietary, Proprietary).
Quoted 7 lines
Quoted 3 lines
numbers for the two distinct modes of functioning. Thus, the problems are from that standpoint unavoidable.
I don't see any problems in this situation. If, for that particular product, webcam and still image functionality are mutually exclusive, then that's how the product (and their drivers) have to work.
Yes.
Quoted 4 lines
If the linux community decided to put webcam functionality in kernel and still image functionality on a completely separate driver, that's entirely our problem.
As I understand, the basic reason why webcam functionality needs to be in the kernel is speed. Quick reaction time, and faster data transmission. Most but not all webcams use isochronous data transport. The ones which do not, cannot get as high a rate of frames per second. Isochronous data transport has historically seemed to need kernel support in order to work properly. Related to this, libusb has not supported isochronous data transport. But still cameras use bulk transport. This is natural because one needs to move data which is stored on the camera to a place on the computer where it can be processed (if needed) and stored. There is no impediment to userspace talking to a USB device, so why not go ahead and do the job that way if the kernel is not needed?
The second thing to mention is that libgphoto2 at this point is supporting well over 1,000 cameras. True, there are not 1,000 driver libraries, but there are quite a few. My impression is that nobody wants to put stuff like that in the kernel unless it is absolutely necessary, just because some of those cameras are dual-mode cameras. Don't people complain on a fairly regular basis about kernel code bloat?
Quoted 9 lines
Quoted 8 lines
2. Until recently in the history of Linux, there was an irreconcilable conflict. If a kernel driver for the video streaming mode was present and installed, it was not possible to use the camera in stillcam mode at all. Thus the only solution to the problem was to blacklist the kernel module so that it would not get loaded automatically and only to install said module by hand if the camera were to be used in video streaming mode, then to rmmod it immediately afterwards. Very cumbersome, obviously.
Quoted 3 lines
true... but why couldn't we combine both in kernel or in userspace altogether ? Why do we have this split ? (words from a newbie in V4L2, go easy :-p)
See above.
Quoted 10 lines
Quoted 6 lines
3. The current state of affairs is an advance on (2) but it is still inelegant. What happens now is, libusb has been revised in such a way that the kernel module is disabled (though still present) if a userspace driver (libgphoto2) wants to access the device (the camera). If it is desired to do video streaming after that, the camera needs to be re-plugged, which then causes the module to be automatically re-loaded.
It's still wrong. This should be just another
USB_REQ_SET_CONFIGURATION(). If this is was just one single driver, you could easily do that.
Well, I think it ought to be possible anyway. Up to that point, I agree. Though I confess I do not exactly know how.
Quoted 5 lines
Quoted 5 lines
4. Hans is absolutely correct about the problem with certain Gnome apps (and certain distros which blindly use them), which load up the libgphoto2 driver for the camera as soon as it is detected. The consequence (cf. item
3) is that the camera can never be used as a webcam. The only solution to that problem is to disable the automatic loading of the libgphoto2 driver.
I would only add to this, that it is not pleasant to have to deal with frustrated users. At this point, my usual advice is that this is a distro problem. Ask your distro for help.
Quoted 3 lines
Or, to move the libgphoto2 driver to kernel, combine it in the same driver that handles streaming. No ?
See above.
Quoted 7 lines
Quoted 5 lines
5. It could be said that those who came up with the "user-friendly" "solution" described in (4) were not very clever, and perhaps they ought to fix their own mess. I would strongly agree that they ought to have thought before coding, as the result is not user-friendly in the least.
I agree here
Excellent. I really did have to restrain myself when writing that, too.
Quoted 12 lines
Quoted 6 lines
6. The question has been asked whether the cameras are always using the same interface. Typically, yes. The same altsetting? That depends on the camera. Some of them use isochronous transport for streaming, and some of them rely exclusively upon bulk transport. Those which use bulk transport only are confined to altsetting 0.
the transfer type or the way the configurations are setup shouldn't matter much for the end functionality.
Quoted 2 lines
Some possible solutions?
Quoted 7 lines
Quoted 5 lines
Well, first of all it needs to be understood that the problem originates as a bad feature of something good, namely the rigid separation of kernelspace and userspace functionality which is an integral part of the Linux security model. Some other operating systems are not so careful, and thus they do not have a problem about supporting dual-mode hardware.
You'd need to describe this fully.
Certain other operating systems are not fussy at all about their device drivers. Those operating systems are also notorious for bad security and for crashing, and one of the things which gets blamed for that is all those hardware drivers from the hardware manufacturers. Linux tries to do things right, which clearly can also cause problems.
What's the problem in having one
Quoted 3 lines
driver handle all modes of operation of that particular camera ? Sounds like a decision from V4L2 folks not to take still image cameras in kernel. Am I wrong ?
Still image cameras can be handled perfectly well in userspace. There is no need for kernel support, at all. Moreover, many still cameras will not stream at all; they simply were not designed, built, or sold to do that. As far as V4L2 was concerned, it was developed in order to meet the need for supporting webcams. Clearly, that was a conscious decision. But you ask whether it is possible to support a dual-mode camera entirely in V4L2. I would say that of course it ought to be possible, but that would require a large portion of the functionality of Gphoto to be duplicated in V4L2, just to support those dual-mode cameras. Having some experience with both Gphoto and V4L2, I have to say that I am not very taken with the idea. There has to be a way which is not so arduous, one would hope.
Quoted 13 lines
Quoted 4 lines
Second, it appears to me that the problem is most appropriately addressed from the userspace side and perhaps also from the kernelspace side.
Quoted 8 lines
In userspace, it would be a really good idea if those who are attempting to write user-friendly apps and to create user-friendly distros would actually learn some of the basics of Linux, such as the rudiments of the security model. Since dual-mode cameras are known to exist, it would have been appropriate, when one is detected, to pop up a dialog window which asks the user to choose a webcam app or a stillcam app. Even this minor innovation would stop a lot of user grief. Frankly, I am mystified that this was not done.
Quoted 5 lines
I think this is still not good. It should be more "fluid". If the camera has separate alternate settings or different configurations, all it takes (from a USB standpoint) is to send the correct SetConfiguration or SetInterface command and the camera should happily change its mode of operation.
Often true, but not necessarily.
Quoted 17 lines
Which means that a simple
ioctl()to change the mode of operation would suffice should the driver be combined.Quoted 9 lines
This still leaves the problem (see item 3, above) that a dual-mode camera needs to be re-plugged in order to re-enable the access to
/dev/video*if it has been used in stillcam mode. If it is possible to do a re-plug in software, this would help a lot. It does not matter if the re-plug is done in userspace or in kernelspace, so long as it can be done, somehow, after libusb relinquishes the camera. Or, fix things up somehow so that the kernel module automatically re-activates itself when the userspace app (using libgphoto2) is closed down.Quoted 3 lines
Finally, another possible alternative would be to figure out how to do something in the kernel module which permits the camera to be accessed by libusb.
Quoted 5 lines
I don't see the possibility of combining both drivers and use
/dev/videoXfor streaming and something like/dev/camXfor still image. Now, the only thing you have to be sure in kernel space is that you don't allow both to be accessed together, but there are ways to handle it.
Which leaves a problem for still cameras. Most of them, in fact, do not need any such /dev/camX as you are proposing, at all. The only cameras which need the kernel are those which need /dev/videoX. For pure still cameras, /dev/videoX is entirely superfluous, at best. Dual-mode cameras only need /dev/videoX when streaming and otherwise need for /dev/videoX to go away.
I do not believe that we have found the optimal solution, yet. The ideal thing would be some kind of hack which allows the kernel to be used when it is needed, and when it is not needed it does not interfere.
Theodore Kilgore
Re: Improving kernel -> userspace (usbfs) usb device hand off
Quoted 3 lines · Theodore Kilgore
I do not believe that we have found the optimal solution, yet. The ideal thing would be some kind of hack which allows the kernel to be used when it is needed, and when it is not needed it does not interfere.
Just wondering if you can use libusb-1.0 for the user space still image functionality.
libusb-1.0 offers the following functions to do that for you under Linux.
int libusb_kernel_driver_active (libusb_device_handle *dev, int interface)
Determine if a kernel driver is active on an interface.
int libusb_detach_kernel_driver (libusb_device_handle *dev, int interface)
Detach a kernel driver from an interface.
int libusb_attach_kernel_driver (libusb_device_handle *dev, int interface)
Re-attach an interface's kernel driver, which was previously
detached using libusb_detach_kernel_driver().
So you can detach the kernel v4l2 driver at the beginning and later re-attach it when you finish.
Signature · 1 line
Xiaofan
Re: Improving kernel -> userspace (usbfs) usb device hand off
Quoted 19 lines · Xiaofan Chen
Quoted 3 lines · Theodore Kilgore
I do not believe that we have found the optimal solution, yet. The ideal thing would be some kind of hack which allows the kernel to be used when it is needed, and when it is not needed it does not interfere.
Just wondering if you can use libusb-1.0 for the user space still image functionality.
libusb-1.0 offers the following functions to do that for you under Linux.
int
libusb_kernel_driver_active(libusb_device_handle*dev, int interface)
Determine if a kernel driver is active on an interface.
intlibusb_detach_kernel_driver(libusb_device_handle*dev, int interface)
Detach a kernel driver from an interface.
intlibusb_attach_kernel_driver(libusb_device_handle*dev, int interface)
Re-attach an interface's kernel driver, which was previously
detached usinglibusb_detach_kernel_driver().
Quoted 2 lines
So you can detach the kernel v4l2 driver at the beginning and later re-attach it when you finish.
Well, then, this solves the problem, doesn't it? Of course, those who deal with creating those "simple" and "user-friendly" GUI environments would probably still do well if they would open a dialog box for dual-mode hardware.
Theodore Kilgore
Re: Improving kernel -> userspace (usbfs) usb device hand off
In reply to Theodore KilgoreHi,
Quoted 7 lines · Theodore Kilgore
Quoted 4 lines
there's nothing in the USB spec that says you need different product IDs for different modes of operation. No matter if it's still or webcam configuration, the underlying function is the same: capture images using a set of lenses and image sensor.
True, true. But I will add that most of these cameras are Class 255, Subclass 255, Protocol 255 (Proprietary, Proprietary, Proprietary).
well, if the manufacturer doesn't want to implement UVC for whatever reason, it's his call ;-)
Signature · 1 line
balbi
Re: Improving kernel -> userspace (usbfs) usb device hand off
Em 13-06-2011 06:05, Felipe Balbi escreveu:
Quoted 8 lines
Hi,
Quoted 5 lines · Theodore Kilgore
Quoted 4 lines
there's nothing in the USB spec that says you need different product IDs for different modes of operation. No matter if it's still or webcam configuration, the underlying function is the same: capture images using a set of lenses and image sensor.
Quoted 5 lines
Quoted 2 lines
True, true. But I will add that most of these cameras are Class 255, Subclass 255, Protocol 255 (Proprietary, Proprietary, Proprietary).
well, if the manufacturer doesn't want to implement UVC for whatever reason, it's his call ;-)
This argument is bogus.
UVC were implemented too late. There are lots of chipsets that are not UVC-compliant, simply because there were no UVC at the time those chipsets were designed.
Still today, newer devices using those chipsets are still at the market.
This is the same as saying that we should not support USB 1.1 or USB 2.0 because they're not fully USB 3.0 compliant.
Mauro.
Re: Improving kernel -> userspace (usbfs) usb device hand off
Hi,
Quoted 17 lines · Mauro Carvalho Chehab
Em 13-06-2011 06:05, Felipe Balbi escreveu:
Quoted 8 lines
Hi,
Quoted 5 lines · Theodore Kilgore
Quoted 4 lines
there's nothing in the USB spec that says you need different product IDs for different modes of operation. No matter if it's still or webcam configuration, the underlying function is the same: capture images using a set of lenses and image sensor.
Quoted 5 lines
Quoted 2 lines
True, true. But I will add that most of these cameras are Class 255, Subclass 255, Protocol 255 (Proprietary, Proprietary, Proprietary).
well, if the manufacturer doesn't want to implement UVC for whatever reason, it's his call ;-)
This argument is bogus.
Quoted 7 lines
UVC were implemented too late. There are lots of chipsets that are not UVC-compliant, simply because there were no UVC at the time those chipsets were designed.
Still today, newer devices using those chipsets are still at the market.
This is the same as saying that we should not support USB 1.1 or USB 2.0 because they're not fully USB 3.0 compliant.
I would think the small wink at the end was enough to label the reply as a joke. Apparently not :-)
Signature · 1 line
balbi
Hi Sarah/Greg,
Em 09-06-2011 21:21, Sarah Sharp escreveu:
Quoted 57 lines
I'm pleased to announce a USB mini-summit at LinuxCon Vancouver.
What: USB mini-summit
When: Tuesday, August 16th, all day
Where: At the conference venue, room TBD pending confirmation from
Angela Brown.Proposed topics include USB virtualization, and improved bandwidth APIs between the USB core and drivers (especially webcam drivers). See the detailed topic list below. Anyone is also welcome to propose or show up with a USB related topic. MUSB? USB 3.0 gadget drivers? USB-IP?
The USB mini-summit does overlap with the virtualization mini-summit by a day, but I'm hoping we can schedule talks so some of the virtualization folks can make it to the USB mini-summit. The other option was on Friday during the conference which was not ideal.
Proposed topics:
Topic 1
-------The KVM folks suggested that it would be good to get USB and virtualization developers together to talk about how to virtualize the xHCI host controller. The xHCI spec architect worked closely with VMWare to get some extra goodies in the spec to help virtualization, and I'd like to see the other virtualization developers take advantage of that. I'd also like us to hash out any issues they have been finding in the USB core or xHCI driver during the virtualization effort.
Topic 2
-------I'd also like to get the V4L and audio developers who work with USB devices together with the core USB folks to talk about bandwidth management under xHCI.
One of the issues is that since the xHCI hardware does bandwidth management, not the xHCI driver, a schedule that will take too much bandwidth will get rejected much sooner than any USB driver currently expects (during a call to
usb_set_interface). This poses issues, since most USB video drivers negotiate the video size and frame rate after they callusb_set_interface, so they don't know whether they can fall back to a less bandwidth-intensive setting. Currently, they just submit URBs with less and less bandwidth until one interval setting gets accepted that won't work under xHCI.A second issue is that that some drivers need less bandwidth than the device advertises, and the xHCI driver currently uses whatever periodic interval the device advertises in its descriptors. This is not what the video/audio driver wants, especially in the case of buggy high speed devices that advertise the interval in frames, not microframes. There needs to be some way for the drivers to communicate their bandwidth needs to the USB core. We've known about this issue for a while, and I think it's time to get everyone in the same room and hash out an API.
Quoted 1 line
(I will send out an API proposal later this month.)
While discussing the topics for the media workshop that will happen together with this year's KS/2011, one issue related to the USB stack came on our
discussions: it is related to multi-function USB devices, as you can see on this thread:
http://www.spinics.net/lists/linux-media/msg36195.html
We have several cases of multi-function devices at the media subsystem. For example, most TV grabber devices provides Remote Controller, Video, audio and MPEG streaming capabilities, either implementing the standard USB API's, or the vendor class API's. There are even some devices that support USB storage, 3G modem and Digital TV.
Some of those devices have some resources that are mutually exclusive.
For example, some Digital Cameras can provide either access to the stored images, or can be used as webcams. Using them as webcams automatically deletes all stored images from it. So, a properly implemented driver should be returning -EBUSY (or -EPERM?) if someone tries to stream for such devices if is there any pictures stored on it. Symmetrically, accessing the stored pictures there should return -EBUSY if the device is streaming.
Currently, the data access for Digital Cameras is implemented via libusb, libgphoto and gvfs, while the streaming interface is implemented via a gspca Kernel driver.
I know that this problem were somewhat solved for 3G modems, with the usage of the userspace problem usb_modeswitch, and with some quirks for the USB storage driver, but I'm not sure if such tricks will scale forever, as more functions are seen on some USB devices.
So, if we have some time, maybe we could start some discussions about that during the USB mini-summit.
It should be noticed that the media subsystem has currently similar problems on devices that provide both analog and digital TV, as some resources, like the tuner can't be used simultaneously by the two API's. This happens even with PCI devices. So, maybe, in the end, we'll be writing some kernel library for resource locking, but it would be good if we can have preliminary discussions there, and let the final discussions to happen during KS/2011.
What do you think?
Thanks!
Mauro
Quoted 4 lines · Mauro Carvalho Chehab
I know that this problem were somewhat solved for 3G modems, with the usage of the userspace problem
usb_modeswitch, and with some quirks for the USB storage driver, but I'm not sure if such tricks will scale forever, as more functions are seen on some USB devices.
Well, no matter how it "scales" it needs to be done in userspace, like usb_modeswitch does. We made that decision a while ago, and it is working out very well. I see no reason why you can't do it in userspace as well as that is the easiest place to control this type of thing.
I thought we had a long discussion about this topic a while ago and came to this very conclusion. Or am I mistaken?
thanks,
greg k-h
A message that is not in the archive (sent off-list or not yet received): CAA6KcBBZv7bvVxvEWOYL83igpNZHyzh=bcGxh6Dr5aKsvJK5Cg@mail.gmail.com
Em 04-08-2011 20:22, Matthew Dharm escreveu:
Quoted 13 lines · On Thu, Aug 4, 2011 at 3:56 PM, Greg KH <[email protected]…
On Thu, Aug 04, 2011 at 07:21:47PM -0300, Mauro Carvalho Chehab wrote:
> I know that this problem were somewhat solved for 3G modems, with the usage > of the userspace problemusb_modeswitch, and with some quirks for the USB > storage driver, but I'm not sure if such tricks will scale forever, as more > functions are seen on some USB devices.Well, no matter how it "scales" it needs to be done in userspace, like
usb_modeswitchdoes. We made that decision a while ago, and it is working out very well. I see no reason why you can't do it in userspace as well as that is the easiest place to control this type of thing.
Quoted 5 lines
I thought we had a long discussion about this topic a while ago and came to this very conclusion. Or am I mistaken?
We keep having the discussion over and over again. But, you are correct: the conclusion was that this all needs to live in userspace.
In the case of 3G modem x USB storage only, it is possible to handle it on userspace.
However, when there are more functions added, an they're not (completely) mutually exclusive, then I don't see an easy way (if is there a way) for doing it at userspace.
Several devices offer more than one function at the same time, but some resources are mutually exclusive. A TV stick with just one tuner, and both analog and digital demods offer both analog and digital TV at the same time. So, both analog and digital parts of the driver will offer the device to userspace. However, unpredictable results will happen if userspace tries to use both at the same time.
The Digital camera devices that offer PTP transfers and V4L support also fall at the same type of trouble. Some of those devices just delete all pictures from the memory, if streaming is started. So, receiving an automatic Skype video call may delete all pictures you took.
Worse than that, currently, the PTP protocol is handled via libusb, while streaming is done via V4L2 API.
The best technical approach, IMO, is to implement the PTP protocol in kernelspace, and do some sort of inter-subsystem locking to prevent such troubles.
Regards,
Mauro
Quoted 11 lines · Greg KH
Quoted 4 lines · Mauro Carvalho Chehab
I know that this problem were somewhat solved for 3G modems, with the usage of the userspace problem
usb_modeswitch, and with some quirks for the USB storage driver, but I'm not sure if such tricks will scale forever, as more functions are seen on some USB devices.Well, no matter how it "scales" it needs to be done in userspace, like
usb_modeswitchdoes. We made that decision a while ago, and it is working out very well. I see no reason why you can't do it in userspace as well as that is the easiest place to control this type of thing.
Quoted 7 lines
I thought we had a long discussion about this topic a while ago and came to this very conclusion. Or am I mistaken?
thanks,
greg k-h
Greg,
A little bit more of precision would help me a lot, here. Precisely what is one supposed to do in userspace? I would naturally assume that the user will control the device by announcing what the user wants to do with the device. Such control would therefore have to start in userspace. But this is such an obvious truism that you must mean something else. What, then?
To say to do things "like usb_modeswitch" is rather vague. For one thing, usb_modeswitch, being something out there in userspace, does _not_ affect what is in the kernel. However, what is (or is not) in the kernel might make it possible (or, in the alternative, impossible) to facilitate the action of some kind of userspace function-switching program similar to usb_modeswitch, not so?
As to the "long discussion about this topic a while ago" that may have been a discussion in which I was also involved. For, I do remember a discussion about this topic a few months ago. In one respect my memory differs from yours, however: It was not my impression that any definite conclusions were reached, certainly not a consensus. BTW, I did try to lay out what I remembered as three alternatives that came up, along with arguments for and against each of them, in my message which kicked off this thread today.
It is of course a real possibility that you have seen the perfect solution in your mind to these problems, which have vexed a lot of people for years, and others have not. If that is the way it is, then perhaps you just wish that we would all shut up, implement what is obvious, and we could all be happy. This could certainly be the case. You have written a lot more code than I have, and you obviously must have started doing that when you were at least 30 years younger, perhaps even 40 years younger, than I was when I started. So I would be the first to say that you are much better at it than I am. But there are others here, too, who seem actively to have been sucked into this discussion, and they are similarly younger and more clever than I am, too.
Giving due consideration to this, it would probably get the job done a lot faster if you just take the time to describe what grand vision you yourself have in mind for the solution of the problem, with a sufficient accounting of the details that people can all see what it is and exactly how and why it would work absolutely perfectly, and then we can just get busy and do it.
Theodore Kilgore
Am Freitag, 5. August 2011, 00:56:03 schrieb Greg KH:
Quoted 6 lines · Mauro Carvalho Chehab
Quoted 4 lines
I know that this problem were somewhat solved for 3G modems, with the usage of the userspace problem
usb_modeswitch, and with some quirks for the USB storage driver, but I'm not sure if such tricks will scale forever, as more functions are seen on some USB devices.
Quoted 7 lines
Well, no matter how it "scales" it needs to be done in userspace, like
usb_modeswitchdoes. We made that decision a while ago, and it is working out very well. I see no reason why you can't do it in userspace as well as that is the easiest place to control this type of thing.I thought we had a long discussion about this topic a while ago and came to this very conclusion. Or am I mistaken?
Circumstances change. We want to keep the stuff in user space as much and as long as we can. However user space has limitations:
- it has by necessity a race between resumption and access by others
- it cannot resume anything we run a (rw) filesystem over.
Furthermore, today PM actions that lead to a loss of mode are initiated by user space. If we ever want to oportunistically suspend a system we also need to restore mode from inside the kernel.
We could avoid all that trouble, if we persuaded vendors to use plain USB configurations for those purposes.
Regards
Oliver
Quoted 21 lines · Oliver Neukum
Am Freitag, 5. August 2011, 00:56:03 schrieb Greg KH:
Quoted 6 lines · Mauro Carvalho Chehab
Quoted 4 lines
I know that this problem were somewhat solved for 3G modems, with the usage of the userspace problem
usb_modeswitch, and with some quirks for the USB storage driver, but I'm not sure if such tricks will scale forever, as more functions are seen on some USB devices.Quoted 7 lines
Well, no matter how it "scales" it needs to be done in userspace, like
usb_modeswitchdoes. We made that decision a while ago, and it is working out very well. I see no reason why you can't do it in userspace as well as that is the easiest place to control this type of thing.I thought we had a long discussion about this topic a while ago and came to this very conclusion. Or am I mistaken?
Circumstances change. We want to keep the stuff in user space as much and as long as we can. However user space has limitations:
- it has by necessity a race between resumption and access by others
- it cannot resume anything we run a (rw) filesystem over.
Quoted 6 lines
Furthermore, today PM actions that lead to a loss of mode are initiated by user space. If we ever want to oportunistically suspend a system we also need to restore mode from inside the kernel.
We could avoid all that trouble, if we persuaded vendors to use plain USB configurations for those purposes.
But that would happen, I suspect, in an alternate universe. Better to anticipate the trouble, I suspect. :-{
Moreover, the spark for the current discussion was the problem of dual-mode cameras, which can work both as webcams and stillcams, not the 3G modems that you mention. The problems are analogous but not identical.
-- dual-mode cameras are, typically, Class Proprietary devices in
all of their functions. None of them that I know of are Mass Storage devices. Therefore, usb_modeswitch would have to be rewritten completely in order to be used for such hardware. As things stand right now, it has nothing at all to do with the problem. Not to say, of course, that the experience gained with usb_modeswitch is totally irrelevant.
-- I don't have one of those modems, but I have the impression
that the need to access the "disk" partition on one of them is basically a one-shot deal. Namely, one needs to load some firmware or so on the "disk" before the gadget can be used. The problem with a dual-mode camera is that the user ought to be able to switch at will between a download-the-pictures stillcam application and a stream application. While, of course, not being able to start one of these activities while the other is going on, because that would cause obvious trouble.
The similarity of the modem-and-mass-storage device and the web-and-still camera is, of course, that both partake of being devices that will do more than one kind of thing and they need to be supported in that respect. Other hardware exists with similar characteristics, but sometimes the functionaliy is not dual but even triple, and one can reasonably suspect that more of this kind of thing is going to come at us in the future. I think it is a good occasion to sit back and think things over a bit.
Theodore Kilgore
Hi,
Quoted 6 lines · Greg KH
Quoted 4 lines · Mauro Carvalho Chehab
I know that this problem were somewhat solved for 3G modems, with the usage of the userspace problem
usb_modeswitch, and with some quirks for the USB storage driver, but I'm not sure if such tricks will scale forever, as more functions are seen on some USB devices.
Quoted 7 lines
Well, no matter how it "scales" it needs to be done in userspace, like
usb_modeswitchdoes. We made that decision a while ago, and it is working out very well. I see no reason why you can't do it in userspace as well as that is the easiest place to control this type of thing.I thought we had a long discussion about this topic a while ago and came to this very conclusion. Or am I mistaken?
I think we've had multiple discussions about this, surrounding various topics / use cases. I would not call the do it in userspace a conclusion.
I rather more have a feeling of getting stonewalled on this by various people surrounding the usb system. Me shutting up on this is basically a case of:
"Discussion not getting anywhere and I don't have the time to do a kernel
proof of concept myself right now".
To be clear about the stonewalling I'm not talking about the dual cam issue here, nor about the morphing devices thing which usb_modeswitch fixes.
I think it is important to separate oranges from apples here, there are at least 3 different problem classes which all seem to have gotten thrown onto a pile here:
1) The reason Mauro suggested having some discussion on this at the USB summit is because of a discussion about dual mode cameras on the linux media list. Dual-mode cameras are (usually very cheap) digital photo cameras which can take still pictures in stand alone mode and store them in onboard memory, like regular digital photo cameras, this is one mode. The other mode is they can operate as a regular webcam, the hardware actually is more regular webcam hardware with some memory and a battery added (usually no viewfinder screen).
These devices typically use 1 usb interface (and thus 1 driver) for both modes. We currently have drivers for both modes, but they are separate drivers, which leads to fighting for device ownership, with the stillmode driver always winning as that is userspace and it simply kicks of the kernel webcam /streaming mode driver making it see a device unplug, even if an app was streaming from the device at the time. The solution here is simple, move both functions to 1 driver, which can then properly return -EBUSY when the device is active in one mode and an app tries to use it in the other mode.
This 1 driver will likely end up being a kernel driver, since doing streaming drivers in userspace is problematic, as the v4l2 API assumes a kernel device node, and we don't want to invent a new API, we want something which works with existing applications.
2) So called morphing devices. For example a USB 3G modem which initially shows up as a cdrom drive with windows software, only to reveal its real identity (with completely different usb descriptors) after some magic command. This is solved by the usb_modeswitch command, nothing to see here move along.
3) Re-direction of usb devices to virtual machines. This works by using the userspace usbfs interface from qemu / vmware / virtualbox / whatever. The basics of this work fine, but it lacks proper locking / safeguards for when a vm takes over a usb device from the in kernel driver.
Example of the problem: usb mass storage devices is mounted, filesystem is in use and dirty, with writebacks pending, user redirects to vm, usb mass storage driver sees the cable beeing yanked out of the device.
Now we've lost the pending writebacks and have a dirty, if not journaling possibly corrupt, fs on the device.
This is the issue on which I feel a bit stonewalled. Simple putting your fingers in your ears and singing la la la do it in userspace is not going to cut it here. There is no way to do this race free in userspace, unless all possible callers of mount get modified. Moreover 99% of the necessary accounting for this is already done in the kernel. We already have the notion of a block device being in use. We simply need to add some code to pass this notion to the usb mass storage driver, and add a new try_disconnect callback for usb drivers. I'm not saying this is going to be completely straight forward, but it ain't rocket science either. And it so very obviously is the *right* thing to do, that I'm getting very tired of the do it in userspace song I keep hearing.
Regards,
Hans
Re: USB mini-summit at LinuxCon Vancouveroliver
Am Freitag, 5. August 2011, 09:45:56 schrieb Hans de Goede:
Quoted 11 lines
This is the issue on which I feel a bit stonewalled. Simple putting your fingers in your ears and singing la la la do it in userspace is not going to cut it here. There is no way to do this race free in userspace, unless all possible callers of mount get modified. Moreover 99% of the necessary accounting for this is already done in the kernel. We already have the notion of a block device being in use. We simply need to add some code to pass this notion to the usb mass storage driver, and add a new
try_disconnectcallback for usb drivers. I'm not saying this is going to be completely straight forward, but it ain't rocket science either. And it so very obviously is the right thing to do, that I'm getting very tired of the do it in userspace song I keep hearing.
Doing a try_disconnect() would also solve the dual camera issue. But it doesn't really interfere with the user space vs. kernel space issue. You simply have to expand the ioctl interface to have an ioctl that triggers this API call in the kernel.
A V4L2 device would return an error if its device node is opened, otherwise disconnect.
Regards
Oliver
Re: USB mini-summit at LinuxCon Vancouveroliver
Hi,
Quoted 13 lines · Oliver Neukum
Am Freitag, 5. August 2011, 09:45:56 schrieb Hans de Goede:
Quoted 11 lines
This is the issue on which I feel a bit stonewalled. Simple putting your fingers in your ears and singing la la la do it in userspace is not going to cut it here. There is no way to do this race free in userspace, unless all possible callers of mount get modified. Moreover 99% of the necessary accounting for this is already done in the kernel. We already have the notion of a block device being in use. We simply need to add some code to pass this notion to the usb mass storage driver, and add a new
try_disconnectcallback for usb drivers. I'm not saying this is going to be completely straight forward, but it ain't rocket science either. And it so very obviously is the right thing to do, that I'm getting very tired of the do it in userspace song I keep hearing.
Quoted 7 lines
Doing a
try_disconnect()would also solve the dual camera issue. But it doesn't really interfere with the user space vs. kernel space issue. You simply have to expand the ioctl interface to have an ioctl that triggers this API call in the kernel.A V4L2 device would return an error if its device node is opened, otherwise disconnect.
Getting a bit offtopic here, but no a try_disconnect will fix the userspace stillcam mode driver being able to disconnect the device while the webcam function is active. If the webcam is not active userspace will still "win", and possibly never return the device back to the kernel driver (this already happens today with gvfs-gphoto creating a fuse mount and keeping the device open indefinitely, locking out the webcam function).
Likewise a v4l2 control panel like app (think alsamixer for videodevs to set brightness / contrast, etc.) can keep the /dev/video node open indefinitely. Unless we rewrite most of userspace, we need to allow the device to be open in bode modes *at the same time* and only fail with -EBUSY when something really exclusive is requested (so not just having the device open, or setting contrast, but trying to stream and read/delete pictures from the stillcam memory at the same time).
Regards,
Hans
Em 05-08-2011 04:45, Hans de Goede escreveu:
Quoted 32 lines
Hi,
Quoted 6 lines · Greg KH
Quoted 4 lines · Mauro Carvalho Chehab
I know that this problem were somewhat solved for 3G modems, with the usage of the userspace problem
usb_modeswitch, and with some quirks for the USB storage driver, but I'm not sure if such tricks will scale forever, as more functions are seen on some USB devices.Quoted 7 lines
Well, no matter how it "scales" it needs to be done in userspace, like
usb_modeswitchdoes. We made that decision a while ago, and it is working out very well. I see no reason why you can't do it in userspace as well as that is the easiest place to control this type of thing.I thought we had a long discussion about this topic a while ago and came to this very conclusion. Or am I mistaken?
I think we've had multiple discussions about this, surrounding various topics / use cases. I would not call the do it in userspace a conclusion.
I rather more have a feeling of getting stonewalled on this by various people surrounding the usb system. Me shutting up on this is basically a case of:
"Discussion not getting anywhere and I don't have the time to do a kernel
proof of concept myself right now".To be clear about the stonewalling I'm not talking about the dual cam issue here, nor about the morphing devices thing which
usb_modeswitchfixes.I think it is important to separate oranges from apples here, there are at least 3 different problem classes which all seem to have gotten thrown onto a pile here:
Quoted 8 lines
1) The reason Mauro suggested having some discussion on this at the USB summit is because of a discussion about dual mode cameras on the linux media list. Dual-mode cameras are (usually very cheap) digital photo cameras which can take still pictures in stand alone mode and store them in onboard memory, like regular digital photo cameras, this is one mode. The other mode is they can operate as a regular webcam, the hardware actually is more regular webcam hardware with some memory and a battery added (usually no viewfinder screen).
Yes, that's my concern.
Quoted 2 lines
These devices typically use 1 usb interface (and thus 1 driver) for both modes.
I also want to cover the case where there are two drivers involved, as this is a typical trouble we need to do with all devices that have both analog and digital TV's.
A typical media device exposes several different API's to userspace, each with its own character device, and sometimes implemented by a different driver.
For example, Conexant cx23102 USB chipsets provide 2 USB interfaces, but require 3 different drivers, and exposes 4 different userspace API's to userspace:
USB interface 0: Remote Controller
event/input API - Handled by drivers/media/rc/mceusb.c
USB interface 1: Audio/Video
ALSA API - Handled by drivers/media/video/cx231xx/cx231xx-audio.c
It is a separate driver, called cx231xx-alsa
V4L2 API - Handled by drivers/media/video/cx231xx/cx231xx-video.c DVB API - Handled by drivers/media/video/cx231xx/cx231xx-dvb.c
Both DVB and V4L2 are currently part of the same driver
(cx231xx).
The mceusb is completely independent of the other drivers. The cx231xx-alsa also is almost independent. It only shares a lock with the main cx231xx driver, and uses the I2C functions implemented by the main driver.
Now, we're starting to see devices like ZTE MF845 that have TV, 3G and USB storage, exposing 2 more userspace API's to userspace (vfs and tty).
I don't think that userspace usb_modeswitch-like type of locking "scales" such complex devices.
Besides that, on a large amount of USB sticks, if you enable both V4L2 and DVB, the power consumption may exceed the 500mA power limit of USB, and can overheat the device. There are several reports of burnt devices due to that. Also, there's no way for both to work, as they share the same tuner, that can either be tuning an analog or a digital channel.
To me, the right solution seems to add some sort of locking schema that will protect the common resources on complex hardware. Something like:
int get_hw_resource(struct device, enum hw_resource_type); void put_hw_resource(struct device, enum hw_resource_type);
enum hw_resource_type {
HW_RES_USB,
HW_RES_I2C,
HW_RES_TUNER,
HW_RES_USB,...
};
So that, when a device need, for example, an exclusive lock for the USB device, for example, it could be doing something like:
rc = get_hw_resource(dev, HW_RES_USB);
if (rc < 0)
return -EBUSY;
and, after finishing, doing a put_hw_resource().
Quoted 47 lines
We currently have drivers for both modes, but they are separate drivers, which leads to fighting for device ownership, with the stillmode driver always winning as that is userspace and it simply kicks of the kernel webcam /streaming mode driver making it see a device unplug, even if an app was streaming from the device at the time. The solution here is simple, move both functions to 1 driver, which can then properly return
-EBUSYwhen the device is active in one mode and an app tries to use it in the other mode.This 1 driver will likely end up being a kernel driver, since doing streaming drivers in userspace is problematic, as the v4l2 API assumes a kernel device node, and we don't want to invent a new API, we want something which works with existing applications.
2) So called morphing devices. For example a USB 3G modem which initially shows up as a cdrom drive with windows software, only to reveal its real identity (with completely different usb descriptors) after some magic command. This is solved by the
usb_modeswitchcommand, nothing to see here move along.3) Re-direction of usb devices to virtual machines. This works by using the userspace usbfs interface from qemu / vmware / virtualbox / whatever. The basics of this work fine, but it lacks proper locking / safeguards for when a vm takes over a usb device from the in kernel driver.
Example of the problem: usb mass storage devices is mounted, filesystem is in use and dirty, with writebacks pending, user redirects to vm, usb mass storage driver sees the cable beeing yanked out of the device.
Now we've lost the pending writebacks and have a dirty, if not journaling possibly corrupt, fs on the device.
This is the issue on which I feel a bit stonewalled. Simple putting your fingers in your ears and singing la la la do it in userspace is not going to cut it here. There is no way to do this race free in userspace, unless all possible callers of mount get modified. Moreover 99% of the necessary accounting for this is already done in the kernel. We already have the notion of a block device being in use. We simply need to add some code to pass this notion to the usb mass storage driver, and add a new
try_disconnectcallback for usb drivers. I'm not saying this is going to be completely straight forward, but it ain't rocket science either. And it so very obviously is the *right* thing to do, that I'm getting very tired of the do it in userspace song I keep hearing.Regards,
Hans
Quoted 11 lines · Hans de Goede
Hi,
Quoted 1 line · Greg KH
On Thu, Aug 04, 2011 at 07:21:47PM -0300, Mauro Carvalho Chehab wrote:
I think it is important to separate oranges from apples here, there are at least 3 different problem classes which all seem to have gotten thrown onto a pile here:
1) The reason Mauro suggested having some discussion on this at the USB summit is because of a discussion about dual mode cameras on the linux media list.
...
Quoted 4 lines
3) Re-direction of usb devices to virtual machines. This works by using the userspace usbfs interface from qemu / vmware / virtualbox / whatever. The basics of this work fine, but it lacks proper locking / safeguards for when a vm takes over a usb device from the in kernel driver.
Hi Hans and Mauro,
We have do room in the schedule for the USB mini-summit for this discussion, since the schedule is still pretty flexible. The preliminary schedule is up here:
http://userweb.kernel.org/~sarah/linuxcon-usb-minisummit.html
I think it's best to discuss the VM redirection in the afternoon when some of the KVM folks join us after Hans' talk on USB redirection over the network.
It sounds like we need a separate topic for the dual mode cameras and TV tuners. Mauro, do you want to lead that discussion in the early morning (in a 9:30 to 10:30 slot) or in the late afternoon (in a 15:30 to 16:30 slot)? I want to be sure we have all the video/media developers who are interested in this topic present, and I don't know if they will be going to the KVM forum.
Link PM was originally slated for the 10am slot, but since we're missing several people for that discussion (Alan Stern, Andiry Xu, and any of the Intel folks who did the Moorestown USB 2.0 LPM), I think it will be more useful to have that discussion on-list or at another conference.
Sarah Sharp
Quoted 30 lines · Sarah Sharp
Quoted 11 lines · Hans de Goede
Hi,
Quoted 1 line · Greg KH
On Thu, Aug 04, 2011 at 07:21:47PM -0300, Mauro Carvalho Chehab wrote:
I think it is important to separate oranges from apples here, there are at least 3 different problem classes which all seem to have gotten thrown onto a pile here:
1) The reason Mauro suggested having some discussion on this at the USB summit is because of a discussion about dual mode cameras on the linux media list.
...
Quoted 4 lines
3) Re-direction of usb devices to virtual machines. This works by using the userspace usbfs interface from qemu / vmware / virtualbox / whatever. The basics of this work fine, but it lacks proper locking / safeguards for when a vm takes over a usb device from the in kernel driver.
Hi Hans and Mauro,
We have do room in the schedule for the USB mini-summit for this discussion, since the schedule is still pretty flexible. The preliminary schedule is up here:
http://userweb.kernel.org/~sarah/linuxcon-usb-minisummit.html
I think it's best to discuss the VM redirection in the afternoon when some of the KVM folks join us after Hans' talk on USB redirection over the network.
Quoted 6 lines
It sounds like we need a separate topic for the dual mode cameras and TV tuners. Mauro, do you want to lead that discussion in the early morning (in a 9:30 to 10:30 slot) or in the late afternoon (in a 15:30 to 16:30 slot)? I want to be sure we have all the video/media developers who are interested in this topic present, and I don't know if they will be going to the KVM forum.
Sarah,
Alas. I would suspect that I am one of the people most interested in the topic of dual-mode cameras, since I have worked on supporting them both in libgphoto2 and in the kernel. But I teach in a university for a living, and the first classes of Fall Semester 2011 start on August 17 in Auburn, Alabama. Knowing this, I decided, months ago, that I simply could not attend a conference which starts on August 16 in Vancouver.
So, after starting all of the current mailing-list discussion on the topic I will not be at the conference. I can only hope that those who do attend will keep me current about what gets discussed.
Theodore Kilgore
Quoted 50 lines · Theodore Kilgore
Quoted 30 lines · Sarah Sharp
Quoted 11 lines · Hans de Goede
Hi,
Quoted 1 line · Greg KH
On Thu, Aug 04, 2011 at 07:21:47PM -0300, Mauro Carvalho Chehab wrote:
I think it is important to separate oranges from apples here, there are at least 3 different problem classes which all seem to have gotten thrown onto a pile here:
1) The reason Mauro suggested having some discussion on this at the USB summit is because of a discussion about dual mode cameras on the linux media list.
...
Quoted 4 lines
3) Re-direction of usb devices to virtual machines. This works by using the userspace usbfs interface from qemu / vmware / virtualbox / whatever. The basics of this work fine, but it lacks proper locking / safeguards for when a vm takes over a usb device from the in kernel driver.
Hi Hans and Mauro,
We have do room in the schedule for the USB mini-summit for this discussion, since the schedule is still pretty flexible. The preliminary schedule is up here:
http://userweb.kernel.org/~sarah/linuxcon-usb-minisummit.html
I think it's best to discuss the VM redirection in the afternoon when some of the KVM folks join us after Hans' talk on USB redirection over the network.
Quoted 6 lines
It sounds like we need a separate topic for the dual mode cameras and TV tuners. Mauro, do you want to lead that discussion in the early morning (in a 9:30 to 10:30 slot) or in the late afternoon (in a 15:30 to 16:30 slot)? I want to be sure we have all the video/media developers who are interested in this topic present, and I don't know if they will be going to the KVM forum.
Sarah,
Alas. I would suspect that I am one of the people most interested in the topic of dual-mode cameras, since I have worked on supporting them both in libgphoto2 and in the kernel. But I teach in a university for a living, and the first classes of Fall Semester 2011 start on August 17 in Auburn, Alabama. Knowing this, I decided, months ago, that I simply could not attend a conference which starts on August 16 in Vancouver.
Quoted 3 lines
So, after starting all of the current mailing-list discussion on the topic I will not be at the conference. I can only hope that those who do attend will keep me current about what gets discussed.
No worries, I'll be taking notes and post them to the Linux USB and Linux media lists.
Sarah Sharp
Hi Sarah,
Em 08-08-2011 14:58, Sarah Sharp escreveu:
Quoted 24 lines · Hans de Goede
Quoted 11 lines
Hi,
Quoted 1 line · Greg KH
On Thu, Aug 04, 2011 at 07:21:47PM -0300, Mauro Carvalho Chehab wrote:
I think it is important to separate oranges from apples here, there are at least 3 different problem classes which all seem to have gotten thrown onto a pile here:
1) The reason Mauro suggested having some discussion on this at the USB summit is because of a discussion about dual mode cameras on the linux media list.
...
Quoted 4 lines
3) Re-direction of usb devices to virtual machines. This works by using the userspace usbfs interface from qemu / vmware / virtualbox / whatever. The basics of this work fine, but it lacks proper locking / safeguards for when a vm takes over a usb device from the in kernel driver.
Hi Hans and Mauro,
We have do room in the schedule for the USB mini-summit for this discussion, since the schedule is still pretty flexible. The preliminary schedule is up here:
Quoted 8 lines
http://userweb.kernel.org/~sarah/linuxcon-usb-minisummit.html
I think it's best to discuss the VM redirection in the afternoon when some of the KVM folks join us after Hans' talk on USB redirection over the network.
It sounds like we need a separate topic for the dual mode cameras and TV tuners.
Yes, I suspect so. This subject is pending for quite some time, and it would be good to have people think together around a solution for that.
Quoted 6 lines
Mauro, do you want to lead that discussion in the early morning (in a 9:30 to 10:30 slot) or in the late afternoon (in a 15:30 to 16:30 slot)? I want to be sure we have all the video/media developers who are interested in this topic present, and I don't know if they will be going to the KVM forum.
Quoted 8 lines
From my side, both slots work ok. Hans, what would fit better for you?
Link PM was originally slated for the 10am slot, but since we're missing several people for that discussion (Alan Stern, Andiry Xu, and any of the Intel folks who did the Moorestown USB 2.0 LPM), I think it will be more useful to have that discussion on-list or at another conference.
Sarah Sharp
Thanks!
Mauro
Hi,
Quoted 24 lines · Sarah Sharp
Quoted 11 lines · Hans de Goede
Hi,
Quoted 1 line · Greg KH
On Thu, Aug 04, 2011 at 07:21:47PM -0300, Mauro Carvalho Chehab wrote:
I think it is important to separate oranges from apples here, there are at least 3 different problem classes which all seem to have gotten thrown onto a pile here:
1) The reason Mauro suggested having some discussion on this at the USB summit is because of a discussion about dual mode cameras on the linux media list.
...
Quoted 4 lines
3) Re-direction of usb devices to virtual machines. This works by using the userspace usbfs interface from qemu / vmware / virtualbox / whatever. The basics of this work fine, but it lacks proper locking / safeguards for when a vm takes over a usb device from the in kernel driver.
Hi Hans and Mauro,
We have do room in the schedule for the USB mini-summit for this discussion, since the schedule is still pretty flexible. The preliminary schedule is up here:
Quoted 6 lines
http://userweb.kernel.org/~sarah/linuxcon-usb-minisummit.html
I think it's best to discuss the VM redirection in the afternoon when some of the KVM folks join us after Hans' talk on USB redirection over the network.
That seems best to me too. I'm available at both proposed time slots, and I would like to join both discussions.
Quoted 6 lines
It sounds like we need a separate topic for the dual mode cameras and TV tuners. Mauro, do you want to lead that discussion in the early morning (in a 9:30 to 10:30 slot) or in the late afternoon (in a 15:30 to 16:30 slot)? I want to be sure we have all the video/media developers who are interested in this topic present, and I don't know if they will be going to the KVM forum.
I would really like to see the dual mode camera and TV tuner discussion separated. They are 2 different issues AFAIK.
1) Dual mode cameras:
In the case of the dual mode camera we have 1 single device (both at the hardware level and at the logical block level), which can do 2 things, but not at the same time. It can stream live video data from a sensor, or it can retrieve earlier taken pictures from some picture memory.
Unfortunately even though these 2 functions live in a single logical block, historically we've developed 2 drivers for them. This leads to fighting over device ownership (surprise surprise), and to me the solution is very clear, 1 logical block == 1 driver.
2) Tv tuners:
Here we have a bunch of logical blocks, each with their own driver (and not 2 drivers for one block), which together form not 1 but 2 video pipelines, typically one pipeline for analog TV, and one for DVB. The problem here is some blocks are shared between the 2 pipelines.
The solution here, at least to me, is clear too, we need some way of claiming/locking the pipeline blocks. If pipe1 starts streaming video data it locks all the blocks it uses. If some app then tries to use pipe2 (and thus for example tune the tuner to a different frequency), it will fail with -EBUSY when pipe2 tries to take the tuner lock and gets told it is in use.
Regards,
Hans
Quoted 10 lines · Hans de Goede
I would really like to see the dual mode camera and TV tuner discussion separated. They are 2 different issues AFAIK.
1) Dual mode cameras:
In the case of the dual mode camera we have 1 single device (both at the hardware level and at the logical block level), which can do 2 things, but not at the same time. It can stream live video data from a sensor, or it can retrieve earlier taken pictures from some picture memory.
Quoted 4 lines
Unfortunately even though these 2 functions live in a single logical block, historically we've developed 2 drivers for them. This leads to fighting over device ownership (surprise surprise), and to me the solution is very clear, 1 logical block == 1 driver.
According to Theodore, we have developed 5 drivers for them because the stillcam modes in different devices use four different vendor-specific drivers. Does it really make sense to combine 5 drivers into one? I think some sort of sharing protocol would work out better.
Alan Stern
Hi
I've been thinking about the Kernel driver side. Mauro and others emailed requirements on Jun or July.
I'm sorry for this spam: maybe you have thought this already.
A linked list of read/write locks as the solution for these protections could be a base for the general solution. Locks could be accessed either by name "string name" or by an integer identifier.
The bridge driver would be the container of the lock list.
Lock take / release handles would be changeable by Bridge driver at driver init. This way bridge could tune the lock taking actions. Sub devices would not have to know the details of the bridge device.
When implemented as a library ".ko" module, this could be
used by all related kernel drivers. The locking code would be very general.
Maybe adding a lock list into each PCI bus device would solve the device export problem to KVM too.
If some module wouldn't handle the proper locking yet,
it would not deliver the protection, but it would work as before: no regressions.
The library could also be called so that a driver would ask three locks at the same time. If the driver would get all three locks, it would return success.
If the driver would not get all three locks, it would not lock any of them (with _trylock case).
I don't have time to implement this feature.
Happy meeting for all of you,
Marko Ristola
Hi,
Quoted 12 lines · Alan Stern
Quoted 10 lines · Hans de Goede
I would really like to see the dual mode camera and TV tuner discussion separated. They are 2 different issues AFAIK.
1) Dual mode cameras:
In the case of the dual mode camera we have 1 single device (both at the hardware level and at the logical block level), which can do 2 things, but not at the same time. It can stream live video data from a sensor, or it can retrieve earlier taken pictures from some picture memory.
Quoted 8 lines
Quoted 4 lines
Unfortunately even though these 2 functions live in a single logical block, historically we've developed 2 drivers for them. This leads to fighting over device ownership (surprise surprise), and to me the solution is very clear, 1 logical block == 1 driver.
According to Theodore, we have developed 5 drivers for them because the stillcam modes in different devices use four different vendor-specific drivers.
Yes, but so the the webcam modes of the different devices, so for the 5 (not sure if that is the right number) dual-cam mode chipsets we support there will be 5 drivers, each supporting both the webcam and the access to pictures stored in memory of the chipset they support. So 5 chipsets -> 5 drivers each supporting 1 chipset, and both functions of the single logical device that chipset represents.
Quoted 1 line
Does it really make sense to combine 5 drivers into one?
Right, that is not the plan. The plan is to simply stop having 2 drivers for 1 logical (and physical) block. So we go from 10 drivers, 5 stillcam
+ 5 webcam, to just 5 drivers. We will also likely be able to share code between the code for the 2 functionalities for things like generic set / get register functions, initialization, etc.
Regards,
Hans
Quoted 4 lines · Hans de Goede
Hi,
On 08/09/2011 04:19 PM, Alan Stern wrote:
Quoted 8 lines
Quoted 1 line
Does it really make sense to combine 5 drivers into one?
Right, that is not the plan. The plan is to simply stop having 2 drivers for 1 logical (and physical) block. So we go from 10 drivers, 5 stillcam
+ 5 webcam, to just 5 drivers. We will also likely be able to share code between the code for the 2 functionalities for things like generic set / get register functions, initialization, etc.
Unfortunately as Theodore recently pointed out you don't go from 10 to 5, you go from 10 to 10 where 5 of the new 10 are only used on Win32, FreeBSD and OSX (but they aren't any simpler because they still rely on libusb) and the other 5 that are only used on Linux become significantly more complicated than they currently are.
It has also just occured to me that it might be possible to solve the issues we are facing just in the kernel. At the moment when the kernel performs a USBDEVFS_DISCONNECT it keeps the kernel driver locked out until userspace performs a USBDEVFS_CONNECT. If the kernel reattached the kernel driver when the device file was closed then, as gvfs doesn't keep the file open the biggest current issue would be solved instantly. If a mechanism could be found to prevent USBDEVFS_DISCONNECT from succeeding when the corresponding /dev/videox file was open then that would seem to be a reasonable solution.
Hans had expressed the opinion that merely having the device open to control the camera not to stream shouldn't prevent stillcam operation - I disagree because if you are setting up the controls you are probably already streaming so you can see what you are doing and if not you are probably about to.
Of course changing the behaviour of USBDEVFS_DISCONNECT is not something to be done lightly. I don't know how many other users there are for it and if the current behaviour is actually correct for any of them. Cleaning up on file close does have the useful side effect though that applications no longer need to worry about the fact that even if they clean up properly on a normal exit, if they crash they leave the kernel driver permanently disabled
Regards
Adam
Hi,
Quoted 1 line · Adam Baker
On Tuesday 09 August 2011, Hans de Goede wrote:
<snip>
Quoted 8 lines
It has also just occured to me that it might be possible to solve the issues we are facing just in the kernel. At the moment when the kernel performs a
USBDEVFS_DISCONNECTit keeps the kernel driver locked out until userspace performs aUSBDEVFS_CONNECT. If the kernel reattached the kernel driver when the device file was closed then, as gvfs doesn't keep the file open the biggest current issue would be solved instantly. If a mechanism could be found to preventUSBDEVFS_DISCONNECTfrom succeeding when the corresponding/dev/videoxfile was open then that would seem to be a reasonable solution.
<sigh>
This has been discussed over and over and over again, playing clever tricks with USBDEVFS_[DIS]CONNECT like adding a new USBDEVFS_TRYDISCONNECT which the v4l2 driver could intercept won't cut it. We need some central manager of the device doing multiplexing between the 2 functions, and you can *not* assume that either side will be nice wrt closing file descriptors.
Examples:
1) You are wrong wrt gvfs, it does keep the libgphoto2 context open all the time, and through that the usbfs device nodes.
2) Lets say a user starts a photo managing app like f-spot, and that opens the device through libgphoto2 on startup, then the user switches to another virtual desktop and forgets all about having f-spot open. Notice that if the user now tries to stream he will not get a busy error, but the app trying to do the streaming will simply not see the camera at all (kernel driver unbound /dev/video# node is gone).
3) Notice that little speaker icon in your panel on your average Linux desktop, that keeps the mixer of the audio device open *all the time* it is quite easy to imagine a similar applet for v4l2 device controls (see for example gtk-v4l) doing the same. Or a user could simply start up a v4l2 control panel app like gtk-v4l, qv4l2 or v4l2ucp, and leave it running minimized ...
4) Some laptops have a Fn + F## key which enables / disables the builtin webcam by turning its power on / off. Effectively plugging it into / out of a usb port. We would like to have an on screen notification of this one day like we have now for brightness and volume controls, based on udev events. But the current dual mode cam stuff causes udev events for a *new* video device being added / an existing one being removed each time libgphoto2 releases / takes control of the camera.
5) More in general, more and more software is dynamically monitoring the addition / removal of (usb) devices using udev, our current solution suggests to this software the /dev/video device is being unplugged / re-plugged all the time, not pretty.
All in all what we've today is a kludge, and if we want to provide a "seamless" user experience we need to fix it.
Don't get me wrong usbfs is a really nice solution for driving usb devices from userspace, like scanners and all other sorts of devices. But what all these devices have in common is that they have no kernel driver. Having a userspace based driver and a kernel driver "fight it out" just does not work well.
Regards,
Hans
Quoted 6 lines · Hans de Goede
This has been discussed over and over and over again, playing clever tricks with USBDEVFS_[DIS]CONNECT like adding a new
USBDEVFS_TRYDISCONNECT
which the v4l2 driver could intercept won't cut it. We need some central manager of the device doing multiplexing between the 2 functions, and you can *not* assume that either side will be nice wrt closing file descriptors.
So it seems to be both good and bad for Linux's ability to do multiplexing between the two drivers (one purpose-built kernel driver and one generic usbfs driver to be used libusb). The good thing is that it is quite useful in many use cases. The ability to detach the existing kernel driver and use usbfs is the base for many libusb based program under Linux and it beats Windows and Mac OS X handsomely in this aspect. On the other hand, based on the discussions, it seems to adds quite some issues/complexities in some other use cases.
Windows typically does not have such an issue. You can use only one driver. In order to use libusb-win32 or libusb-1.0 Windows backend, you have to replace the existing kernel driver with libusb-win32 kernel driver (libusb0.sys) or libusb-1.0 Windows backend driver (currently winusb.sys).
The exception is the libusb0.sys filter driver. In that case, you use libusb0.sys as the upper filter driver on top of the existing kernel driver. On the other hand, libusb-win32 filter driver is not really widely used due to issues with older versions.
Quoted 9 lines
All in all what we've today is a kludge, and if we want to provide a "seamless" user experience we need to fix it.
Don't get me wrong usbfs is a really nice solution for driving usb devices from userspace, like scanners and all other sorts of devices. But what all these devices have in common is that they have no kernel driver. Having a userspace based driver and a kernel driver "fight it out" just does not work well.
All in all, I think this is a good summary. There are cases where a kernel driver and a usbfs are both used, an example is FTDI device where ftdi-sio driver and the user space libftdi (based on libusb) are used, but there are problems with that as well with regard to switching between the two drivers.
There are a few discussions here.
http://libusb.6.n5.nabble.com/Patch-libusb-os-linux-usbfs-c-Distingush-between-usbfs-and-other-kernel-mode-drivers-td3199947.html
http://libusb.6.n5.nabble.com/open-device-exclusively-td4524397.html
And there are no good solutions to a "simple" issue as the above mentioned.
Not so sure if this is a good suggestion or not: just wondering how the other side (eg Windows) deal with these dual mode cameras? Does Windows use single driver or does Windows use two drivers for these dual mode cameras? Since they are customized device, then there must be a vendor driver for them. And since they will function as a video cam and a still cam device, I assume there will be standard Windows drivers on top of the vendor drivers. So there will be more than one drivers associated with them.
Signature · 1 line
Xiaofan
Quoted 25 lines · Hans de Goede
Hi,
Quoted 1 line · Adam Baker
On Tuesday 09 August 2011, Hans de Goede wrote:
<snip>
Quoted 9 lines
It has also just occured to me that it might be possible to solve the issues we are facing just in the kernel. At the moment when the kernel performs a
USBDEVFS_DISCONNECTit keeps the kernel driver locked out until userspace performs aUSBDEVFS_CONNECT. If the kernel reattached the kernel driver when the device file was closed then, as gvfs doesn't keep the file open the biggest current issue would be solved instantly. If a mechanism could be found to preventUSBDEVFS_DISCONNECTfrom succeeding when the corresponding/dev/videoxfile was open then that would seem to be a reasonable solution.<sigh>
This has been discussed over and over and over again, playing clever tricks with USBDEVFS_[DIS]CONNECT like adding a new
USBDEVFS_TRYDISCONNECTwhich the v4l2 driver could intercept won't cut it. We need some central manager of the device doing multiplexing between the 2 functions, and you can *not* assume that either side will be nice wrt closing file descriptors.
Quoted 3 lines
Examples:
1) You are wrong wrt gvfs, it does keep the libgphoto2 context open all the time, and through that the usbfs device nodes.
It seems that that depends, on my system gvfs isn't actually automounting the camera after it detects it and the file is only open (according to lsof) when the device is actually mounted. As soon as you unmount it the device gets closed again. Because it does do a brief open, USBDEVFS_DISCONNECT then close at connection time it does still disable the kernel driver.
Quoted 7 lines
2) Lets say a user starts a photo managing app like f-spot, and that opens the device through libgphoto2 on startup, then the user switches to another virtual desktop and forgets all about having f-spot open. Notice that if the user now tries to stream he will not get a busy error, but the app trying to do the streaming will simply not see the camera at all (kernel driver unbound /dev/video# node is gone).
This does seem like a situation where your approach could potentially give a better user experience. I'm wondering slightly how you define busy though. For webcams the streamon and streamoff ioctls tell you if you are using mmap or userptr transfers but you don't know if when the user has finished if they just use read. For stillcam mode it is again hard to determine a busy condition other than being in the middle of transfering an individual picture.
Quoted 8 lines
3) Notice that little speaker icon in your panel on your average Linux desktop, that keeps the mixer of the audio device open *all the time* it is quite easy to imagine a similar applet for v4l2 device controls (see for example gtk-v4l) doing the same. Or a user could simply start up a v4l2 control panel app like gtk-v4l, qv4l2 or v4l2ucp, and leave it running minimized ...
This again needs a usable concept of busy
Quoted 8 lines
4) Some laptops have a Fn + F## key which enables / disables the builtin webcam by turning its power on / off. Effectively plugging it into / out of a usb port. We would like to have an on screen notification of this one day like we have now for brightness and volume controls, based on udev events. But the current dual mode cam stuff causes udev events for a *new* video device being added / an existing one being removed each time libgphoto2 releases / takes control of the camera.
Would such a system know what camera is supposed to be the internal one so it doesn't show the camera as turned on just because you plug in an external camera. If so then it won't turn on and off as an external camera changes modes. If not then showing on when any camera is usable and off when it isn't seems like sensible behaviour.
Quoted 8 lines
5) More in general, more and more software is dynamically monitoring the addition / removal of (usb) devices using udev, our current solution suggests to this software the
/dev/videodevice is being unplugged / re-plugged all the time, not pretty.All in all what we've today is a kludge, and if we want to provide a "seamless" user experience we need to fix it.
I think in summary I'm concerned about the possibility of perfect being the enemy of good enough. At the moment we've got a significant usability problem (a web search for gvfs-gphoto2-volume-monitor turns up mostly instructions on how to disable it). If we come up with a solution that whilst it would be perfect there isn't enough effort available to implement then that is worse than a solution that fixes most of the problem. This is an even greater concern when the technically superior solution has a higher long term maintenance overhead (as we no longer get Win32 and OSX users helping to maintain the stillcam drivers).
I'm not sure if there is anything in this discussion that is relevant to the cameras in phones or tablets. These appear to the user as if they are dual mode devices but they don't have any independent storage - taking a photo is more like capturing a single higher than usual res frame so I suspect they aren't going to be an issue.
Regards
Adam
Hi,
Quoted 26 lines · Adam Baker
Quoted 25 lines · Hans de Goede
Hi,
Quoted 1 line · Adam Baker
On Tuesday 09 August 2011, Hans de Goede wrote:
<snip>
Quoted 9 lines
It has also just occured to me that it might be possible to solve the issues we are facing just in the kernel. At the moment when the kernel performs a
USBDEVFS_DISCONNECTit keeps the kernel driver locked out until userspace performs aUSBDEVFS_CONNECT. If the kernel reattached the kernel driver when the device file was closed then, as gvfs doesn't keep the file open the biggest current issue would be solved instantly. If a mechanism could be found to preventUSBDEVFS_DISCONNECTfrom succeeding when the corresponding/dev/videoxfile was open then that would seem to be a reasonable solution.<sigh>
This has been discussed over and over and over again, playing clever tricks with USBDEVFS_[DIS]CONNECT like adding a new
USBDEVFS_TRYDISCONNECTwhich the v4l2 driver could intercept won't cut it. We need some central manager of the device doing multiplexing between the 2 functions, and you can *not* assume that either side will be nice wrt closing file descriptors.
Quoted 7 lines
Quoted 3 lines
Examples:
1) You are wrong wrt gvfs, it does keep the libgphoto2 context open all the time, and through that the usbfs device nodes.It seems that that depends, on my system gvfs isn't actually automounting the camera after it detects it and the file is only open (according to lsof) when the device is actually mounted.
Right, but on some systems it does auto-mount, and even if it does not auto-mount and the user then just clicks on its icon in the filemanager to look at the photo's it will mount, and it will be non obvious to the user to umount it.
Once mounted, the /dev/video# node will be gone. So if for example skype gets started it will say the user has no camera. If it were to say the device is busy, the user just might make a link to the mounted gvfs share, but even then that is a big might.
The whole disappear / reappear act of the /dev/video# node is one of my concerns with the current solution.
Quoted 3 lines
As soon as you unmount it the device gets
closed again. Because it does do a brief open,USBDEVFS_DISCONNECTthen close at connection time it does still disable the kernel driver.
Then you've a bug in your libgphoto2, Fedora had a similar bug which I fixed (and send the fix upstream). libgphoto2 now a days does make the call to re-attach the kernel driver.
Quoted 9 lines
Quoted 8 lines
2) Lets say a user starts a photo managing app like f-spot, and that opens the device through libgphoto2 on startup, then the user switches to another virtual desktop and forgets all about having f-spot open. Notice that if the user now tries to stream he will not get a busy error, but the app trying to do the streaming will simply not see the camera at all (kernel driver unbound /dev/video# node is gone).
Quoted 6 lines
This does seem like a situation where your approach could potentially give a better user experience. I'm wondering slightly how you define busy though. For webcams the streamon and streamoff ioctls tell you if you are using mmap or userptr transfers but you don't know if when the user has finished if they just use read. For stillcam mode it is again hard to determine a busy condition other than being in the middle of transfering an individual picture.
Right, so my plan is to define busy for stillcam mode as not being in the middle of some operation (be it download or delete).
Quoted 9 lines
Quoted 7 lines
3) Notice that little speaker icon in your panel on your average Linux desktop, that keeps the mixer of the audio device open *all the time* it is quite easy to imagine a similar applet for v4l2 device controls (see for example gtk-v4l) doing the same. Or a user could simply start up a v4l2 control panel app like gtk-v4l, qv4l2 or v4l2ucp, and leave it running minimized ...
This again needs a usable concept of busy
Actually this one is easy. Most streaming drivers already handle controls while not streaming by remembering the set values, and these cached values usually then get set on stream start. There are multiple reasons for this:
- usb transfers are expensive no need to send a value to the device if it may change before it gets actually used
- in many cases parts of the device are powered down when not streaming, so sending values while not streaming just drops them in a black hole
<snip>
Quoted 3 lines
If we come up with a solution that whilst it would be
perfect there isn't enough effort available to implement then that is worse than a solution that fixes most of the problem.
Agreed.
Quoted 4 lines
This is an even greater
concern when the technically superior solution has a higher long term maintenance overhead (as we no longer get Win32 and OSX users helping to maintain the stillcam drivers).
I've been subscribed to the libgphoto2-devel list for quite some time now, I believe I still have to see the first win32 / OS-X user input. We do get occasional FreeBSD input, usually in the form of "it no longer compiles".
I'm not saying that portability is not important, but contributions from users on other platforms should not be overrated either.
Regards,
Hans
Quoted 33 lines · Hans de Goede
Hi,
Quoted 18 lines · Alan Stern
Quoted 10 lines · Hans de Goede
I would really like to see the dual mode camera and TV tuner discussion separated. They are 2 different issues AFAIK.
1) Dual mode cameras:
In the case of the dual mode camera we have 1 single device (both at the hardware level and at the logical block level), which can do 2 things, but not at the same time. It can stream live video data from a sensor, or it can retrieve earlier taken pictures from some picture memory.
Quoted 5 lines
Unfortunately even though these 2 functions live in a single logical block,
historically we've developed 2 drivers for them. This leads to fighting over device ownership (surprise surprise), and to me the solution is very clear, 1 logical block == 1 driver.Quoted 3 lines
According to Theodore, we have developed 5 drivers for them because the stillcam modes in different devices use four different vendor-specific drivers.
Yes, but so the the webcam modes of the different devices, so for the 5 (not sure if that is the right number) dual-cam mode chipsets we support there will be 5 drivers, each supporting both the webcam and the access to pictures stored in memory of the chipset they support. So 5 chipsets -> 5 drivers each supporting 1 chipset, and both functions of the single logical device that chipset represents.
Quoted 7 lines
Quoted 1 line
Does it really make sense to combine 5 drivers into one?
Right, that is not the plan. The plan is to simply stop having 2 drivers for 1 logical (and physical) block. So we go from 10 drivers, 5 stillcam
+ 5 webcam, to just 5 drivers. We will also likely be able to share code between the code for the 2 functionalities for things like generic set / get register functions, initialization, etc.
Probably this is a good time for some inventory to be done.
sure of the exact number of drivers. The jeilinj driver does not count
here. It is a standard mass storage device as a still camera, and if it is going to be hooked up as a webcam then it has to have some buttons pushed just so, prior to hookup -- after which it comes up with a different USB Product number as a proprietary webcam device, supported by the jeilinj driver. Thus, this is not one of the droids we are looking for.
Specifically, the currently affected drivers are sq905, sq905c, mr97310a, and sn9c2028.
Fuji Finepix seems to produce both still cameras (PTP protocol) and webcams. If there are currently any dual-mode cameras, it is not obvious. Perhaps any of these which can be dual mode are like the jeilinj cameras and come up under different Product numbers in the different modes, which is not a problem.
There do seem to exist stv06xx dual-mode cameras:
stv0680/stv0680.c: { "STM:USB Dual-mode camera", 0x0553, 0x0202, 1
},
is found in camlibs/stv0680, for example, along with a whole bunch of other cameras with the same Vendor:Product number. It seems, too, that the same Vendor:Product number is listed in gspca/stv0680.c
So I guess we found another affected driver.
The intersection seems to be void between the lists of the spca50x cameras and webcams, though one can not be certain about the future. I do know of another dualmode camera, powered by the JL2005A chip. I wrote a kernel module for it some months ago. Said module has not been sent upward because of mysterious issues with one of the three cameras used for testing it.
We also find among the gspca driver files this
* GSPCA sub driver for W996[78]CF JPEG USB Dual Mode Camera Chip. *
* Copyright (C) 2009 Hans de Goede <[email protected]>
but I am not sure about which cameras are using this.
Theodore Kilgore
Quoted 16 lines · Hans de Goede
Hi,
Quoted 4 lines · Alan Stern
On Tue, 9 Aug 2011, Hans de Goede wrote:
According to Theodore, we have developed 5 drivers for them because the stillcam modes in different devices use four different vendor-specific drivers.Yes, but so the the webcam modes of the different devices, so for the 5 (not sure if that is the right number) dual-cam mode chipsets we support there will be 5 drivers, each supporting both the webcam and the access to pictures stored in memory of the chipset they support. So 5 chipsets -> 5 drivers each supporting 1 chipset, and both functions of the single logical device that chipset represents.
Quoted 7 lines
Quoted 1 line
Does it really make sense to combine 5 drivers into one?
Right, that is not the plan. The plan is to simply stop having 2 drivers for 1 logical (and physical) block. So we go from 10 drivers, 5 stillcam
+ 5 webcam, to just 5 drivers. We will also likely be able to share code between the code for the 2 functionalities for things like generic set / get register functions, initialization, etc.
Okay, I didn't realize that the different cameras used different webcam drivers as well as different stillcam drivers.
As far as I can see, there's nothing to stop anybody from adding the stillcam functionality into the webcam drivers right now. If some common code can be abstracted out into a shared source file, so much the better.
That would solve the problem, right?
Alan Stern
Quoted 20 lines · Alan Stern
Quoted 18 lines · Hans de Goede
Hi,
Quoted 4 lines · Alan Stern
On Tue, 9 Aug 2011, Hans de Goede wrote:
According to Theodore, we have developed 5 drivers for them because the stillcam modes in different devices use four different vendor-specific drivers.Yes, but so the the webcam modes of the different devices, so for the 5 (not sure if that is the right number) dual-cam mode chipsets we support there will be 5 drivers, each supporting both the webcam and the access to pictures stored in memory of the chipset they support. So 5 chipsets -> 5 drivers each supporting 1 chipset, and both functions of the single logical device that chipset represents.
Quoted 1 line
Does it really make sense to combine 5 drivers into one?
Quoted 8 lines
Quoted 5 lines
Right, that is not the plan. The plan is to simply stop having 2 drivers for 1 logical (and physical) block. So we go from 10 drivers, 5 stillcam
+ 5 webcam, to just 5 drivers. We will also likely be able to share code between the code for the 2 functionalities for things like generic set / get register functions, initialization, etc.Okay, I didn't realize that the different cameras used different webcam drivers as well as different stillcam drivers.
Oh, yes. They are Proprietary devices. And that means what it says. :-) And all different from each other, too.
Quoted 6 lines
As far as I can see, there's nothing to stop anybody from adding the stillcam functionality into the webcam drivers right now. If some common code can be abstracted out into a shared source file, so much the better.
That would solve the problem, right?
I think everyone involved believes that it would solve the problem.
The question has been all along whether or not there is any other way which would work. Also the question of what, exactly, "belongs" in the kernel and what does not. For, if something has been historically supported in userspace (stillcam support, in this case) and has worked well there, I would think it is kind of too bad to have to move said support into the kernel just because the same hardware requires kernel support for another functionality and the two sides clash. I mean, the kernel is already big enough, no? But the logic that Hans has set forth seems rather compelling.
Theodore Kilgore
Quoted 15 lines · Theodore Kilgore
Quoted 2 lines
Okay, I didn't realize that the different cameras used different webcam drivers as well as different stillcam drivers.
Oh, yes. They are Proprietary devices. And that means what it says. :-) And all different from each other, too.
Quoted 6 lines
As far as I can see, there's nothing to stop anybody from adding the stillcam functionality into the webcam drivers right now. If some common code can be abstracted out into a shared source file, so much the better.
That would solve the problem, right?
I think everyone involved believes that it would solve the problem.
Quoted 9 lines
The question has been all along whether or not there is any other way which would work. Also the question of what, exactly, "belongs" in the kernel and what does not. For, if something has been historically supported in userspace (stillcam support, in this case) and has worked well there, I would think it is kind of too bad to have to move said support into the kernel just because the same hardware requires kernel support for another functionality and the two sides clash. I mean, the kernel is already big enough, no? But the logic that Hans has set forth seems rather compelling.
The alternative seems to be to define a device-sharing protocol for USB drivers. Kernel drivers would implement a new callback (asking them to give up control of the device), and usbfs would implement new ioctls by which a program could ask for and relinquish control of a device. The amount of rewriting needed would be relatively small.
A few loose ends would remain, such as how to handle suspends, resumes, resets, and disconnects. Assuming usbfs is the only driver that will want to share a device in this way, we could handle them.
Hans, what do you think?
Alan Stern
Quoted 33 lines · Alan Stern
Quoted 15 lines · Theodore Kilgore
Quoted 2 lines
Okay, I didn't realize that the different cameras used different webcam drivers as well as different stillcam drivers.
Oh, yes. They are Proprietary devices. And that means what it says. :-) And all different from each other, too.
Quoted 6 lines
As far as I can see, there's nothing to stop anybody from adding the stillcam functionality into the webcam drivers right now. If some common code can be abstracted out into a shared source file, so much the better.
That would solve the problem, right?
I think everyone involved believes that it would solve the problem.
Quoted 9 lines
The question has been all along whether or not there is any other way which would work. Also the question of what, exactly, "belongs" in the kernel and what does not. For, if something has been historically supported in userspace (stillcam support, in this case) and has worked well there, I would think it is kind of too bad to have to move said support into the kernel just because the same hardware requires kernel support for another functionality and the two sides clash. I mean, the kernel is already big enough, no? But the logic that Hans has set forth seems rather compelling.
The alternative seems to be to define a device-sharing protocol for USB drivers. Kernel drivers would implement a new callback (asking them to give up control of the device), and usbfs would implement new ioctls by which a program could ask for and relinquish control of a device. The amount of rewriting needed would be relatively small.
Quoted 7 lines
A few loose ends would remain, such as how to handle suspends, resumes, resets, and disconnects. Assuming usbfs is the only driver that will want to share a device in this way, we could handle them.
Hans, what do you think?
Alan Stern
Alan,
Hans seems to have argued cogently for doing all of this in the kernel and for abandoning the usbfs-based drivers for these particular drivers for dual-mode cameras and, I would conjecture, for drivers for dual-mode hardware in general. Therefore, I anticipate that he won't like that very much.
My position:
I do not have preconceptions about how the problem gets handled, and at this point I remain agnostic and believe that all approaches ought to be carefully analysed. I can imagine, abstractly, that things like this could be handled by
-- moving all basic functionality to the kernel, and fixing the relevant libgphoto2 drivers to look to the kernel instead of to libusb. (What Hans argues for, and I am not opposed if his arguments convince other concerned parties)
-- doing some kind of patch job to make current arrangement somehow to work better (this seems to be the position of Adam Baker; I do share the skepticism Hans has expressed about how well this could all be pasted together)
-- doing something like the previous, but also figuring out how to bring udev rules into play, which would make it all work better (just tossing this one in, for laughs, but who knows someone might like it)
-- moving the kernel webcam drivers out of the kernel and doing with these cameras _everything_ including webcam function through libusb. I myself do not have the imagination to be able to figure out how this could be done without a rather humongous amount of work (for example, which streaming apps that are currently available would be able to live with this?) but unless I misunderstand what he was saying, Greg K-H seems to think that this would be the best thing to do.
Which one of these possibile approaches gets adopted is a policy issue which I would consider is ultimately way above my pay grade.
My main motivation for bringing up the issue was to get it to the front burner so that _something_ gets done. It is a matter which has been left alone for too long. Therefore, I am very glad that the matter is being addressed.
Let me add to this that I have gotten permission for time off and for a expense money which might possibly cover my air fare. I hope to arrive in Vancouver by sometime on Monday and intend to attend the mini-summit. I suggest that we get all intersted parties together and figure out what is the best way to go.
I hope everyone who is actively concerned can meet in Vancouver, and if all goes well then on Monday as well as Tuesday. I can hang around for another day or two after Tuesday, but I do not expect to register for LinuxCon or be involved in it. When I leave Vancouver I will probably go to Seattle and spend a couple of days with my oldest son, the musician, before coming home on the next weekend.
Theodore Kilgore
Quoted 73 lines · On Wednesday, August 10, 2011 20:33:25 Theodore Kilgore wrote:
Quoted 33 lines · Alan Stern
Quoted 15 lines · Theodore Kilgore
Quoted 2 lines
Okay, I didn't realize that the different cameras used different webcam drivers as well as different stillcam drivers.
Oh, yes. They are Proprietary devices. And that means what it says. :-) And all different from each other, too.
Quoted 6 lines
As far as I can see, there's nothing to stop anybody from adding the stillcam functionality into the webcam drivers right now. If some common code can be abstracted out into a shared source file, so much the better.
That would solve the problem, right?
I think everyone involved believes that it would solve the problem.
Quoted 9 lines
The question has been all along whether or not there is any other way which would work. Also the question of what, exactly, "belongs" in the kernel and what does not. For, if something has been historically supported in userspace (stillcam support, in this case) and has worked well there, I would think it is kind of too bad to have to move said support into the kernel just because the same hardware requires kernel support for another functionality and the two sides clash. I mean, the kernel is already big enough, no? But the logic that Hans has set forth seems rather compelling.
The alternative seems to be to define a device-sharing protocol for USB drivers. Kernel drivers would implement a new callback (asking them to give up control of the device), and usbfs would implement new ioctls by which a program could ask for and relinquish control of a device. The amount of rewriting needed would be relatively small.
Quoted 7 lines
A few loose ends would remain, such as how to handle suspends, resumes, resets, and disconnects. Assuming usbfs is the only driver that will want to share a device in this way, we could handle them.
Hans, what do you think?
Alan Stern
Alan,
Hans seems to have argued cogently for doing all of this in the kernel and for abandoning the usbfs-based drivers for these particular drivers for dual-mode cameras and, I would conjecture, for drivers for dual-mode hardware in general. Therefore, I anticipate that he won't like that very much.
My position:
I do not have preconceptions about how the problem gets handled, and at this point I remain agnostic and believe that all approaches ought to be carefully analysed. I can imagine, abstractly, that things like this could be handled by
-- moving all basic functionality to the kernel, and fixing the relevant libgphoto2 drivers to look to the kernel instead of to libusb. (What Hans argues for, and I am not opposed if his arguments convince other concerned parties)
-- doing some kind of patch job to make current arrangement somehow to work better (this seems to be the position of Adam Baker; I do share the skepticism Hans has expressed about how well this could all be pasted together)
-- doing something like the previous, but also figuring out how to bring udev rules into play, which would make it all work better (just tossing this one in, for laughs, but who knows someone might like it)
Quoted 7 lines
-- moving the kernel webcam drivers out of the kernel and doing with these cameras _everything_ including webcam function through libusb. I myself do not have the imagination to be able to figure out how this could be done without a rather humongous amount of work (for example, which streaming apps that are currently available would be able to live with this?) but unless I misunderstand what he was saying, Greg K-H seems to think that this would be the best thing to do.
Webcam drivers can rely on i2c sensor drivers to handle the sensor. These sensor drivers can be used in other non-USB devices as well (particularly embedded SoC devices). Moving webcam drivers out of the kernel would likely mean that you end up with duplicate kernel and userspace sensor drivers. Not a good idea.
In all fairness I have to say that most webcam drivers program the sensor directly instead of relying on an i2c driver. Often because they were reverse engineered. But I know of several that should really be split into a webcam driver and a separate i2c sensor driver.
Video hardware tends to rely heavily on i2c devices hooked up to them. The big advantage of keeping everything in the kernel is that that makes i2c driver sharing between USB, PCI, SoC, etc. much easier. Video hardware also often has to toggle between different modes or functionalities (e.g. analog TV vs DVB, MPEG encoded video vs raw). Having all components in the kernel makes this a much easier job.
I think the approach favored by Hans de Goede is the right one based on what I've read.
Whatever solution is chosen, in no way should that ever force us to duplicate i2c sensor control in userspace. This would completely defeat all our efforts to make sensor drivers reusable between webcams and SoCs.
Regards,
Hans
Quoted 28 lines
Which one of these possibile approaches gets adopted is a policy issue which I would consider is ultimately way above my pay grade.
My main motivation for bringing up the issue was to get it to the front burner so that _something_ gets done. It is a matter which has been left alone for too long. Therefore, I am very glad that the matter is being addressed.
Let me add to this that I have gotten permission for time off and for a expense money which might possibly cover my air fare. I hope to arrive in Vancouver by sometime on Monday and intend to attend the mini-summit. I suggest that we get all intersted parties together and figure out what is the best way to go.
I hope everyone who is actively concerned can meet in Vancouver, and if all goes well then on Monday as well as Tuesday. I can hang around for another day or two after Tuesday, but I do not expect to register for LinuxCon or be involved in it. When I leave Vancouver I will probably go to Seattle and spend a couple of days with my oldest son, the musician, before coming home on the next weekend.
Theodore Kilgore
--
To unsubscribe from this list: send the line "unsubscribe linux-media" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 7 lines · Theodore Kilgore
-- moving the kernel webcam drivers out of the kernel and doing with these cameras _everything_ including webcam function through libusb. I myself do not have the imagination to be able to figure out how this could be done without a rather humongous amount of work (for example, which streaming apps that are currently available would be able to live with this?) but unless I misunderstand what he was saying, Greg K-H seems to think that this would be the best thing to do.
No, I never said that. Or if I accidentally did, I do not think that is the best thing to do at all, sorry for any confusion.
Quoted 2 lines
Which one of these possibile approaches gets adopted is a policy issue which I would consider is ultimately way above my pay grade.
As no one is being paid here, there are no "pay grades", so don't worry about that :)
greg k-h
Quoted 11 lines · Greg KH
Quoted 7 lines · Theodore Kilgore
-- moving the kernel webcam drivers out of the kernel and doing with these cameras _everything_ including webcam function through libusb. I myself do not have the imagination to be able to figure out how this could be done without a rather humongous amount of work (for example, which streaming apps that are currently available would be able to live with this?) but unless I misunderstand what he was saying, Greg K-H seems to think that this would be the best thing to do.
No, I never said that. Or if I accidentally did, I do not think that is the best thing to do at all, sorry for any confusion.
OK.
Quoted 5 lines
Quoted 2 lines
Which one of these possibile approaches gets adopted is a policy issue which I would consider is ultimately way above my pay grade.
As no one is being paid here, there are no "pay grades", so don't worry about that :)
I meant this, of course, in a metaphorical sense.
Cheers,
Theodore Kilgore
Em 10-08-2011 15:33, Theodore Kilgore escreveu:
Quoted 13 lines
Hans seems to have argued cogently for doing all of this in the kernel and for abandoning the usbfs-based drivers for these particular drivers for dual-mode cameras and, I would conjecture, for drivers for dual-mode hardware in general. Therefore, I anticipate that he won't like that very much.
My position:
I do not have preconceptions about how the problem gets handled, and at this point I remain agnostic and believe that all approaches ought to be carefully analysed. I can imagine, abstractly, that things like this could be handled by
Quoted 4 lines
-- moving all basic functionality to the kernel, and fixing the relevant libgphoto2 drivers to look to the kernel instead of to libusb. (What Hans argues for, and I am not opposed if his arguments convince other concerned parties)
Not looking on the amount of work to be done, I think that this would give better results, IMO.
Quoted 4 lines
-- doing some kind of patch job to make current arrangement somehow to work better (this seems to be the position of Adam Baker; I do share the skepticism Hans has expressed about how well this could all be pasted together)
Adam Baker's proposal of a locking between usbfs and the kernel driver seems to be interesting, but, as he pointed, there are some side effects to consider, like suspend/resume, PM, etc.
Quoted 3 lines
-- doing something like the previous, but also figuring out how to bring udev rules into play, which would make it all work better (just tossing this one in, for laughs, but who knows someone might like it)
I don't think this is a good alternative.
Quoted 7 lines
-- moving the kernel webcam drivers out of the kernel and doing with these cameras _everything_ including webcam function through libusb. I myself do not have the imagination to be able to figure out how this could be done without a rather humongous amount of work (for example, which streaming apps that are currently available would be able to live with this?) but unless I misunderstand what he was saying, Greg K-H seems to think that this would be the best thing to do.
I also don't think that this a good alternative. As Hans V. pointed, one of our long term targets is to create per-sensor I2C drivers that are independent from the bridges. Also, moving it to userspace would require lots of work with the duplication of V4L and gspca core into userspace for the devices that would be moved, and may have some performance impacts.
Quoted 14 lines
Which one of these possibile approaches gets adopted is a policy issue which I would consider is ultimately way above my pay grade.
My main motivation for bringing up the issue was to get it to the front burner so that _something_ gets done. It is a matter which has been left alone for too long. Therefore, I am very glad that the matter is being addressed.
Let me add to this that I have gotten permission for time off and for a expense money which might possibly cover my air fare. I hope to arrive in Vancouver by sometime on Monday and intend to attend the mini-summit. I suggest that we get all intersted parties together and figure out what is the best way to go.
Quoted 4 lines
I hope everyone who is actively concerned can meet in Vancouver, and if all goes well then on Monday as well as Tuesday. I can hang around for another day or two after Tuesday, but I do not expect to register for LinuxCon or be involved in it.
It will be great to have you there for those discussions.
Quoted 5 lines
When I leave Vancouver I will probably go
to Seattle and spend a couple of days with my oldest son, the musician, before coming home on the next weekend.Theodore Kilgore
Quoted 15 lines · Mauro Carvalho Chehab
Em 10-08-2011 15:33, Theodore Kilgore escreveu:
Quoted 13 lines
Hans seems to have argued cogently for doing all of this in the kernel and for abandoning the usbfs-based drivers for these particular drivers for dual-mode cameras and, I would conjecture, for drivers for dual-mode hardware in general. Therefore, I anticipate that he won't like that very much.
My position:
I do not have preconceptions about how the problem gets handled, and at this point I remain agnostic and believe that all approaches ought to be carefully analysed. I can imagine, abstractly, that things like this could be handled by
Quoted 7 lines
Quoted 4 lines
-- moving all basic functionality to the kernel, and fixing the relevant libgphoto2 drivers to look to the kernel instead of to libusb. (What Hans argues for, and I am not opposed if his arguments convince other concerned parties)
Not looking on the amount of work to be done, I think that this would give better results, IMO.
Okay. I would guess that I am one of the guys who gets to do the work, though.
Quoted 10 lines
Quoted 5 lines
-- doing some kind of patch job to make current arrangement somehow to work better (this seems to be the position of Adam Baker; I do share the skepticism Hans has expressed about how well this could all be pasted together)
Adam Baker's proposal of a locking between usbfs and the kernel driver seems to be interesting, but, as he pointed, there are some side effects to consider, like suspend/resume, PM, etc.
Quoted 5 lines
Quoted 3 lines
-- doing something like the previous, but also figuring out how to bring udev rules into play, which would make it all work better (just tossing this one in, for laughs, but who knows someone might like it)
I don't think this is a good alternative.
I was trying to mention all alternatives. I should have also mentioned "leave things the way they are" but that is certainly out the window.
Quoted 9 lines
Quoted 8 lines
-- moving the kernel webcam drivers out of the kernel and doing with these cameras _everything_ including webcam function through libusb. I myself do not have the imagination to be able to figure out how this could be done without a rather humongous amount of work (for example, which streaming apps that are currently available would be able to live with this?) but unless I misunderstand what he was saying, Greg K-H seems to think that this would be the best thing to do.
Quoted 5 lines
I also don't think that this a good alternative. As Hans V. pointed, one of our long term targets is to create per-sensor I2C drivers that are independent from the bridges. Also, moving it to userspace would require lots of work with the duplication of V4L and gspca core into userspace for the devices that would be moved, and may have some performance impacts.
A good argument, though it probably does not affect the devices on my list one way or the other.
Quoted 15 lines
Quoted 15 lines
Which one of these possibile approaches gets adopted is a policy issue which I would consider is ultimately way above my pay grade.
My main motivation for bringing up the issue was to get it to the front burner so that _something_ gets done. It is a matter which has been left alone for too long. Therefore, I am very glad that the matter is being addressed.
Let me add to this that I have gotten permission for time off and for a expense money which might possibly cover my air fare. I hope to arrive in Vancouver by sometime on Monday and intend to attend the mini-summit. I suggest that we get all intersted parties together and figure out what is the best way to go.
Quoted 6 lines
Quoted 4 lines
I hope everyone who is actively concerned can meet in Vancouver, and if all goes well then on Monday as well as Tuesday. I can hang around for another day or two after Tuesday, but I do not expect to register for LinuxCon or be involved in it.
It will be great to have you there for those discussions.
My take on this was that it seems to have become important for me to attend, which, frankly, I was not expecting. So thanks for the nice words.
Quoted 7 lines
Quoted 6 lines
When I leave Vancouver I will probably go
to Seattle and spend a couple of days with my oldest son, the musician, before coming home on the next weekend.Theodore Kilgore
Quoted 8 lines · On Wed, 10 Aug 2011 17:14:05 -0300 Mauro Carvalho Chehab…
Quoted 7 lines
-- moving the kernel webcam drivers out of the kernel and doing with these cameras _everything_ including webcam function through libusb. I myself do not have the imagination to be able to figure out how this could be done without a rather humongous amount of work (for example, which streaming apps that are currently available would be able to live with this?) but unless I misunderstand what he was saying, Greg K-H seems to think that this would be the best thing to do.
Quoted 5 lines
I also don't think that this a good alternative. As Hans V. pointed, one of our long term targets is to create per-sensor I2C drivers that are independent from the bridges. Also, moving it to userspace would require lots of work with the duplication of V4L and gspca core into userspace for the devices that would be moved, and may have some performance impacts.
If only gspca subdrivers are concerned, moving them to usespace should be easy: only the gspca main has to be rewritten, and its best place is in the v4l library.
Signature · 2 lines
Ken ar c'hentañ | ** Breizh ha Linux atav! ** Jef | http://moinejf.free.fr/
Em 11-08-2011 05:57, Jean-Francois Moine escreveu:
Quoted 17 lines
Quoted 8 lines · On Wed, 10 Aug 2011 17:14:05 -0300 Mauro Carvalho Chehab…
Quoted 7 lines
-- moving the kernel webcam drivers out of the kernel and doing with these cameras _everything_ including webcam function through libusb. I myself do not have the imagination to be able to figure out how this could be done without a rather humongous amount of work (for example, which streaming apps that are currently available would be able to live with this?) but unless I misunderstand what he was saying, Greg K-H seems to think that this would be the best thing to do.
Quoted 5 lines
I also don't think that this a good alternative. As Hans V. pointed, one of our long term targets is to create per-sensor I2C drivers that are independent from the bridges. Also, moving it to userspace would require lots of work with the duplication of V4L and gspca core into userspace for the devices that would be moved, and may have some performance impacts.
Quoted 3 lines
If only gspca subdrivers are concerned, moving them to usespace should be easy: only the gspca main has to be rewritten, and its best place is in the v4l library.
It would end by having two userspace drivers, as the data access driver is at libgphoto2. So, the problem will just be moved to userspace. You'll additionally need to deal with conflicts between kernelspace ALSA driver and userspace video driver.
Regards,
Mauro.
Hi,
Quoted 33 lines · Alan Stern
Quoted 15 lines · Theodore Kilgore
Quoted 2 lines
Okay, I didn't realize that the different cameras used different webcam drivers as well as different stillcam drivers.
Oh, yes. They are Proprietary devices. And that means what it says. :-) And all different from each other, too.
Quoted 6 lines
As far as I can see, there's nothing to stop anybody from adding the stillcam functionality into the webcam drivers right now. If some common code can be abstracted out into a shared source file, so much the better.
That would solve the problem, right?
I think everyone involved believes that it would solve the problem.
Quoted 9 lines
The question has been all along whether or not there is any other way which would work. Also the question of what, exactly, "belongs" in the kernel and what does not. For, if something has been historically supported in userspace (stillcam support, in this case) and has worked well there, I would think it is kind of too bad to have to move said support into the kernel just because the same hardware requires kernel support for another functionality and the two sides clash. I mean, the kernel is already big enough, no? But the logic that Hans has set forth seems rather compelling.
The alternative seems to be to define a device-sharing protocol for USB drivers. Kernel drivers would implement a new callback (asking them to give up control of the device), and usbfs would implement new ioctls by which a program could ask for and relinquish control of a device. The amount of rewriting needed would be relatively small.
Quoted 6 lines
A few loose ends would remain, such as how to handle suspends, resumes, resets, and disconnects. Assuming usbfs is the only driver that will want to share a device in this way, we could handle them.
Hans, what do you think?
First of all thanks for the constructive input!
When you say: "device-sharing protocol", do you mean 2 drivers been attached, but only 1 being active. Or just some additional glue to make hand-over between them work better?
I've 2 concerns with this approach:
1) Assuming we are going for the just make hand over work better solution we will still have the whole disappear / reappear act of the /dev/video# node, which I don't like at all.
If for example skype gets started it will say the user has no camera. If it were to say the device is busy, the user just might make a link to some application using the device in stillcam mode still being open.
2) The whole notion of the device being in use is rather vague when it comes to the userspace parts of this. Simply leaving say F-Spot running, or having a gvfs libgphoto share mounted, should not lead to not being able to use the device in webcam mode. But currently it will.
Fixing all users of libgphoto2 wrt this is unlikely to happen, and even if we do that now, more broken ones will likely come along later. I estimate 98% of all cameras are not dual mode cameras, so the average stillcam application developer will not test for this.
That leaves us with fixing the busy notion inside libgphoto2, iow, release the device as soon as an operation has completed. This will be quite slow, since both drivers don't know anything about each other, they will just know there is some $random_other_driver. So they need to assume the device state is unclean and re-init the device from scratch each time.
Where as if we have both functions in one driver, that can remember the actual device state and only make changes if needed, so downloading + deleting 10 photos will lead to setting it to stillcam mode once, rather then 20 times.
Regards,
Hans
Quoted 10 lines · Hans de Goede
Quoted 10 lines
The alternative seems to be to define a device-sharing protocol for USB drivers. Kernel drivers would implement a new callback (asking them to give up control of the device), and usbfs would implement new ioctls by which a program could ask for and relinquish control of a device. The amount of rewriting needed would be relatively small.
A few loose ends would remain, such as how to handle suspends, resumes, resets, and disconnects. Assuming usbfs is the only driver that will want to share a device in this way, we could handle them.
Quoted 8 lines
Quoted 2 lines
Hans, what do you think?
First of all thanks for the constructive input!
When you say: "device-sharing protocol", do you mean 2 drivers been attached, but only 1 being active. Or just some additional glue to make hand-over between them work better?
I was thinking that the webcam driver would always be attached, but from time to time usbfs would ask to use the device. When the webcam driver gives away control, it remains bound to the device but does not send any URBs. If it needs to send an URB, first it has to ask usbfs to give control back.
Quoted 4 lines
I've 2 concerns with this approach:
1) Assuming we are going for the just make hand over work better solution we will still have the whole disappear / reappear act of the /dev/video# node, which I don't like at all.
That will not happen any more, because the webcam driver will always be bound to the device.
Quoted 8 lines
If for example skype gets started it will say the user has no camera. If it were to say the device is busy, the user just might make a link to some application using the device in stillcam mode still being open.
2) The whole notion of the device being in use is rather vague when it comes to the userspace parts of this. Simply leaving say F-Spot running, or having a gvfs libgphoto share mounted, should not lead to not being able to use the device in webcam mode. But currently it will.
That's true -- but it's true no matter what solution we adopt. The various drivers (whether in the kernel or in userspace) will have to decide for themselves when they can give up control.
Quoted 4 lines
Fixing all users of libgphoto2 wrt this is unlikely to happen, and even if we do that now, more broken ones will likely come along later. I estimate 98% of all cameras are not dual mode cameras, so the average stillcam application developer will not test for this.
Not all users of libgphoto2 have to be changed; only those which manage dual-mode cameras. Adding a few ioctls to ask for and give up control at the appropriate times must be a lot easier than porting the entire driver into the kernel.
Quoted 5 lines
That leaves us with fixing the busy notion inside libgphoto2, iow, release the device as soon as an operation has completed. This will be quite slow, since both drivers don't know anything about each other, they will just know there is some $random_other_driver. So they need to assume the device state is unclean and re-init the device from scratch each time.
Well, a user program can assume that the kernel driver left the device in a clean state. The reverse isn't always true, however -- it's one of the drawbacks of using a userspace driver.
Besides, even though drivers don't always have to re-init the device from scratch every time, they do send all the current settings each time they use the device. So maybe the extra overhead is tolerable.
Quoted 4 lines
Where as if we have both functions in one driver, that can remember the actual device state and only make changes if needed, so downloading + deleting 10 photos will lead to setting it to stillcam mode once, rather then 20 times.
Depends how the ask-for-control ioctl is implemented. It might return a value indicating whether or not the webcam driver took control during the interval when the user program wasn't using the device. If usbfs retained control the entire time, the program should be able to assume the device's state hasn't changed.
I'm not claiming that this is a better solution than putting everything in the kernel. Just that it is a workable alternative which would involve a lot less coding.
Alan Stern
Em 11-08-2011 11:56, Alan Stern escreveu:
Quoted 39 lines · Hans de Goede
Quoted 11 lines
Quoted 11 lines
The alternative seems to be to define a device-sharing protocol for USB drivers. Kernel drivers would implement a new callback (asking them to give up control of the device), and usbfs would implement new ioctls by which a program could ask for and relinquish control of a device. The amount of rewriting needed would be relatively small.
A few loose ends would remain, such as how to handle suspends, resumes, resets, and disconnects. Assuming usbfs is the only driver that will want to share a device in this way, we could handle them.
Quoted 8 lines
Quoted 2 lines
Hans, what do you think?
First of all thanks for the constructive input!
When you say: "device-sharing protocol", do you mean 2 drivers been attached, but only 1 being active. Or just some additional glue to make hand-over between them work better?
I was thinking that the webcam driver would always be attached, but from time to time usbfs would ask to use the device. When the webcam driver gives away control, it remains bound to the device but does not send any URBs. If it needs to send an URB, first it has to ask usbfs to give control back.
Quoted 4 lines
I've 2 concerns with this approach:
1) Assuming we are going for the just make hand over work better solution we will still have the whole disappear / reappear act of the /dev/video# node, which I don't like at all.That will not happen any more, because the webcam driver will always be bound to the device.
Quoted 4 lines
If for example skype gets started it will say the user has no camera. If it were to say the device is busy, the user just might make a link to some application using the device in stillcam mode still being open.
Quoted 8 lines
Quoted 4 lines
2) The whole notion of the device being in use is rather vague when it comes to the userspace parts of this. Simply leaving say F-Spot running, or having a gvfs libgphoto share mounted, should not lead to not being able to use the device in webcam mode. But currently it will.
That's true -- but it's true no matter what solution we adopt. The various drivers (whether in the kernel or in userspace) will have to decide for themselves when they can give up control.
Between two or more kernel drivers, a resource locking mechanism like the one you've proposed works fine, but, when the driver is on userspace, there's one additional issue that needs to be addressed: What happens if, for example, if a camera application using libgphoto2 crashes? The lock will be enabled, and the V4L driver will be locked forever. Also, if the lock is made generic enough to protect between two different userspace applications, re-starting the camera application won't get the control back.
To avoid such risk, kernel might need to implement some ugly hacks to detect when the application was killed, and put the device into a sane state, if this happens.
Quoted 9 lines
Quoted 4 lines
Fixing all users of libgphoto2 wrt this is unlikely to happen, and even if we do that now, more broken ones will likely come along later. I estimate 98% of all cameras are not dual mode cameras, so the average stillcam application developer will not test for this.
Not all users of libgphoto2 have to be changed; only those which manage dual-mode cameras. Adding a few ioctls to ask for and give up control at the appropriate times must be a lot easier than porting the entire driver into the kernel.
Again, applications that won't implement this control will take the lock forever.
Quoted 31 lines
Quoted 5 lines
That leaves us with fixing the busy notion inside libgphoto2, iow, release the device as soon as an operation has completed. This will be quite slow, since both drivers don't know anything about each other, they will just know there is some $random_other_driver. So they need to assume the device state is unclean and re-init the device from scratch each time.
Well, a user program can assume that the kernel driver left the device in a clean state. The reverse isn't always true, however -- it's one of the drawbacks of using a userspace driver.
Besides, even though drivers don't always have to re-init the device from scratch every time, they do send all the current settings each time they use the device. So maybe the extra overhead is tolerable.
Quoted 4 lines
Where as if we have both functions in one driver, that can remember the actual device state and only make changes if needed, so downloading + deleting 10 photos will lead to setting it to stillcam mode once, rather then 20 times.
Depends how the ask-for-control ioctl is implemented. It might return a value indicating whether or not the webcam driver took control during the interval when the user program wasn't using the device. If usbfs retained control the entire time, the program should be able to assume the device's state hasn't changed.
I'm not claiming that this is a better solution than putting everything in the kernel. Just that it is a workable alternative which would involve a lot less coding.
Alan Stern
Quoted 7 lines
Between two or more kernel drivers, a resource locking mechanism like the one you've proposed works fine, but, when the driver is on userspace, there's one additional issue that needs to be addressed: What happens if, for example, if a camera application using libgphoto2 crashes? The lock will be enabled, and the V4L driver will be locked forever. Also, if the lock is made generic enough to protect between two different userspace applications, re-starting the camera application won't get the control back.
Actually there are more issues than that - you've also got to worry about a security/permission model, and that is hard to get right, especially if you are not very careful that anything that can be retrieved which might violate the security model (eg the last frame on the capture) has been blanked before handover etc.
Quoted 3 lines
To avoid such risk, kernel might need to implement some ugly hacks to detect when the application was killed, and put the device into a sane state, if this happens.
At which point history says it's easier to do the job right, once, in the kernel.
Quoted 1 line
Again, applications that won't implement this control will take the lock forever.
And applications that are touching both video (even indirectly) and still camera may get surprise deadlocks if they accidentally reference both the still and video device even via some library or service.
Quoted 2 lines
Quoted 2 lines
Well, a user program can assume that the kernel driver left the device in a clean state. The reverse isn't always true, however -- it's one
Not it cannot - the user program doesn't know
a) if the kernel driver has ever been loaded
b) the values the kernel driver leaves set (and those will change no doubt at times)
c) if the camera has been plugged and unplugged and not yet had the kernel driver loaded
To me it sounds like a recipe for disaster. For those tiny number of devices involved just use V4L and if need be some small V4L tweaks to handle still mode. In most cases the interface is basically identical and I'd bet much of the code is identical too.
Alan
Quoted 5 lines · Alan Cox
Actually there are more issues than that - you've also got to worry about a
security/permissionmodel, and that is hard to get right, especially if you are not very careful that anything that can be retrieved which might violate the security model (eg the last frame on the capture) has been blanked before handover etc.
As far as I can tell, these same security issues exist today. I don't see them getting any worse than they are now.
Quoted 3 lines
And applications that are touching both video (even indirectly) and still camera may get surprise deadlocks if they accidentally reference both the still and video device even via some library or service.
No, not deadlocks. Just -EBUSY errors.
Quoted 10 lines
Quoted 2 lines
Quoted 2 lines
Well, a user program can assume that the kernel driver left the device in a clean state. The reverse isn't always true, however -- it's one
Not it cannot - the user program doesn't know
a) if the kernel driver has ever been loaded
b) the values the kernel driver leaves set (and those will change no doubt at times)
c) if the camera has been plugged and unplugged and not yet had the kernel driver loaded
That's true. The program can't assume that a kernel driver was ever bound to the device; all it can assume is that _if_ a kernel driver was bound then it left the device in a sane state -- whatever "sane" might mean in this context.
Quoted 4 lines
To me it sounds like a recipe for disaster. For those tiny number of devices involved just use V4L and if need be some small V4L tweaks to handle still mode. In most cases the interface is basically identical and I'd bet much of the code is identical too.
I'm not against moving the whole thing into the kernel. I'm just pointing out that an easier-to-code-up solution will accomplish much the same result.
Alan Stern
Quoted 38 lines · Alan Stern
Quoted 5 lines · Alan Cox
Actually there are more issues than that - you've also got to worry about a
security/permissionmodel, and that is hard to get right, especially if you are not very careful that anything that can be retrieved which might violate the security model (eg the last frame on the capture) has been blanked before handover etc.As far as I can tell, these same security issues exist today. I don't see them getting any worse than they are now.
Quoted 3 lines
And applications that are touching both video (even indirectly) and still camera may get surprise deadlocks if they accidentally reference both the still and video device even via some library or service.
No, not deadlocks. Just
-EBUSYerrors.Quoted 10 lines
Quoted 2 lines
Quoted 2 lines
Well, a user program can assume that the kernel driver left the device in a clean state. The reverse isn't always true, however -- it's one
Not it cannot - the user program doesn't know
a) if the kernel driver has ever been loaded
b) the values the kernel driver leaves set (and those will change no doubt at times)
c) if the camera has been plugged and unplugged and not yet had the kernel driver loadedThat's true. The program can't assume that a kernel driver was ever bound to the device; all it can assume is that _if_ a kernel driver was bound then it left the device in a sane state -- whatever "sane" might mean in this context.
Quoted 4 lines
To me it sounds like a recipe for disaster. For those tiny number of devices involved just use V4L and if need be some small V4L tweaks to handle still mode. In most cases the interface is basically identical and I'd bet much of the code is identical too.
Quoted 6 lines
I'm not against moving the whole thing into the kernel. I'm just pointing out that an easier-to-code-up solution will accomplish much the same result.
Alan Stern
Alan,
As I said, I am agnostic, though leaning in the direction that Hans de Goede is pointing. What he says about a single control mechanism seems to make a lot of sense. If you can come up with an outline of the "easier to code" solution, that would be interesting, though.
I assume you are also going to be in Vancouver? If you will be there on Monday, then Hans and I are already planning to meet and discuss.
BTW, as to using V4L with "tweaks" to handle still mode, it would probably be more difficult than is imagined. For, though the operations required to process still images and webcam frames are in principle similar, the priorities and constraints are too different. Therefore, my understanding is that the libgphoto2 image processing routines, not the libv4l image processing routines, would still be used for still images.
Theodore Kilgore
Em 11-08-2011 17:01, Theodore Kilgore escreveu:
Quoted 8 lines
As I said, I am agnostic, though leaning in the direction that Hans de Goede is pointing. What he says about a single control mechanism seems to make a lot of sense. If you can come up with an outline of the "easier to code" solution, that would be interesting, though.
I assume you are also going to be in Vancouver? If you will be there on Monday, then Hans and I are already planning to meet and discuss.
Quoted 6 lines
BTW, as to using V4L with "tweaks" to handle still mode, it would probably be more difficult than is imagined. For, though the operations required to process still images and webcam frames are in principle similar, the priorities and constraints are too different. Therefore, my understanding is that the libgphoto2 image processing routines, not the libv4l image processing routines, would still be used for still images.
I agree with Alan Cox: most of the code that the driver needs is already there:
register read/write routines, bulk transfer support, etc. The amount of extra code for adding still cam functionality is probably not big.
Quoted 1 line
From the kernel driver's perspective, it doesn't matter if the access will come
via libv4l, libgphoto2 or whatever. The driver should be able to allow simultaneous open, while protecting the data access when userspace requests data stream or still image retrieve.
instead of using the V4L2 device node to access the stored images, it probably makes more sense to use a separate device for that, that will handle a separate set of ioctl's, and just use read() to retrieve the image data, after selecting the desired image number, via ioctl().
It probably makes sense to add a new set of callbacks at the gspca core in order to handle the new device node, and letting it to avoid start streaming while the store access is happening, and vice-versa. Alternatively, we may create a separate "still cam" core library to handle the new device node,.
If all agree around such solution, I suggest to take the most complex case and try to map it into the driver and core, and see how it behaves, testing with some simple command line applications, only changing the libgphoto2 code after those initial tests. Writing a simple code for reading still images should be easy, and we have already some testing tools for V4L2.
After coding the core changes that are common to all drives, I suspect that adding the remaining 4 drivers will be quick.
With regards to libgphoto2, all it needs to do is to test if the new device nodes exist. If they exist, then the new code will be used. Otherwise, it will fallback to the libusb. This way, we can incrementally add the Dual mode drivers into the kernel.
There is one advantage on using this strategy: if, in the future, new Dual Cams arise, one can write first a still cam kernel driver, adding V4L support later.
Cheers,
Mauro
Quoted 50 lines · Mauro Carvalho Chehab
Em 11-08-2011 17:01, Theodore Kilgore escreveu:
Quoted 8 lines
As I said, I am agnostic, though leaning in the direction that Hans de Goede is pointing. What he says about a single control mechanism seems to make a lot of sense. If you can come up with an outline of the "easier to code" solution, that would be interesting, though.
I assume you are also going to be in Vancouver? If you will be there on Monday, then Hans and I are already planning to meet and discuss.
Quoted 6 lines
BTW, as to using V4L with "tweaks" to handle still mode, it would probably be more difficult than is imagined. For, though the operations required to process still images and webcam frames are in principle similar, the priorities and constraints are too different. Therefore, my understanding is that the libgphoto2 image processing routines, not the libv4l image processing routines, would still be used for still images.
I agree with Alan Cox: most of the code that the driver needs is already there:
register read/write routines, bulk transfer support, etc. The amount of extra code for adding still cam functionality is probably not big.Quoted 1 line
From the kernel driver's perspective, it doesn't matter if the access will come
via libv4l, libgphoto2 or whatever. The driver should be able to allow simultaneous open, while protecting the data access when userspace requests data stream or still image retrieve.
instead of using the V4L2 device node to access the stored images, it probably makes more sense to use a separate device for that, that will handle a separate set of ioctl's, and just use
read()to retrieve the image data, after selecting the desired image number, viaioctl().It probably makes sense to add a new set of callbacks at the gspca core in order to handle the new device node, and letting it to avoid start streaming while the store access is happening, and vice-versa. Alternatively, we may create a separate "still cam" core library to handle the new device node,.
If all agree around such solution, I suggest to take the most complex case and try to map it into the driver and core, and see how it behaves, testing with some simple command line applications, only changing the libgphoto2 code after those initial tests. Writing a simple code for reading still images should be easy, and we have already some testing tools for V4L2.
After coding the core changes that are common to all drives, I suspect that adding the remaining 4 drivers will be quick.
With regards to libgphoto2, all it needs to do is to test if the new device nodes exist. If they exist, then the new code will be used. Otherwise, it will fallback to the libusb. This way, we can incrementally add the Dual mode drivers into the kernel.
Quoted 6 lines
There is one advantage on using this strategy: if, in the future, new Dual Cams arise, one can write first a still cam kernel driver, adding V4L support later.
Cheers,
Mauro
Try it out with one camera seems to me to be a reasonable way forward.
Theodore Kilgore
Hi,
On 08/11/2011 10:32 PM, Mauro Carvalho Chehab wrote:
<snip stuff I agree with>
Quoted 4 lines
instead of using the V4L2 device node to access the stored images, it probably makes more sense to use a separate device for that, that will handle a separate set of ioctl's, and just use
read()to retrieve the image data, after selecting the desired image number, viaioctl().
I don't see a lot of added value in doing things this way. We can simply have a set of new ioctls on the /dev/video# node for this and a new V4L2_CAP_STILL_IMAGE to indicate the availability of these ioctls. This will keep the driver a lot simpler then doing 2 separate device nodes for 1 device.
Regards,
Hans
Quoted 4 lines
instead of using the V4L2 device node to access the stored images, it probably makes more sense to use a separate device for that, that will handle a separate set of ioctl's, and just use
read()to retrieve the image data, after selecting the desired image number, viaioctl().
How will you handle the permission model, what about devices where you need the V4L2 controls to take the picture in the first place ?
It seems it should really be the same device and just a minor tweak to the API. V4L2 already provides frame reading and mapping functionality and the media layer provides post processing services which I could conceive some setups needing to use.
gphoto would just open the v4l device ask for still image stuff and discover it wasn't available.
Quoted 6 lines · Theodore Kilgore
Alan,
As I said, I am agnostic, though leaning in the direction that Hans de Goede is pointing. What he says about a single control mechanism seems to make a lot of sense. If you can come up with an outline of the "easier to code" solution, that would be interesting, though.
That approach has been outlined in the other emails I have posted recently.
Quoted 2 lines
I assume you are also going to be in Vancouver? If you will be there on Monday, then Hans and I are already planning to meet and discuss.
No, I'm not going to Vancouver. However I will attend the Linux Plumbers conference in Santa Rosa.
Alan Stern
On Thu, 11 Aug 2011, Alan Stern wrote:
<snip>
Quoted 1 line
No, I'm not going to Vancouver.
Well, someday, somewhere, we will be together in the same place at the same time and able to enjoy a good seafood restaurant.
However I will attend the Linux
Quoted 1 line
Plumbers conference in Santa Rosa.
Which I expect I will not.
Until next time we meet, cheers.
Theodore Kilgore
Alan Cox raised a bunch of good points. I'm not going to respond to them; they pretty much speak for themselves.
Quoted 2 lines · Mauro Carvalho Chehab
Between two or more kernel drivers, a resource locking mechanism like the one you've proposed works fine,
It's not a locking mechanism. More like cooperative multi-access.
Quoted 6 lines
but, when the driver is on userspace, there's one
additional issue that needs to be addressed: What happens if, for example, if a camera application using libgphoto2 crashes? The lock will be enabled, and the V4L driver will be locked forever. Also, if the lock is made generic enough to protect between two different userspace applications, re-starting the camera application won't get the control back.
You're wrong, because what I proposed isn't a lock. If the user program dies, the usbfs device file will automatically be closed and then usbfs will freely give control back to the webcam driver.
If the program hangs at the wrong time, then the webcam driver won't be able to regain control. At least the user will have the option of killing the program when this happens; a similar hang in a kernel driver tends to be much uglier.
Two different user programs trying to drive the device at the same time doesn't necessarily have to cause a problem. At any particular moment, only one of them would be in control of the device.
Quoted 3 lines
To avoid such risk, kernel might need to implement some ugly hacks to detect when the application was killed, and put the device into a sane state, if this happens.
No ugly hack is needed. usbfs can directly inform the webcam driver whether or not the device file was closed while the user program retained control. If that didn't happen, it's safe to assume that the program gave up control voluntarily.
Quoted 6 lines
Quoted 4 lines
Not all users of libgphoto2 have to be changed; only those which manage dual-mode cameras. Adding a few ioctls to ask for and give up control at the appropriate times must be a lot easier than porting the entire driver into the kernel.
Again, applications that won't implement this control will take the lock forever.
No, because there is no lock. Programs that haven't been changed will continue to behave as they do today -- they will unbind the webcam driver and assume full control of the device.
Alan Stern
Hi,
Quoted 21 lines · Alan Stern
Quoted 10 lines · Hans de Goede
Quoted 10 lines
The alternative seems to be to define a device-sharing protocol for USB drivers. Kernel drivers would implement a new callback (asking them to give up control of the device), and usbfs would implement new ioctls by which a program could ask for and relinquish control of a device. The amount of rewriting needed would be relatively small.
A few loose ends would remain, such as how to handle suspends, resumes, resets, and disconnects. Assuming usbfs is the only driver that will want to share a device in this way, we could handle them.
Quoted 8 lines
Quoted 2 lines
Hans, what do you think?
First of all thanks for the constructive input!
When you say: "device-sharing protocol", do you mean 2 drivers been attached, but only 1 being active. Or just some additional glue to make hand-over between them work better?
Quoted 6 lines
I was thinking that the webcam driver would always be attached, but from time to time usbfs would ask to use the device. When the webcam driver gives away control, it remains bound to the device but does not send any URBs. If it needs to send an URB, first it has to ask usbfs to give control back.
Oh, interesting...
<snip lots of good stuff>
Quoted 3 lines
I'm not claiming that this is a better solution than putting everything in the kernel. Just that it is a workable alternative which would involve a lot less coding.
This is definitely an interesting proposal, something to think about ...
I have 2 concerns wrt this approach:
1) It feels less clean then just having a single driver; and
2) I agree it will be less coding, but I doubt it will really be that much less work. It will likely need less new code (but a lot can be more or less copy pasted), but it will need changes across a wider array of subsystems / userspace components, requiring a lot of coordinating, getting patches merged in different projects, etc. So in the end I think it too will be quite a bit of work.
I guess that what I'm trying to say here is, that if we are going to spend a significant amount of time on this, we might just as well go for the best solution we can come up with even if that is some more work.
Regards,
Hans
Quoted 9 lines · Hans de Goede
Quoted 3 lines
I'm not claiming that this is a better solution than putting everything in the kernel. Just that it is a workable alternative which would involve a lot less coding.
This is definitely an interesting proposal, something to think about ...
I have 2 concerns wrt this approach:
1) It feels less clean then just having a single driver; and
Agreed.
Quoted 11 lines
2) I agree it will be less coding, but I doubt it will really be that much less work. It will likely need less new code (but a lot can be more or less copy pasted), but it will need changes across a wider array of subsystems / userspace components, requiring a lot of coordinating, getting patches merged in different projects, etc. So in the end I think it too will be quite a bit of work.
I guess that what I'm trying to say here is, that if we are going to spend a significant amount of time on this, we might just as well go for the best solution we can come up with even if that is some more work.
Okay, go ahead. I have no objection.
Alan Stern
Quoted 44 lines · Hans de Goede
Hi,
Quoted 24 lines · Sarah Sharp
Quoted 11 lines · Hans de Goede
Hi,
Quoted 1 line · Greg KH
On Thu, Aug 04, 2011 at 07:21:47PM -0300, Mauro Carvalho Chehab wrote:
I think it is important to separate oranges from apples here, there are at least 3 different problem classes which all seem to have gotten thrown onto a pile here:
1) The reason Mauro suggested having some discussion on this at the USB summit is because of a discussion about dual mode cameras on the linux media list.
...
Quoted 4 lines
3) Re-direction of usb devices to virtual machines. This works by using the userspace usbfs interface from qemu / vmware / virtualbox / whatever. The basics of this work fine, but it lacks proper locking / safeguards for when a vm takes over a usb device from the in kernel driver.
Hi Hans and Mauro,
We have do room in the schedule for the USB mini-summit for this discussion, since the schedule is still pretty flexible. The preliminary schedule is up here:
Quoted 6 lines
http://userweb.kernel.org/~sarah/linuxcon-usb-minisummit.html
I think it's best to discuss the VM redirection in the afternoon when some of the KVM folks join us after Hans' talk on USB redirection over the network.
That seems best to me too. I'm available at both proposed time slots, and I would like to join both discussions.
Quoted 6 lines
It sounds like we need a separate topic for the dual mode cameras and TV tuners. Mauro, do you want to lead that discussion in the early morning (in a 9:30 to 10:30 slot) or in the late afternoon (in a 15:30 to 16:30 slot)? I want to be sure we have all the video/media developers who are interested in this topic present, and I don't know if they will be going to the KVM forum.
Quoted 2 lines
I would really like to see the dual mode camera and TV tuner discussion separated. They are 2 different issues AFAIK.
Ok, great, I've put both topics in the morning, in separate slots.
Sarah Sharp