Simple userspace interface for PCI drivers

So, here's the code. I think it does a bit too much all at once, but it is an example of how this can be done. This is working today in some industrial environments, successfully handling hardware controls of very large and scary machines. So it is possible to use this interface to successfully build your own laser wielding robot, all from userspace, allowing you to keep your special-secret-how-to-control-the-laser algorithm closed source if you so desire.

In looking at the proposed kevent interface, I think a few things in this proposed interface can be dropped in favor of using kevents instead, but I haven't looked at the latest version of that code to make sure of this.

And the name is a bit ackward, anyone have a better suggestion?

Thomas has also promised to come up with some userspace code that uses this interface to show how to use it, but seems to have forgotten. Consider this a reminder :)

A while ago, Thomas and I were sitting in the back of a conference presentation where the presenter was trying to describe what they did in order to add the ability to write a userspace PCI driver. As was usual in a presentation like this, the presenter totaly ignored the real-world needs for such a framework, and only got it up and working on a single type of embedded system. But the charts and graphs were quite pretty :)

Thomas and I lamented that we were getting tired of seeing stuff like this, and it was about time that we added the proper code to the kernel to provide everything that would be needed in order to write PCI drivers in userspace in a sane manner. Really all that is needed is a way to handle the interrupt, everything else can already be done in userspace (look at X for an example of this...)

Thomas mentioned that he had code to do all of this working in some customer sites already and that he would get it to me.

Fast forward to OLS of this year, and I bugged Thomas to send me the code. He did, and then I sat on it for a while longer...

So, here's the code. I think it does a bit too much all at once, but it is an example of how this can be done. This is working today in some industrial environments, successfully handling hardware controls of very large and scary machines. So it is possible to use this interface to successfully build your own laser wielding robot, all from userspace, allowing you to keep your special-secret-how-to-control-the-laser algorithm closed source if you so desire.

In looking at the proposed kevent interface, I think a few things in this proposed interface can be dropped in favor of using kevents instead, but I haven't looked at the latest version of that code to make sure of this.

And the name is a bit ackward, anyone have a better suggestion?

Thomas has also promised to come up with some userspace code that uses this interface to show how to use it, but seems to have forgotten. Consider this a reminder :)

Quoted 6 lines · Greg KH

Thomas and I lamented that we were getting tired of seeing stuff like this, and it was about time that we added the proper code to the kernel to provide everything that would be needed in order to write PCI drivers in userspace in a sane manner. Really all that is needed is a way to handle the interrupt, everything else can already be done in userspace (look at X for an example of this...)

What about portable access to the PCI DMA API from userspace? I didn't think we had a way to handle DMA buffer management entirely from userspace. Looking at this code, it seems focused on one particular problem space of industrial control. If the goal is to allow most PCI drivers to be implemented in user space then I think it makes further sense to not limit the subsystem to PCI but any device (like things that are implemented as platform devices in kernel space).

Quoted 1 line

And the name is a bit ackward, anyone have a better suggestion?

Well, if it's focused on industrial controls like it appears from the code here then the name is fine. If it's a starting point to become someting more generic then User Space Driver (USD) subsystem might be nice.

A more generic system would go a long way to get rid of the dubious secret-sauce binary kernel modules that are popular in embedded linux projects.

Quoted 3 lines

Thomas has also promised to come up with some userspace code that uses this interface to show how to use it, but seems to have forgotten. Consider this a reminder :)

That would be nice to see. I can't see how devices and drivers are registered from user space with what's here. I see iio_register_device() exported but no clue how userspace tells the kernel to claim a device or register an iio driver.

Quoted 7 lines · Greg KH

A while ago, Thomas and I were sitting in the back of a conference presentation where the presenter was trying to describe what they did in order to add the ability to write a userspace PCI driver. As was usual in a presentation like this, the presenter totaly ignored the real-world needs for such a framework, and only got it up and working on a single type of embedded system. But the charts and graphs were quite pretty :)

Quoted 6 lines

Thomas and I lamented that we were getting tired of seeing stuff like this, and it was about time that we added the proper code to the kernel to provide everything that would be needed in order to write PCI drivers in userspace in a sane manner. Really all that is needed is a way to handle the interrupt, everything else can already be done in userspace (look at X for an example of this...)

What about portable access to the PCI DMA API from userspace?

Quoted 14 lines

Thomas mentioned that he had code to do all of this working in some customer sites already and that he would get it to me.

Fast forward to OLS of this year, and I bugged Thomas to send me the code. He did, and then I sat on it for a while longer...

So, here's the code. I think it does a bit too much all at once, but it is an example of how this can be done. This is working today in some industrial environments, successfully handling hardware controls of very large and scary machines. So it is possible to use this interface to successfully build your own laser wielding robot, all from userspace, allowing you to keep your special-secret-how-to-control-the-laser algorithm closed source if you so desire.

Quoted 6 lines

In looking at the proposed kevent interface, I think a few things in this proposed interface can be dropped in favor of using kevents instead, but I haven't looked at the latest version of that code to make sure of this.

And the name is a bit ackward, anyone have a better suggestion?

Well, if it's focused on industrial controls like it appears from the code here then the name is fine. If it's a starting point to become someting more generic then User Space Driver (USD) subsystem might be nice.

A more generic system would go a long way to get rid of the dubious secret-sauce binary kernel modules that are popular in embedded linux projects.

Quoted 3 lines

Thomas has also promised to come up with some userspace code that uses this interface to show how to use it, but seems to have forgotten. Consider this a reminder :)

That would be nice to see. I can't see how devices and drivers are registered from user space with what's here. I see iio_register_device() exported but no clue how userspace tells the kernel to claim a device or register an iio driver.

Quoted 2 lines · Greg KH

Quoted 1 line

A while ago, Thomas and I were sitting in the back of a conference

more generic than that.

"USD" is a bit too close to "USB" for a name. Any other ideas?

Quoted 9 lines

Quoted 4 lines

Thomas has also promised to come up with some userspace code that uses this interface to show how to use it, but seems to have forgotten. Consider this a reminder :)

That would be nice to see. I can't see how devices and drivers are registered from user space with what's here. I see iio_register_device() exported but no clue how userspace tells the kernel to claim a device or register an iio driver.

His posted example should show you how this all works together.

thanks,

greg k-h

Quoted 5 lines · Matt Porter

Well, if it's focused on industrial controls like it appears from the code here then the name is fine. If it's a starting point to become someting more generic then User Space Driver (USD) subsystem might be nice.

It's not focused on industrial controls at all, that just happens to be the name of it right now, as that is why it was written. It is much more generic than that.

"USD" is a bit too close to "USB" for a name. Any other ideas?

Quoted 11 lines · Greg KH

Quoted 5 lines · Matt Porter

Well, if it's focused on industrial controls like it appears from the code here then the name is fine. If it's a starting point to become someting more generic then User Space Driver (USD) subsystem might be nice.

It's not focused on industrial controls at all, that just happens to be the name of it right now, as that is why it was written. It is much more generic than that.

"USD" is a bit too close to "USB" for a name. Any other ideas?

DFU - Drivers for Userspace
DFU - Driver from Userspace
DFUF - Drivers for Userspace Framework DIFUS - Drive It From Userspace :-) FUD - Framework (for) Userspace Drivers ;-) GUDI - Generic Userspace Driver Interface UDF - Userspace Driver Framework :-) UDI - Userspace Driver Interface :-) UUDI - Unified Userspace Driver Interface UFD - Userspace Framework (for) Drivers

Bis denn

User mode drivers are NOT good. For them to work, too much stuff needs to be exposed. Then there is the problem of interrupts. You are not going to be able to make an interrupt 'thunker' like you could in the old DOS days. The kernel isn't going to call user-mode code because nobody is going to provide such a potentially destructive interface. If you think you can handle hardware interrupt requests with the poll() interface, you are going to be in a world of hurt for throughput.

Quoted 16 lines · Matt Porter

Quoted 14 lines · Greg KH

A while ago, Thomas and I were sitting in the back of a conference presentation where the presenter was trying to describe what they did in order to add the ability to write a userspace PCI driver. As was usual in a presentation like this, the presenter totaly ignored the real-world needs for such a framework, and only got it up and working on a single type of embedded system. But the charts and graphs were quite pretty :)

Thomas and I lamented that we were getting tired of seeing stuff like this, and it was about time that we added the proper code to the kernel to provide everything that would be needed in order to write PCI drivers in userspace in a sane manner. Really all that is needed is a way to handle the interrupt, everything else can already be done in userspace (look at X for an example of this...)

Quoted 2 lines

What about portable access to the PCI DMA API from userspace?

It currently does not provide this, but if you know how your card works, I'm pretty sure you can get this working without such an interface.

If you wish to add this functionality, to make it easier, that would be great.

I might be nice to have something like a copy-block where the application writes/reads data, while the device does DMA only from/to there. We would need an easy way to mmap some anonymous DMA-ready memory in user-space, and something to give the corresponding DMA addresses to the application.

Additionally, the current code might not be flexible enough regarding acknowledging of interrupts. It might be good to use the bit that PCI 2.2 defines in the config space to mask/unmask interrupt in a generic way. Something like : when an interrupt comes, the driver mask the interrupts using this bit, and then passes the event to user-space. The user-space interrupt handler acknowledges the interrupt with the device specific code, and then unmask with the PCI 2.2 bit.

Quoted 8 lines

Additionally, the current code might not be flexible enough regarding acknowledging of interrupts. It might be good to use the bit that PCI 2.2 defines in the config space to mask/unmask interrupt in a generic way. Something like : when an interrupt comes, the driver mask the interrupts using this bit, and then passes the event to user-space. The user-space interrupt handler acknowledges the interrupt with the device specific code, and then unmask with the PCI 2.2 bit.

You can do that today with this code. Remember, you have to have a tiny kernelspace portion of your driver to handle the interrupt. You can do whatever you want in that interrupt handler.

Quoted 9 lines · Greg KH

Quoted 8 lines

Additionally, the current code might not be flexible enough regarding acknowledging of interrupts. It might be good to use the bit that PCI 2.2 defines in the config space to mask/unmask interrupt in a generic way. Something like : when an interrupt comes, the driver mask the interrupts using this bit, and then passes the event to user-space. The user-space interrupt handler acknowledges the interrupt with the device specific code, and then unmask with the PCI 2.2 bit.

Quoted 4 lines

You can do that today with this code. Remember, you have to have a tiny kernelspace portion of your driver to handle the interrupt. You can do whatever you want in that interrupt handler.

The whole point of masking interrupt with this config-space bit is that we might not need any tiny kernelspace portion for our driver anymore. It won't work for devices that are not PCI 2.2 compliant. But, it might be good for the ones that are?

What about portable access to the PCI DMA API from userspace?

Quoted 2 lines · Matt Porter

What about portable access to the PCI DMA API from userspace?

We'll definitely need this for X11 anyways. Currently it is not possible to run the standard X server with a IOMMU that isolates the graphics card because it has no way to get at the GPU MMIO registers then.

My long-term plan was to integrate it in /sys/bus/pci mmaps (together with PAT etc.). When you mmap it there the kernel allocates a DMA mapping and then frees it on unmap.

A while ago, Thomas and I were sitting in the back of a conference presentation where the presenter was trying to describe what they did in order to add the ability to write a userspace PCI driver. As was usual in a presentation like this, the presenter totaly ignored the real-world needs for such a framework, and only got it up and working on a single type of embedded system. But the charts and graphs were quite pretty :)

Thomas and I lamented that we were getting tired of seeing stuff like this, and it was about time that we added the proper code to the kernel to provide everything that would be needed in order to write PCI drivers in userspace in a sane manner. Really all that is needed is a way to handle the interrupt, everything else can already be done in userspace (look at X for an example of this...)

Thomas mentioned that he had code to do all of this working in some customer sites already and that he would get it to me.

Fast forward to OLS of this year, and I bugged Thomas to send me the code. He did, and then I sat on it for a while longer...

So, here's the code. I think it does a bit too much all at once, but it is an example of how this can be done. This is working today in some industrial environments, successfully handling hardware controls of very large and scary machines. So it is possible to use this interface to successfully build your own laser wielding robot, all from userspace, allowing you to keep your special-secret-how-to-control-the-laser algorithm closed source if you so desire.

In looking at the proposed kevent interface, I think a few things in this proposed interface can be dropped in favor of using kevents instead, but I haven't looked at the latest version of that code to make sure of this.

And the name is a bit ackward, anyone have a better suggestion?

Thomas has also promised to come up with some userspace code that uses this interface to show how to use it, but seems to have forgotten. Consider this a reminder :)

Comments?

Being a bit excited and it is really interesting to have such a proposal, it would simplify the matters that held us up even more, probably. The name sounds fine though. All i was wondering whether there would be any high latencies for the same using in such a context. But since the transfers would occur in any way, even with a kernel mode driver, i think it should be pretty much fine.

Quoted 6 lines · Manu Abraham

Being a bit excited and it is really interesting to have such a proposal, it would simplify the matters that held us up even more, probably. The name sounds fine though. All i was wondering whether there would be any high latencies for the same using in such a context. But since the transfers would occur in any way, even with a kernel mode driver, i think it should be pretty much fine.

As mentioned, this framework is being used in industrial settings right now, where latencies are a huge issue. It works just fine, so I do not think there are any problems in this area.

Quoted 8 lines · Greg KH

Quoted 6 lines · Manu Abraham

Being a bit excited and it is really interesting to have such a proposal, it would simplify the matters that held us up even more, probably. The name sounds fine though. All i was wondering whether there would be any high latencies for the same using in such a context. But since the transfers would occur in any way, even with a kernel mode driver, i think it should be pretty much fine.

Quoted 3 lines

As mentioned, this framework is being used in industrial settings right now, where latencies are a huge issue. It works just fine, so I do not think there are any problems in this area.

Cool.

Is there some way we can avoid the poll ? It would be a real gain indeed, if a POLL can be avoided.

Quoted 11 lines · Manu Abraham

Quoted 10 lines · Greg KH

Quoted 6 lines · Manu Abraham

Being a bit excited and it is really interesting to have such a proposal, it would simplify the matters that held us up even more, probably. The name sounds fine though. All i was wondering whether there would be any high latencies for the same using in such a context. But since the transfers would occur in any way, even with a kernel mode driver, i think it should be pretty much fine.

As mentioned, this framework is being used in industrial settings right now, where latencies are a huge issue. It works just fine, so I do not think there are any problems in this area.

Quoted 4 lines

Cool.

Is there some way we can avoid the poll ? It would be a real gain indeed, if a POLL can be avoided.

Use the signal that will be sent to your userspace program when an interrupt happens.

If you can handle the small latency that causes it should be fine, but if you can't, then you should be using poll :)

It all depends on the hardware you are using, your processor, and what your tolerances are on your interrupt handling latency.

Usually in the typical application we have (where latency is an issue), most probably many of the people have a saturated PCI bus. In most cases, the IPTV guys have such a scenario. Say > 6 or 7 DVB adapters and the latency goes very high.

What i have seen is that when the bus gets saturated, the CPU usage shoots of rather abnormally.

When the latency goes higher, the resultant stream is useless and packets needs to be dropped, eventually that results in Transport Stream discontinuities.

Currently we already have a latency issue, based on the loud cries from some people.

drivers/iio/iio_dummy.c new file +20 -0

+static ssize_t store_sig_pid(struct device *dev, struct device_attribute *attr,
+			     const char *buf, size_t count)
+{
+	iio_dummy_signal.pid = simple_strtol(buf, NULL, 10);
+	if (iio_dummy_signal.pid == 0) {
+		if (iio_dummy_signal.it_process) {
+			put_task_struct(iio_dummy_signal.it_process);
+			iio_dummy_signal.it_process = NULL;
+		}
+
+		iio_dummy_signal.pid = 0;
+		return count;
+	}
+
+	if (iio_dummy_signal.pid == 1)
+		goto out;
+
+	iio_dummy_signal.it_process = find_task_by_pid(iio_dummy_signal.pid);
+	if (iio_dummy_signal.it_process) {
+		get_task_struct(iio_dummy_signal.it_process);

Quoted 20 lines · On Tue, 29 Aug 2006 23:23:38 -0700 Greg KH <[email protected]> wrote:

drivers/iio/iio_dummy.c new file +20 -0

+static ssize_t store_sig_pid(struct device *dev, struct device_attribute *attr,
+			     const char *buf, size_t count)
+{
+	iio_dummy_signal.pid = simple_strtol(buf, NULL, 10);
+	if (iio_dummy_signal.pid == 0) {
+		if (iio_dummy_signal.it_process) {
+			put_task_struct(iio_dummy_signal.it_process);
+			iio_dummy_signal.it_process = NULL;
+		}
+
+		iio_dummy_signal.pid = 0;
+		return count;
+	}
+
+	if (iio_dummy_signal.pid == 1)
+		goto out;
+
+	iio_dummy_signal.it_process = find_task_by_pid(iio_dummy_signal.pid);
+	if (iio_dummy_signal.it_process) {
+		get_task_struct(iio_dummy_signal.it_process);

drivers/iio/iio_dummy.c line 118

Quoted 10 lines

drivers/iio/iio_dummy.c new file +10 -0

+		iio_dummy_signal.it_sigev_notify = SIGEV_SIGNAL;
+		iio_dummy_signal.it_sigev_signo = SIGALRM;
+		iio_dummy_signal.it_sigev_value.sival_int = 0;
+
+		return count;
+	}
+out:
+	iio_dummy_signal.pid = 0;
+	return -EINVAL;
+}

This is racy: find_task_by_pid() needs tasklist_lock or rcu_read_lock().

It doesn't work as a module due to missing __put_task_struct.

It is also rather nasty. Why go shoving some random pid into a sysfs file, then hang onto a ref on a task_struct for some process which exitted last week? It would be cleaner and more idiomatic to require that the controlling process hold an fd open against the instance so that resources can be managed correctly. Maybe use SIGIO too, so the driver doesn't need to know about pids and task_structs and things. Which all maps better onto ioctls than sysfs (ties self to stake)

<looks>

iio_dev.c seems to already be doing this. Why does iio_dummy.c exist?

Thomas and I lamented that we were getting tired of seeing stuff like this, and it was about time that we added the proper code to the kernel to provide everything that would be needed in order to write PCI drivers in userspace in a sane manner. Really all that is needed is a way to handle the interrupt, everything else can already be done in userspace (look at X for an example of this...)

So, here's the code.

Hi Greg,

On Wed, 2006-08-30 at 08:23, Greg KH wrote:
[...]

Quoted 6 lines

Thomas and I lamented that we were getting tired of seeing stuff like this, and it was about time that we added the proper code to the kernel to provide everything that would be needed in order to write PCI drivers in userspace in a sane manner. Really all that is needed is a way to handle the interrupt, everything else can already be done in userspace (look at X for an example of this...)

[...]

Quoted 1 line

So, here's the code.

I know I look like I'm trolling here, but as this is the perfect stable-binary-driver-ABI some people have been looking for, it would be wise to make it taint the kernel with its own flag.

Why? There's no technical or legal requirement for userspace drivers to be GPLed.

Quoted 2 lines · Lee Revell

Why? There's no technical or legal requirement for userspace drivers to be GPLed.

I could see a benefit to tainting the kernel once userspace starts poking at the hardware directly. That way at least we'd know that a crash might be due to userspace doing bad things.

For instance, consider the case where userspace misprograms a DMA engine, which starts overwriting random kernel memory.

Quoted 11 lines · Xavier Bestel

Hi Greg,

On Wed, 2006-08-30 at 08:23, Greg KH wrote:
[...]

Quoted 6 lines

Thomas and I lamented that we were getting tired of seeing stuff like this, and it was about time that we added the proper code to the kernel to provide everything that would be needed in order to write PCI drivers in userspace in a sane manner. Really all that is needed is a way to handle the interrupt, everything else can already be done in userspace (look at X for an example of this...)

[...]

Quoted 5 lines

Quoted 1 line

So, here's the code.

I know I look like I'm trolling here, but as this is the perfect stable-binary-driver-ABI some people have been looking for, it would be wise to make it taint the kernel with its own flag.

Why? There's no technical or legal requirement for userspace drivers to be GPLed.

Precisely. How do you know the bugreport you received isn't caused by some weird binary userspace driver hosing the PCI bus ?

Quoted 2 lines · Xavier Bestel

Precisely. How do you know the bugreport you received isn't caused by some weird binary userspace driver hosing the PCI bus ?

Can't X, or any application that access hardware directly by mmaping /dev/mem, do this now?

Quoted 5 lines · Lee Revell

Quoted 2 lines · Xavier Bestel

Precisely. How do you know the bugreport you received isn't caused by some weird binary userspace driver hosing the PCI bus ?

Can't X, or any application that access hardware directly by mmaping /dev/mem, do this now?

Yes they can, but X's behavior is pretty well known by now :) and it's open source.

Le jeudi 31 ao?t 2006 ? 16:39 -0400, Lee Revell a ?crit :

Quoted 18 lines · Xavier Bestel

Quoted 16 lines

Hi Greg,

On Wed, 2006-08-30 at 08:23, Greg KH wrote:
[...]

Quoted 6 lines

Thomas and I lamented that we were getting tired of seeing stuff like this, and it was about time that we added the proper code to the kernel to provide everything that would be needed in order to write PCI drivers in userspace in a sane manner. Really all that is needed is a way to handle the interrupt, everything else can already be done in userspace (look at X for an example of this...)

[...]

Quoted 1 line

So, here's the code.

I know I look like I'm trolling here, but as this is the perfect stable-binary-driver-ABI some people have been looking for, it would be wise to make it taint the kernel with its own flag.

Quoted 2 lines

Why? There's no technical or legal requirement for userspace drivers to be GPLed.

Precisely. How do you know the bugreport you received isn't caused by some weird binary userspace driver hosing the PCI bus ?

And the name is a bit ackward, anyone have a better suggestion?

Hi!

Quoted 1 line

And the name is a bit ackward, anyone have a better suggestion?

drivers/uio (for userspace io driver)?

And yes, I think this one _should_ taint the kernel. When userspace starts playing with interrupts, chances of kernel crash are high.

Does it work properly with pci shared interrupts?

So, here's the code. I think it does a bit too much all at once, but it is an example of how this can be done. This is working today in some industrial environments, successfully handling hardware controls of very large and scary machines.

Quoted 5 lines · Greg KH

So, here's the code. I think it does a bit too much all at once, but it is an example of how this can be done. This is working today in some industrial environments, successfully handling hardware controls of very large and scary machines.

At present it is named uio in -mm. But that seems to conflict with a few cases (uio.h + Solaris).

How about:
Universal User IO => uuio

A quick google did not turn up anything conflicting.