userspace infrastructure services
I expected someone else to bring this topic up...
One thing that is happening is that there is lots of activity in moving core features of the kernel into userspace (networking, storage, security). I don't want to get into an argument over whether that is good or bad; but the problem from the kernel point of view is that the existing API's are getting stretched to the boundary.
The area I am most familiar with is the DPDK which has to have its own UIO drivers to work in all environments (get device into userspace). And then has to have its own drivers to simulate network device (put device back into kernel). This leads to unmanageable ABI and development technical debt. I have heard that same thing is planned for storage as well.
The question is should the kernel accommodate this by absorbing/integrating/fixing this; or just let them suffer?
Quoted 5 lines · Stephen Hemminger
I expected someone else to bring this topic up...
One thing that is happening is that there is lots of activity in moving core features of the kernel into userspace (networking, storage, security). I don't want to get into an argument over whether that is good or bad;
Which I think is the most important question of them all. From all the pain caused and the little gains I'm a believer it's mostly bad, especially for cases like:
Quoted 4 lines
The area I am most familiar with is the DPDK which has to have its own UIO drivers to work in all environments (get device into userspace). And then has to have its own drivers to simulate network device (put device back into kernel). This leads to unmanageable ABI and development technical debt.
Which interacts on both sides. In that case "let them suffer for their stupidity" is the only valid answer.
For a case where it's more reasonable there might be a different answer.
Quoted 16 lines · Christoph Hellwig
Quoted 5 lines · Stephen Hemminger
I expected someone else to bring this topic up...
One thing that is happening is that there is lots of activity in moving core features of the kernel into userspace (networking, storage, security). I don't want to get into an argument over whether that is good or bad;
Which I think is the most important question of them all. From all the pain caused and the little gains I'm a believer it's mostly bad, especially for cases like:
Quoted 4 lines
The area I am most familiar with is the DPDK which has to have its own UIO drivers to work in all environments (get device into userspace). And then has to have its own drivers to simulate network device (put device back into kernel). This leads to unmanageable ABI and development technical debt.
Quoted 4 lines
Which interacts on both sides. In that case "let them suffer for their stupidity" is the only valid answer.
For a case where it's more reasonable there might be a different answer.
What about something like CRIU:
?
in-kernel checkpoint/restore had been proposed many times before and always rejected as being hugely complex (most publicly at the Kernel Summit in Cambridge, where Oren Laadan proposed it). CRIU took a completely different approach where we broke the problem down into sections and extended kernel interfaces to export state where it wasn't available. The net result is that we now have checkpoint/restore (and thus process migration) for Linux at a significant reduction in the complexity over the original code and realistically not a huge technical debt.
What I like to think CRIU shows us is that if we look hard enough, there is a way to separate the problem up so that a userspace becomes workable without a massive increase in the kernel ABI surface.
The big problem is finding the way to do the split. You can tell if you get it wrong because the userspace component has a huge amount of state shared with the kernel and the ABI balloons to cope with this (cough, DPDK, cough)... however, what's not clear is how to get it right in the first place.
Perhaps we should look first at how CRIU got it right and then discuss how we might use those same principles to fix other projects, like DPDK.
James
Quoted 2 lines · James Bottomley
Perhaps we should look first at how CRIU got it right and then discuss how we might use those same principles to fix other projects, like DPDK.
Well one thing is to look at the existing implementation of the APIs that are already available in the IBverbs/RMDA and cover both networking as well as storage. They already separate the control and the data transfer areas. Kernel provides control but gets out of the way of data transfers. Arguably this could be cleaner but here is an existing way of doing this that we already support and that has a decade long history of use in the industry.
This is not about minimal gains but being able to get more than just a fraction of the possible performance out of high end devices.
Quoted 16 lines · On Wed, 12 Aug 2015 11:21:03 -0700 Christoph Hellwig <[email protected]> wrote:
Quoted 5 lines · Stephen Hemminger
I expected someone else to bring this topic up...
One thing that is happening is that there is lots of activity in moving core features of the kernel into userspace (networking, storage, security). I don't want to get into an argument over whether that is good or bad;
Which I think is the most important question of them all. From all the pain caused and the little gains I'm a believer it's mostly bad, especially for cases like:
Quoted 4 lines
The area I am most familiar with is the DPDK which has to have its own UIO drivers to work in all environments (get device into userspace). And then has to have its own drivers to simulate network device (put device back into kernel). This leads to unmanageable ABI and development technical debt.
Quoted 4 lines
Which interacts on both sides. In that case "let them suffer for their stupidity" is the only valid answer.
For a case where it's more reasonable there might be a different answer.
I think this is like earlier work with embedded. If the community does not engage, then the pain only increases on both sides over time. Overall, the only long term solution is something like:
"Yes, I see what you want, but how you are doing is stupid.
Let's work together and see if there is a better way to do this"
In most cases, the kernel has already exposed a bunch of API's that make the whole effort possible (uio, tun/tap, ...); either we say "ignore those API's, they are toys you are not supposed to actually use for real products"
or be professional and make them into first class citizens.