Skip to contentSkip to conversation

considering kevent - the kernel development process

DISCUSSION Johann Borck 4 participants 4 messages Mar 13, 2007 – Mar 14, 2007

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

if the next section seems obvious, please skip it
-------------------------------------------------------------------------
There is a need for a generic event handling mechanism in Linux - maybe it is enough to cite the current CGL Performance Requirements Definition Version 4.0 at
http://developer.osdl.org/dev/cgl/cgl40/cgl40-performance.pdf , page 13:

PRF.5.0 Efficient Low-Level Asynchronous Events Priority P1 Description: OSDL CGL specifies that carrier grade Linux shall provide an efficient capability for handling a large number of essentially simultaneous asynchronous events arriving on multiple channels, such as
multiple sockets or other similar paths. This mechanism is needed to enforce system scalability and soft real-time responsiveness by reducing contentions appearing at the kernel level, especially under high load.

Ulrich Drepper has written a paper The Need for Aynchronous, Zero­Copy Network I/O,
http://people.redhat.com/drepper/newni-slides.pdf

A bit older but nice wrt performance-issues is APPLYING THE PROACTOR PATTERN TO HIGH-PERFORMANCE WEB SERVERS at http://www.cs.wustl.edu/~schmidt/PDF/proactor-jaws.pdf along with many other research papers on that site.

---------------------------------------------------------------------------

So the question is not to be or not to be (at least not for an async event mechanism) - Linux needs it. There have been approaches to this, the both I'm aware of are:
kevent by Evgeniy Polyakov
AEM by Frederic Rossi at http://aem.sourceforge.net/ , several years ago.

The discussion about kevent has stalled since Ulrich Drepper left the thread in November without commenting on many changes Evgeniy did to the interface and implementation in response to requirements formulated by Drepper. The only further statement was this:

Quoted 7 lines
Quoted 3 lines
Quoted 2 lines · Evgeniy Polyakov

I think that mean that everybody is happy with APi, design and set of features

Quoted 2 lines · Ulrich Drepper

No comment means that I still have not been able to test anything since regardless of what version I tried, it failed to build.

Quoted 5 lines · Evgeniy Polyakov

I think if you would provide error message you saw, I fixed that in a couple of moments, doesn't it?

I will send new patchset next week when rc1 will be out and your feedback is greatly appreciated.

This is hard to believe - the person that the whole process was blocking on (in a sane linux-world, hopefully for competence-reasons) wasn't able to build a kernel with that patch? And I was? With almost any version, without problems? very hard to believe.
Then a thread about threadlets/syslets and aio came up - the kernel developers discussed some kevent-related stuff, Evgeniy brought up his project again, and after some discussion about benchmarks and behavior, it was gone again.
Somehow Evgeniy backed up then - and wrote 'eventfs' to make signal notifications epollable mentioning that he'd add Posix Timer support if the concept will be accepted - which wasn't commented at all, but instead Davide Libenzi came up with two patches for exactly the same purpose, which raised a lot of feedback.

It can't be more obvious.. Is the consideration of patches (in case of kevent
major contributions) completely independent of content and intent?
Although Evgeniy showed remarkable productivity and dedication to his project, included support for stuff that was requested even though he had objections(posix timers), redesigned parts on request (mainly asked for by U. Drepper), implemented example programs, implemented many different event-types and drafted network-aio support, wrote patches for lighttpd and libevent to show usability in real-life apps, which he was asked to do (because people would then be able to make up their mind), nothing happened.

After the implementation of most of Dreppers requests for change the thread remained almost silent - at a stage where one would think that the work on details could start.

Quoted 9 lines · Andrew Morton
Quoted 8 lines
Quoted 2 lines
Quoted 2 lines · On Mon, 12 Feb 2007 13:35:10 +0300 Evgeniy Polyakov wrote:

Andrew, do you consider kevent for inclusion or declining?

I haven't had time to think about it in the past month or two, sorry.

However we might as well get it back in there for review-and-test - please send a new patchset once 2.6.21-rc1 is released and copy me on it?

I don't want to attack anyone here, but there is a problem in the way (at least) kevent is considered. As stated an asynchronous event mechanism is a crucial thing, and would be a major contribution to the kernel. It is easy to come to the conclusion that the main kernel-developers just don't want to accept such a contribution by someone not already inside that hall of fame. Otherwise, there would be arguments, discussion, there would be reasons. The attempts to explain it with lack of time, lack of ability to test or whatever are weak. They suffer from the evidence, that related stuff is discussed all over - compare eventfs vs. signalfd/timerfd threads and look at syslets/threadlets thread. Of course one could argue that people with less reputation are treated differently, because one estimates the relevance of their contributions lower. But the relevance of the gap that kevent might fill, and therefore kevent itself, is tremendous. It is priority 1 as stated in the CGL Performance Requirements Definition - but it is not discussed.

So my questions are:

What in general is wrong with kevent?

What, in detail is wrong with kevent / its proposed interface?

What features are missing?

What are the (real) reasons for the way it is discussed?

One argument that can be read often in this context is that no one would use it. That's nonsense, or why is it P1 as stated in the CGL Performance Requirements Definition? It would be definitely used, yes it would change the way people write and think about writing software for Linux.

To me this looks quite scary - though the need is there, and the needed functionality is simulated by many frameworks (ACE/Proactor/TProactor, Twisted, Boost::asio, libevent etc.) the discussion stops - as if it wouldn't matter. But it does, and so there are things wrong with the development process, as wrong as contributions like kevent are important for Linux. All big organizations adhere to institutionalized procedures and become dull wrt change. Their departments tend to preserve their influence. Is the institution that is in charge of the final decisions here (the core kernel developers) open enough to deal with major contributions from "outside", to discuss their *value* based on need, relevance, maturity? Or has the control over the process and isolation from mere mortals become more important than the functionality of the kernel itself? Don't get me wrong, this is *not* about inclusion, this is about *discussion*!

Johann Borck

#1 lore raw

please cut your cc lines next time, thanks.

Quoted 7 lines · Johann Borck

if the next section seems obvious, please skip it
-------------------------------------------------------------------------
There is a need for a generic event handling mechanism in Linux - maybe it is enough to cite the current CGL Performance Requirements Definition Version 4.0 at
http://developer.osdl.org/dev/cgl/cgl40/cgl40-performance.pdf , page 13:

Quoted 8 lines

PRF.5.0 Efficient Low-Level Asynchronous Events Priority P1
Description: OSDL CGL specifies that carrier grade Linux shall provide an efficient capability for handling a large number of essentially simultaneous asynchronous events arriving on multiple channels, such as multiple sockets or other similar paths. This mechanism is needed to enforce system scalability and soft real-time responsiveness by reducing contentions appearing at the kernel level, especially under high load.

It's not obvious, but utter bullshit. Please go away if you think starting post with quotes from CGL specs is even remotely close to a good idea.

#2 lore raw
Quoted 4 lines · Christoph Hellwig

It's not obvious, but utter bullshit. Please go away if you think starting post with quotes from CGL specs is even remotely close to a good idea.

I don't know about CGL, but a unified async notification mechanism _is_ needed. In ~2002, I needed one, and what I ended up with was a Red Hat 2.4 kernel with IO_CMD_POLL hacked in. Five years later, mainline still can't do that.

Whether it's syslets, threadlets, io_getevents with networking support, kevents, epoll with kaio support, queued signals, or something else, users don't care, but they need _something_. Please don't ignore the users just because they quote the wrong text. The need is very real.

Signature · 1 line
Do not meddle in the internals of kernels, for they are subtle and quick to panic.
#3 lore raw
In reply to Johann Borck
Quoted 9 lines · Johann Borck

So my questions are:

What in general is wrong with kevent?

What, in detail is wrong with kevent / its proposed interface?

What features are missing?

What are the (real) reasons for the way it is discussed?

Fsck it, Johann :)

It is a trend (a politics) to hide head into the sand and feel like nothing happend - open source does not have mechanisms to force people to do something (like comment on foreign ideas/implementations), so people make a look like problem does not exist and solve it using own way.

There will be tons of other interesting stuff, stay tuned :)

Signature · 1 line
	Evgeniy Polyakov