Skip to contentSkip to conversation

Scheduler ( was: Just a second ) ...

DISCUSSION Linus Torvalds 27 participants 91 messages Dec 18, 2001 – Dec 27, 2001 1 message not in the archive

lore
Share & tools
lore
#1 lore raw
In reply to a message not in the archive
Quoted 6 lines · Benjamin LaHaise
Quoted 3 lines · Linus Torvalds
Quoted 1 line

Well, we've got serious chicken and egg problems then.

Why?

The code can't go into glibc without syscall numbers being reserved.

It sure as hell can.

And I'll bet $5 USD that glibc wouldn't take the patches anyway before the kernel interfaces are _tested_.

Quoted 2 lines

I've posted the code, there are people playing with it. I can't make them comment.

Well, if people aren't interested, then it doesn't _ever_ go in.

Remember: we do not add features just because we can.

Quite frankly, I don't think you've told that many people. I haven't seen any discussion about the aio stuff on linux-kernel, which may be because you posted several announcements and nobody cared, or it may be that you've only mentioned it fleetingly and people didn't notice.

Take a look at how long it took for ext3 to be "standard" - I put them in my tree when I started getting real feedback that it was used and people liked using it. I simply do not like applying patches "just to get users". Not even reservations - because I reserve the right to _never_ apply something if critical review ends up saying that "that doesn't make sense".

Quite frankly, the fact that it is being tested out at places like Oracle etc is secondary - those people will use anything. That's proven by history. That doesn't mean that _I_ accept anything.

Now, the fact that I like the interfaces is actually secondary - it does make me much more likely to include it even in a half-baked thing, but it does NOT mean that I trust my own taste so much that I'd do it "under the covers" with little open discussion, use and modification.

Where _is_ the discussion on linux-kernel?

Where are the negative comments from Al? (Al _always_ has negative comments and suggestions for improvements, don't try to say that he also liked it unconditionally ;)

Linus

#2 lore raw
Quoted 1 line · Linus Torvalds

Where _is_ the discussion on linux-kernel?

Which mailing lists do you want to be subscribed to ? ;)

Rik

Signature · 3 lines
DMCA, SSSCA, W3C?  Who cares?  http://thefreeworld.net/

http://www.surriel.com/		http://distro.conectiva.com/
#3 lore raw
Quoted 5 lines · Rik van Riel
Quoted 1 line · Linus Torvalds

Where _is_ the discussion on linux-kernel?

Which mailing lists do you want to be subscribed to ? ;)

I'm not subscribed to any, thank you very much. I read them through a news gateway, which gives me access to the common ones.

And if the discussion wasn't on the common ones, then it wasn't an open discussion.

And no, I don't think IRC counts either, sorry.

Linus

#90 lore raw
In reply to Linus Torvalds
Quoted 4 lines · Linus Torvalds

Quite frankly, I don't think you've told that many people. I haven't seen any discussion about the aio stuff on linux-kernel, which may be because you posted several announcements and nobody cared, or it may be that you've only mentioned it fleetingly and people didn't notice.

This is not to ask the inclusion of /dev/epoll inside the kernel ( it can be easily merged by users that want to use it ) but i've had its users to prefer talking about that out of the mailing list. Maybe because they're scared to be eaten by some gurus when asking easy questions :)

- Davide

#4 lore raw
In reply to Linus Torvalds
Quoted 5 lines

I'm not subscribed to any, thank you very much. I read them through a news gateway, which gives me access to the common ones.

And if the discussion wasn't on the common ones, then it wasn't an open discussion.

If the discussion was on the l/k list then most kernel developers arent going to read it because tey dont have time to wade through all the crap that doesnt matter to them.

Quoted 1 line

And no, I don't think IRC counts either, sorry.

IRC is where most stuff, especially cross vendor stuff is initially discussed nowdays, along with kernelnewbies where most of the intro stuff is - but thats disussed rather than formally proposed and studied

#75 lore raw
In reply to Linus Torvalds
Quoted 1 line · Linus Torvalds

And no, I don't think IRC counts either, sorry.

Whether you think it counts or not, IRC is where
most stuff is happening nowadays.

cheers,

Rik

Signature · 3 lines
DMCA, SSSCA, W3C?  Who cares?  http://thefreeworld.net/

http://www.surriel.com/		http://distro.conectiva.com/
#91 lore raw
In reply to Linus Torvalds
Quoted 3 lines · Linus Torvalds

Where are the negative comments from Al? (Al _always_ has negative comments and suggestions for improvements, don't try to say that he also liked it unconditionally ;)

Heh.

Aside of a _big_ problem with exposing async API to userland (for a lot of reasons, including usual quality of async code in general and event-drivel one in particular) there is more specific one - Ben's long-promised full-async writepage() and friends. I'll believe it when I see it and so far it didn't appear.

So for the time being I'm staying the fsck out of that - I don't like it, but I'm sick and tired of this sort of religious wars.

#76 lore raw

Re: in defense of the linux-kernel mailing list

In reply to Rik van Riel
Quoted 4 lines · Rik van Riel
Quoted 1 line

And no, I don't think IRC counts either, sorry.

Whether you think it counts or not, IRC is where most stuff is happening nowadays.

most of the useful traffic on lkml cannot be expressed well on IRC. While IRC might be useful as an additional form of communication channel, email lists IMO should still be the main driving force of Linux kernel development, else we'll only concentrate on those minute ideas that can be expressed in 1-2 lines on irc and which are simple enough to be understood until the next message comes. Also, the lack of reliable archiving of IRC traffic prevents newcomers of reproducing the thought process afterwards. While IRC might result in the seasoned kernel developer doing the next super-patch quickly, it will in the end effect only isolate and alienate newcomers and will only result in an aging, personality-driven elitist old-boys network and a dying OS.

Regarding the use of IRC as the main development medium for the Linux kernel - the fast pace of IRC often prevents deeper thoughts - while this is definitely the point for many people who use IRC, it cannot result in a much better kernel. [that having said, i'm using irc on a daily basis as well so this is not irc-bashing, but i rarely use it for development purposes.]

It's true that reading off-topic emails on lkml isnt a wise use of developer powers either, but this has to be taken into account just like spam - it's the price of having an open forum.

and honestly, much of the complaints about lkml's quality are exagerated. What you dont take into account is the fact that while 3 or 5 years ago you found perhaps every email on lkml exciting and challenging, today you are an experienced kernel hacker and find perhaps 90% of the traffic 'boring'. I've just done a test - and perhaps i picked the wrong set of emails - but the majority of lkml traffic is pretty legitimate, and i would have found most of them 'interesting and exciting' just 5 years ago. Today i know what they mean and might find them less challenging to understand - but that is one of the bad side-effects of experience. Today there are more people on lkml, more bugs get reported, and more patches are discussed - so keeping up with lkml traffic is harder. Perhaps it might make sense to separate linux-kernel into two lists:
linux-kernel-bugs and linux-kernel-devel (without moderation), but otherwise the current form and quality of discussions (knock on wood) is pretty OK i think.

also, more formal emails match actual source code format better than the informal IRC traffic. So by being kindof forced to structure information into a larger set of ASCII text, it will also be the first step towards good kernel code.

(on IRC one might be the super-hacker with a well-known nick, entering and exiting channels, being talked to by newbies. It might boost one's ego. But it should not cloud one's judgement.)

Ingo

#78 lore raw
In reply to Linus Torvalds

Maybe I'm an old stick in the mud, but IRC seems like a big waste of time to me. It's perfect for off the cuff answers and fairly useless for thoughtful answers. We used to write well thought out papers and specifications for OS work. These days if you can't do it in a paragraph on IRC it must not be worth doing, eh?

Quoted 20 lines · Alan Cox
Quoted 5 lines

I'm not subscribed to any, thank you very much. I read them through a news gateway, which gives me access to the common ones.

And if the discussion wasn't on the common ones, then it wasn't an open discussion.

If the discussion was on the l/k list then most kernel developers arent going to read it because tey dont have time to wade through all the crap that doesnt matter to them.

Quoted 1 line

And no, I don't think IRC counts either, sorry.

IRC is where most stuff, especially cross vendor stuff is initially discussed nowdays, along with kernelnewbies where most of the intro stuff is - but thats disussed rather than formally proposed and studied -
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Signature · 2 lines
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 
#79 lore raw

From: Larry McVoy <[email protected]>
Date: Tue, 18 Dec 2001 13:02:28 -0800

Maybe I'm an old stick in the mud, but IRC seems like a big waste of time to me.

It's like being at a Linux conference all the time. :-)

It does kind of make sense given that people are so scattered across the planet. Sometimes I want to just grill someone on something, and email would be too much back and forth, IRC is one way to accomplish that.

#81 lore raw
In reply to Linus Torvalds
Quoted 6 lines · David S. Miller

From: Larry McVoy <[email protected]>
Date: Tue, 18 Dec 2001 13:02:28 -0800

Maybe I'm an old stick in the mud, but IRC seems like a big waste of time to me.

Quoted 6 lines

It's like being at a Linux conference all the time. :-)

It does kind of make sense given that people are so scattered across the planet. Sometimes I want to just grill someone on something, and email would be too much back and forth, IRC is one way to accomplish that.

Let me introduce you to this neat invention called a telephone. It's the black thing next to your desk, it rings, has buttons. If you push the right buttons, well, it's magic...

:-)

Signature · 2 lines
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 
#80 lore raw
In reply to Larry McVoy
Quoted 5 lines · Larry McVoy

Maybe I'm an old stick in the mud, but IRC seems like a big waste of time to me. It's perfect for off the cuff answers and fairly useless for thoughtful answers. We used to write well thought out papers and specifications for OS work. These days if you can't do it in a paragraph on IRC it must not be worth doing, eh?

Actually, we tend to use multiple media at the same time.

It happens very often that because of some discussion on IRC we end up writing up a few paragraphs and sending it to people by email.

For other things, email is clearly too slow, so stuff is done on IRC (eg. walking somebody through a piece of code to identify and agree on a bug).

cheers,

Rik
--
DMCA, SSSCA, W3C? Who cares? http://thefreeworld.net/

http://www.surriel.com/ http://distro.conectiva.com/

#82 lore raw
In reply to Larry McVoy
Quoted 4 lines · Larry McVoy
Quoted 3 lines · David S. Miller

From: Larry McVoy <[email protected]>
Date: Tue, 18 Dec 2001 13:02:28 -0800

Quoted 8 lines
Quoted 4 lines

Maybe I'm an old stick in the mud, but IRC seems like a big waste of time to me.

It's like being at a Linux conference all the time. :-)

Let me introduce you to this neat invention called a telephone. It's the black thing next to your desk, it rings, has buttons. If you push the right buttons, well, it's magic...

Yeah, but you can't scroll up a page on the phone...

(also, talking with multiple people at the same time is kind of annoying in audio, while it's ok on irc)

Rik
--
DMCA, SSSCA, W3C? Who cares? http://thefreeworld.net/

http://www.surriel.com/ http://distro.conectiva.com/

#83 lore raw
In reply to Larry McVoy

From: Larry McVoy <[email protected]>
Date: Tue, 18 Dec 2001 13:17:13 -0800

Let me introduce you to this neat invention called a telephone. It's the black thing next to your desk, it rings, has buttons. If you push the right buttons, well, it's magic...

I'm not calling Holland every time I want to poke Jens about something in a patch we're working on :-)

I hate telephones for technical stuff, because people can call the fucking thing when I am not behind my computer or even worse when I AM behind my computer and I want to concentrate on the code on my screen without being disturbed. With IRC it is MY CHOICE to get involved in the discussion, I can choose to respond or not respond to someone, I can choose to be available or not available at any given time. It's just a real-time version of email. And the "passive, I can ignore you" part is what I like about it.

Telephones frankly suck for discussing technical topics. I can't cut and paste pieces of code from my other editor buffer to show you over the phone, as another example as to why.

A lot of people like to use telephones specifically because it does not give the other party the option of ignoring you once they pick up the phone. I value the ability to make the choice to ignore people because a lot of ideas I don't give a crap about come under my nose.

In fact that may be one of the best parts about Linux development compared to doing stuff at a company, one isn't required to listen to someone's idea or to even read it. If today I don't give a crap about Joe's filesystem idea, hey guess what I'm not going to read any of his emails about the thing.

#5 lore raw
In reply to Alan Cox
Quoted 9 lines · Alan Cox
Quoted 5 lines

I'm not subscribed to any, thank you very much. I read them through a news gateway, which gives me access to the common ones.

And if the discussion wasn't on the common ones, then it wasn't an open discussion.

If the discussion was on the l/k list then most kernel developers arent going to read it because tey dont have time to wade through all the crap that doesnt matter to them.

Hi Alan,

It's AIO we're talking about, right? AIO is interesting to quite a few people. I'd read the thread. I'd also read any background material that Ben would be so kind as to supply.

--
Daniel

#84 lore raw

IRC (was: Scheduler)

In reply to Linus Torvalds
Quoted 5 lines · Larry McVoy

Maybe I'm an old stick in the mud, but IRC seems like a big waste of time to me. It's perfect for off the cuff answers and fairly useless for thoughtful answers. We used to write well thought out papers and specifications for OS work. These days if you can't do it in a paragraph on IRC it must not be worth doing, eh?

Hi Larry,

It's a question of using the right tool for the job. As you know, email is no substitute for a traditional everybody-in-one-room design meeting. These days, with development distributed all over the world it's just not practical for everyone to physically get together more than a few times a year, so what can we do? Right, hang on IRC.

In some ways IRC is more efficient than a face-to-face meeting:

- You can do other things at the same time without offending anyone (usually)

- Everything is logged for reference

- You can copy code examples and URLs into the channel

- It's normal to send/forward emails, perhaps with traditional papers attached, patches, whatever, while talking on the channel, or as a result of talking on the channel

- It's there 24 hours a day

- You can leave the meeting and do work any time you want to, as opposed to keeping some portion of a group of highly paid engineers bored and idle for hours at at time.

IRC also solves a big problem for distributed companies: how can you be sure that your people are actually on the job? (You ping them on IRC and they respond.)

While there's no doubt about IRC's value, there's also a danger: IRC is addictive. You can easily end up spending all your time there, and doing very little design/coding as a result. That's a matter of self-discipline.

To put this into a more immediate perspective for you, suppose you wanted to get some traction under your SMP Clusters proposal? I'd suggest it's already been kicked around as much as it's going to be on lkml, and you already wrote your paper, so the next step would be to get together face-to-face with some folks who have a clue. Well, unless you're willing to wait months for the right people to show up in the Bay Area, IRC is the way to go.

Come on in, the water's fine ;-)

--
Daniel

#88 lore raw

Re: IRC (was: Scheduler)

In reply to Linus Torvalds
Quoted 7 lines · Daniel Phillips
Quoted 5 lines · Larry McVoy

Maybe I'm an old stick in the mud, but IRC seems like a big waste of time to me. It's perfect for off the cuff answers and fairly useless for thoughtful answers. We used to write well thought out papers and specifications for OS work. These days if you can't do it in a paragraph on IRC it must not be worth doing, eh?

Quoted 6 lines

To put this into a more immediate perspective for you, suppose you wanted to get some traction under your SMP Clusters proposal? I'd suggest it's already been kicked around as much as it's going to be on lkml, and you already wrote your paper, so the next step would be to get together face-to-face with some folks who have a clue. Well, unless you're willing to wait months for the right people to show up in the Bay Area, IRC is the way to go.

Actually, I haven't written a paper. A paper is something which lays out

goals
architecture
milestones
design details

and should be sufficient to make the project happen should I be hit by a bus. That's my main complaint with IRC, it requires me to keep coming back and explaining the same thing over and over again.

Here's an idea: you go try and get some traction on the OS cluster idea. I'll give you 6 months and we'll see what happens. If nothing has happened, I'll produce a decent paper describing it and then we wait another 6 months to see what happens. I'll bet you 10:1 odds I get a lot more action from a lot more people than you do. Nope, wait, make that 100:1 odds.

I've seen how little I manage to get done by talking. Talk is cheap. I've also seen how much I get done when I write a paper which other people can pass around, think about, discuss, and implement. A senior guy at Morgan Stanley (hi marc) once told me "if you want to get things done, write them down". And in my case, since people tend to like to argue with me rather than listen to me (yup, it's my fault, my "style" leaves "room for improvement" translation: sucks rocks), a paper is far more effective. My style is pretty much removed from the equation.

I can just see me on IRC, all I'd be getting is style complaints while people successfully avoid the real points. Look at the last 8 years of LKML. I'd say most of the effect was from the LMbench paper and maybe a few threads on performance which would have been more effective if I'd written a detailed paper explaining my point of view.

Signature · 2 lines
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 
#85 lore raw

Re: IRC (was: Scheduler)

In reply to Daniel Phillips
Quoted 8 lines · Daniel Phillips

To put this into a more immediate perspective for you, suppose you wanted to get some traction under your SMP Clusters proposal? I'd suggest it's already been kicked around as much as it's going to be on lkml, and you already wrote your paper, so the next step would be to get together face-to-face with some folks who have a clue. Well, unless you're willing to wait months for the right people to show up in the Bay Area, IRC is the way to go.

Come on in, the water's fine ;-)

I've watched with great interest the discussion of IRC for Linux folk and have yet to see anyone mention server/network names and channel names. I've been on IRC for 2.5 years -- I tracked the Y2K transition on IRC despite all the dire warnings that evil impulses were going to shoot down the wire and fry the LCD screen on my laptop. So -- just where exactly *is* this water that is so fine? mIRC 5.91 and I await with bated breath. (Yes, I do use a Windows IRC client -- wanna make something of it? :-)
--
Ed Borasky [email protected] http://www.borasky-research.net
(sometimes known as znmeb on IRC :-)

How to Stop A Folksinger Cold # 4
"Tie me kangaroo down, sport..."
Tie your own kangaroo down -- and stop calling me "sport"!

#89 lore raw

Re: IRC (was: Scheduler)

In reply to Linus Torvalds
Quoted 26 lines · Larry McVoy
Quoted 7 lines · Daniel Phillips
Quoted 5 lines · Larry McVoy

Maybe I'm an old stick in the mud, but IRC seems like a big waste of time to me. It's perfect for off the cuff answers and fairly useless for thoughtful answers. We used to write well thought out papers and specifications for OS work. These days if you can't do it in a paragraph on IRC it must not be worth doing, eh?

Quoted 6 lines

To put this into a more immediate perspective for you, suppose you wanted to get some traction under your SMP Clusters proposal? I'd suggest it's already been kicked around as much as it's going to be on lkml, and you already wrote your paper, so the next step would be to get together face-to-face with some folks who have a clue. Well, unless you're willing to wait months for the right people to show up in the Bay Area, IRC is the way to go.

Actually, I haven't written a paper. A paper is something which lays out

goals
architecture
milestones
design details

and should be sufficient to make the project happen should I be hit by a bus. That's my main complaint with IRC, it requires me to keep coming back and explaining the same thing over and over again.

Quoted 6 lines

Here's an idea: you go try and get some traction on the OS cluster idea. I'll give you 6 months and we'll see what happens. If nothing has happened, I'll produce a decent paper describing it and then we wait another 6 months to see what happens. I'll bet you 10:1 odds I get a lot more action from a lot more people than you do. Nope, wait, make that 100:1 odds.

Sorry, reverse psychology doesn't doesn't work that well on me ;)

Quoted 9 lines

I've seen how little I manage to get done by talking. Talk is cheap. I've also seen how much I get done when I write a paper which other people can pass around, think about, discuss, and implement. A senior guy at Morgan Stanley (hi marc) once told me "if you want to get things done, write them down". And in my case, since people tend to like to argue with me rather than listen to me (yup, it's my fault, my "style" leaves "room for improvement" translation: sucks rocks), a paper is far more effective. My style is pretty much removed from the equation.

Quoted 5 lines

I can just see me on IRC, all I'd be getting is style complaints while people successfully avoid the real points. Look at the last 8 years of LKML. I'd say most of the effect was from the LMbench paper and maybe a few threads on performance which would have been more effective if I'd written a detailed paper explaining my point of view.

By all means, write the detailed paper if you've got time, then make sure people have read it before you talk to them. But trust me, there are people hanging out on IRC right now who have more than a clue about exactly the subject you're interested in, who would need no more than a short note to be up to speed and ready to address the real issues intelligently.

--
Daniel

#86 lore raw

Re: IRC (was: Scheduler)

In reply to M. Edward (Ed) Borasky
Quoted 9 lines · M. Edward (Ed) Borasky

On Wed, 19 Dec 2001, Daniel Phillips wrote:
I've watched with great interest the discussion of IRC for Linux folk and have yet to see anyone mention server/network names and channel names. I've been on IRC for 2.5 years -- I tracked the Y2K transition on IRC despite all the dire warnings that evil impulses were going to shoot down the wire and fry the LCD screen on my laptop. So -- just where exactly *is* this water that is so fine? mIRC 5.91 and I await with bated breath. (Yes, I do use a Windows IRC client -- wanna make something of it? :-)

/server irc.openprojects.net

/list

--
Daniel

#87 lore raw

Re: IRC (was: Scheduler)

In reply to M. Edward (Ed) Borasky
Quoted 3 lines · M. Edward (Ed) Borasky

mIRC 5.91 and I await with
bated breath. (Yes, I do use a Windows IRC client -- wanna make something of it? :-)

(shrug) whatever turns you on, I guess...

I will mention that there is this cool OS
called Linux, you might have heard of it -

There are a number of very nice irc clients
available for it

;-)

jjs

#6 lore raw

aio

In reply to Daniel Phillips
Quoted 10 lines · Linus Torvalds
Quoted 4 lines · Daniel Phillips

It's AIO we're talking about, right? AIO is interesting to quite a few people. I'd read the thread. I'd also read any background material that Ben would be so kind as to supply.

Case closed.

Dan didn't even _know_ of the patches.

He doesn't read l-k apparently.

Quoted 4 lines

Ben: end of discussion. I will _not_ apply any patches for aio if they aren't openly discussed. We're not microsoft, and we're not Sun. We're "Open Source", not "cram things down peoples throat and spring new features on them as a fait accompli".

Discuss them then to your heart's content. I've posted announcements to both l-k and linux-aio which are both on marc.theaimsgroup.com if you're too lazy to get your IS to add a new list yo the internal news gateway.

Quoted 3 lines

The ghost of "binary compatibility" is not an issue - if Ben or anytbody else finds a flaw with the design, it's a hell of a lot better to have that flaw fixed _before_ it's part of my kernel rather than afterwards.

Thanks for the useful feedback on the userland interface then. Evidently nobody cares within the community about improving functionality on a reasonable timescale. If this doesn't change soon, Linux is doomed.

-ben

#34 lore raw

Re: aio

In reply to Daniel Phillips
Quoted 7 lines · Ben LaHaise
Quoted 4 lines

Ben: end of discussion. I will _not_ apply any patches for aio if they aren't openly discussed. We're not microsoft, and we're not Sun. We're "Open Source", not "cram things down peoples throat and spring new features on them as a fait accompli".

Discuss them then to your heart's content. I've posted announcements to both l-k and linux-aio which are both on marc.theaimsgroup.com ...

Ben, I think maybe we need to get people excited about your patches, and build up a user base, before putting them in the mainline kernel. The volume on the linux-aio list has been pretty light, and the visibility of the patches has been pretty low.

I know I volunteered to write some doc for your aio, and haven't delivered; thus I'm contributing to the problem. Mea culpa. But there are some small things that could be done. A freshmeat.net entry for the project, for instance. Shall I create one, or would you rather do it?
A home page for linux-aio would be great, too.
- Dan

#35 lore raw

Re: aio

In reply to Daniel Phillips
Quoted 8 lines · Ben LaHaise
Quoted 7 lines · Linus Torvalds
Quoted 4 lines · Daniel Phillips

It's AIO we're talking about, right? AIO is interesting to quite a few people. I'd read the thread. I'd also read any background material that Ben would be so kind as to supply.

Quoted 5 lines
Quoted 3 lines

Case closed.

Dan didn't even _know_ of the patches.

He doesn't read l-k apparently.

Dan Kegel put it succinctly:

http://marc.theaimsgroup.com/?l=linux-aio&m=100879005201064&w=2

Your original patch is here, and I do remember the post at the time:

http://marc.theaimsgroup.com/?l=linux-kernel&m=98114243104171&w=2

This post provides *zero* context. Ever since, I've been expecting to see some explanation of what the goals are, what the design principles are, what the historical context is, etc. etc., and that hasn't happened.

I've got a fairly recent version of the patch too, it's a little too long to just sit down and read, to reverse-engineer the above information. What's missing here is some kind of writeup like Suparna did for Jens' bio patch (hint, hint). There's no reason why every single person who might be interested should have to take the time to reverse-engineer the patch without context.

As Linus points out, the active discussion hasn't happened yet.

--
Daniel

#7 lore raw

Re: aio

In reply to Ben LaHaise
Quoted 3 lines · Ben LaHaise

Thanks for the useful feedback on the userland interface then. Evidently nobody cares within the community about improving functionality on a reasonable timescale. If this doesn't change soon, Linux is doomed.

Ben, maybe it's true, nobody cares :( This could be either bad or good. On one side it could be good because this means that everyone is happy with the kernel performance level and this could be due the fact that real world loads does not put their applications under stress. It could be bad because it's possible that exist applications that are currently under stress ( yes ), but their developers do not understand that by using different interfaces they can improve their software ( or they simply do not understand that the application is under stress ). Or maybe application developers are not in lk. Or maybe they're not willing to rewrite/experiment new APIs. On one side i understand that you can have an intrinsic attitude to push/defend your patch, while one the other side i can agree with the Linus point to have some kind of broad discussion/adoption about it. But if applications developers are not in this list there won't be a broad discussion and if the patch does not go inside the mainstream kernel "external" applications developers are not going to use it. The Linus point could be: "why do i have to merge a new api that has had a so cold discussion/adoption inside lk ?".
Yes egg-chicken draws the picture very well.

- Davide

#8 lore raw

Re: aio

In reply to Ben LaHaise

From: Ben LaHaise <[email protected]>
Date: Wed, 19 Dec 2001 13:57:08 -0500

Thanks for the useful feedback on the userland interface then. Evidently nobody cares within the community about improving functionality on a reasonable timescale. If this doesn't change soon, Linux is doomed.

Maybe it's because the majority of people don't care nor would ever need to use AIO. Are you willing to accept this possibly? :-) Linux is anything but doomed, because you will notice that the things that actually matter for most people are in fact improved and worked on within a reasonable timescale.

Only very specialized applications can even benefit from AIO. This doesn't make it useless, but it does decrease the amount of interest (and priority) anyone in the community will have in working on it.

Now, if these few and far between people who are actually interested in AIO are willing to throw money at the problem to get it worked on, that is how the "reasonable timescale" will be arrived at. And if they aren't willing to toss money at the problem, how important can it really be to them? :-)

Maybe, just maybe, most people simply do not care one iota about AIO.

Linux caters to the general concerns not the nooks and cranies, that is why it is anything but doomed.

#36 lore raw

Re: aio

In reply to Daniel Phillips
Quoted 5 lines · David S. Miller

Now, if these few and far between people who are actually interested in AIO are willing to throw money at the problem to get it worked on, that is how the "reasonable timescale" will be arrived at. And if they aren't willing to toss money at the problem, how important can it really be to them? :-)

People are throwing money at the problem. We're now at a point that in order to provide the interested people with something they can use, we need some kind of way to protect their applications against calling an unsuspecting new mmap syscall instead of the aio syscall specified in the kernel they compiled against.

Quoted 4 lines

Maybe, just maybe, most people simply do not care one iota about AIO.

Linux caters to the general concerns not the nooks and cranies, that is why it is anything but doomed.

What I'm saying is that for more people to play with it, it needs to be more widely available. The set of developers that read linux-kernel and linux-aio aren't giving much feedback. I do not expect the code to go into 2.5 at this point in time. All I need is a set of syscall numbers that aren't going to change should this implementation stand up to the test of time.

-ben

Signature · 1 line
Fish.
#38 lore raw

Re: aio

In reply to Daniel Phillips
Quoted 3 lines · Benjamin LaHaise

All I need is a set of syscall numbers that aren't going to change should this implementation stand up to the test of time.

The aio_* functions are part of POSIX and SUS, so merely reserving system call numbers for them does not seems a completely dumb thing to do, IMO.

-

#37 lore raw

Re: aio

In reply to Benjamin LaHaise
Quoted 6 lines · Benjamin LaHaise

What I'm saying is that for more people to play with it, it needs to be more widely available. The set of developers that read linux-kernel and linux-aio aren't giving much feedback. I do not expect the code to go into 2.5 at this point in time. All I need is a set of syscall numbers that aren't going to change should this implementation stand up to the test of time.

It would be nice to have a cooperation between glibc and the kernel to have syscalls mapped by name, not by number.
With name->number resolved by crtbegin.o reading a public kernel table or accessing a fixed-ID kernel map function and filling a map. So if internally ( at the application ) sys_getpid has index 0, the sysmap[0] will be filled with the id retrieved inside the kernel by looking up "sys_getpid".
Eat too spicy today ?

- Davide

#39 lore raw

Re: aio

In reply to Daniel Phillips
Followup to:  <[email protected]>
By author:    Andrew Morton <[email protected]>
In newsgroup: linux.dev.kernel
Quoted 5 lines

The aio_* functions are part of POSIX and SUS, so merely reserving system call numbers for them does not seems a completely dumb thing to do, IMO.

Yes, it is, unless you already have a design for how to map the aio_* library functions onto system calls.

-hpa

Signature · 3 lines
<[email protected]> at work, <[email protected]> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt	<[email protected]>
#40 lore raw

Re: aio

In reply to Daniel Phillips
Quoted 9 lines · David S. Miller

Maybe it's because the majority of people don't care nor would ever need to use AIO. Are you willing to accept this possibly? :-) Linux is anything but doomed, because you will notice that the things that actually matter for most people are in fact improved and worked on within a reasonable timescale.

Only very specialized applications can even benefit from AIO. This doesn't make it useless, but it does decrease the amount of interest (and priority) anyone in the community will have in working on it.

Folks doing serious server side Java and runtime internals would definitely be able to use this stuff, namely me. It'll remove the abuse of threading used to deal with large IO systems when NIO comes out for 1.4. And as a JVM engineer for the FreeBSD community I'm drooling over stuff like that.

Quoted 5 lines

Now, if these few and far between people who are actually interested in AIO are willing to throw money at the problem to get it worked on, that is how the "reasonable timescale" will be arrived at. And if they aren't willing to toss money at the problem, how important can it really be to them? :-)

Like the Java folks ? few and far between ? What you're saying it just plain outdated and from a previous generation of thinking that has become irrelevant as the community has grown.

Quoted 4 lines

Maybe, just maybe, most people simply do not care one iota about AIO.

Linux caters to the general concerns not the nooks and cranies, that is why it is anything but doomed.

Again, Linux collectively has outgrown that thinking and the scope of what the previous generation of engineers can responsible for, which is why it's important for folks like Ben should be encourage to take it to the next level.

bill

#41 lore raw

Re: aio

From: Bill Huey <[email protected]>
Date: Wed, 19 Dec 2001 17:16:31 -0800

Like the Java folks ? few and far between ?

Precisely, in fact. Anyone who can say that Java is going to be relevant in a few years time, with a straight face, is only kidding themselves.

Java is not something to justify a new kernel feature, that is for certain.

#9 lore raw

Re: aio

In reply to Ben LaHaise
Quoted 3 lines · David S. Miller

Precisely, in fact. Anyone who can say that Java is going to be relevant in a few years time, with a straight face, is only kidding themselves.

Oh give me coke shooting, Steeley Dan, late 70s bitter kernel programmer break...

Quoted 2 lines

Java is not something to justify a new kernel feature, that is for certain.

Java is here now and used extensively on server side applications. Simply dismissing it doesn't invalidate the claim that I made before about how this mentality is outdated.

The economic inertia of Java driven server applications should have enough force that it is justifyable to RedHat and other commerical organizations to support it regardless of what your current view is on this topic.

Even within the BSD/OS group at BSDi/WindRiver, (/me former BSD/OS engineer) some kind of dedicated async IO system inside kernel was talked about as highly desireable and possibly a more direct way of dealing with VM page/async IO event issues that don't map conceptually to a scheduler context cleanly.

AIO is good, plain and simple.

bill

#31 lore raw

Re: aio

In reply to Ben LaHaise

On Wed, Dec 19, 2001 at 05:20:46PM -0800, David S. Miller <[email protected]> wrote:

|    From: Bill Huey <[email protected]>
|    Like the Java folks ? few and far between ?
| Precisely, in fact.  Anyone who can say that Java is going to be
| relevant in a few years time, with a straight face, is only kidding
| themselves.

Maybe. I'm good at that.

| Java is not something to justify a new kernel feature, that is for | certain.

Of itself, maybe. (Though an attitude like yours is a core reason Java is spreading as slowly as it is - much like Linux desktops...)

However, heavily threaded apps regardless of language are hardly likely to disappear; threads are the natural way to write many many things. And if the kernel implements threads as on Linux, then the scheduler will become much more important to good performance.

Signature · 4 lines
Cameron Simpson, DoD#743        [email protected]    http://www.zip.com.au/~cs/

Questions are a burden to others,
	Answers, a prison for oneself.
#10 lore raw

Re: aio

In reply to Bill Huey

From: Bill Huey <[email protected]>
Date: Wed, 19 Dec 2001 18:26:28 -0800

The economic inertia of Java driven server applications should have enough force that it is justifyable to RedHat and other commerical organizations to support it regardless of what your current view is on this topic.

So they'll get paid to implement and support it, and that is precisely what is happening right now. And the whole point I'm trying to make is that that is where it's realm is right now.

If AIO was so relevant+sexy we'd be having threads of discussion about the AIO implementation instead of threads about how relevant it is or is not for the general populace. Wouldn't you concur? :-)

The people doing Java server applets are such a small fraction of the Linux user community.

#32 lore raw

Re: aio

In reply to Cameron Simpson

From: Cameron Simpson <[email protected]>
Date: Thu, 20 Dec 2001 13:37:05 +1100

(Though an attitude like yours is a core reason Java is spreading as slowly as it is - much like Linux desktops...)

It's actually Sun's fault more than anyone else's.

However, heavily threaded apps regardless of language are hardly likely to disappear; threads are the natural way to write many many things. And if the kernel implements threads as on Linux, then the scheduler will become much more important to good performance.

We are not talking about the scheduler, we are talking about AIO.

#47 lore raw

Re: aio

In reply to Bill Huey

On Wed, Dec 19, 2001 at 06:47:18PM -0800, David S. Miller <[email protected]> wrote:

|    (Though an attitude like yours is a core reason Java is
|    spreading as slowly as it is - much like Linux desktops...)
| It's actually Sun's fault more than anyone else's.

Debatable. But fortunately off topic.

|    However, heavily threaded apps regardless of language are hardly likely
|    to disappear; threads are the natural way to write many many things. And
|    if the kernel implements threads as on Linux, then the scheduler will
|    become much more important to good performance.
| We are not talking about the scheduler, we are talking about
| AIO.

It was in the same thread - I must have ignored the detail switch. Ignore me in turn. But while I'm here, tell me why async I/O is important to Java and not to anything else, which still seems the thrust of your remarks.

Signature · 5 lines
Cameron Simpson, DoD#743        [email protected]    http://www.zip.com.au/~cs/

Always code as if the guy who ends up maintaining your code will be a violent
psychopath who knows where you live.
	- Martin Golding, DoD #0236, [email protected]
#48 lore raw

Re: aio

From: Cameron Simpson <[email protected]>
Date: Thu, 20 Dec 2001 13:52:21 +1100

tell me why async I/O is important
to Java and not to anything else, which still seems the thrust of your remarks.

Not precisely my thrust, which is that AIO is not important to any significant population of Linux users, it is "nook and cranny" in scope. And that those "nook and cranny" folks who really find it important can get paid implementation+support of AIO.

#11 lore raw

Re: aio

In reply to David S. Miller
Quoted 19 lines · David S. Miller

Date: Wed, 19 Dec 2001 18:45:27 -0800 (PST)
From: "David S. Miller" <[email protected]>
To: [email protected]
Cc: [email protected], [email protected], [email protected],
[email protected]
Subject: Re: aio

From: Bill Huey <[email protected]>
Date: Wed, 19 Dec 2001 18:26:28 -0800

The economic inertia of Java driven server applications should have enough force that it is justifyable to RedHat and other commerical organizations to support it regardless of what your current view is on this topic.

So they'll get paid to implement and support it, and that is precisely what is happening right now. And the whole point I'm trying to make is that that is where it's realm is right now.

Quoted 6 lines

If AIO was so relevant+sexy we'd be having threads of discussion about the AIO implementation instead of threads about how relevant it is or is not for the general populace. Wouldn't you concur? :-)

The people doing Java server applets are such a small fraction of the Linux user community.

True for now, but if we want to expand linux into the enterprise and the desktop to a greater degree, then we need to support the Java community to draw them and their management in, rather than delaying beneficial features until their number on lkml reaches critical mass for a design discussion.

-
-----------------------------------------------------------------
John Heil
South Coast Software
Custom systems software for UNIX and IBM MVS mainframes 1-714-774-6952
[email protected]
http://www.sc-software.com
-----------------------------------------------------------------

#12 lore raw

Re: aio

From: John Heil <[email protected]>
Date: Wed, 19 Dec 2001 18:57:34 +0000 ( )

True for now, but if we want to expand linux into the enterprise and the desktop to a greater degree, then we need to support the Java community to draw them and their management in, rather than delaying beneficial features until their number on lkml reaches critical mass for a design discussion.

Firstly, you say this as if server java applets do not function at all or with acceptable performance today. That is not true for the vast majority of cases.

If java server applet performance in all cases is dependent upon AIO (it is not), that would be pretty sad. But it wouldn't be the first time I've heard crap like that. There is propaganda out there telling people that 64-bit address spaces are needed for good java performance. Guess where that came from? (hint: they invented java and are in the buisness of selling 64-bit RISC processors)

#62 lore raw

Re: aio

In reply to Bill Huey
Quoted 7 lines · David S. Miller

So they'll get paid to implement and support it, and that is precisely what is happening right now. And the whole point I'm trying to make is that that is where it's realm is right now.

If AIO was so relevant+sexy we'd be having threads of discussion about the AIO implementation instead of threads about how relevant it is or is not for the general populace. Wouldn't you concur? :-)

I attribute the lack of technical discussion to the least common denomiator culture of the Linux community and not the merits of the actual technical system itself. That's what linux-aio@ is for...

And using lkml as a AIO forum is probably outside of the scope of this list and group.

Quoted 2 lines

The people doing Java server applets are such a small fraction of the Linux user community.

Yeah, but the overall Unix community probably has something different to say about that, certainly. Even in BSD/OS, this JVM project I've been working on is recognized as one of the most important systems second (probably) only to the kernel itself. And, IMO, they have a more balanced view of this language system and the value of it economically as a money making platform instead of showing off to their peers. It's a greatly anticipated project in all of the BSDs.

That's my semi-obnoxious take on it. ;-)

bill

#63 lore raw

Re: aio

From: Bill Huey <[email protected]>
Date: Wed, 19 Dec 2001 19:07:16 -0800

And using lkml as a AIO forum is probably outside of the scope of this list and group.

This whole thread exists because Linus wants public general and technical discussion on lkml of new features to happen before he considers putting them into the tree, and the fact that they are not in the tree because he isn't seeing such enthusiastic discussions happening at all.

I don't think AIO, because of it's non-trivial impact to the tree, is at all outside the scope of this list. This is in fact the place where major stuff like AIO is meant to be discussed, not some special list where only "AIO people" hang out, of course people on that list will be enthusiastic about AIO!

Frankly, on your other comments, I don't give a rats ass what BSD/OS people are doing about, nor how highly they rate, Java. That is neither here nor there. Java is going to be dead in a few years, and let's just agree to disagree about this particular point ok?

#15 lore raw

Re: aio

In reply to David S. Miller
Quoted 6 lines · David S. Miller

Firstly, you say this as if server java applets do not function at all or with acceptable performance today. That is not true for the vast majority of cases.

If java server applet performance in all cases is dependent upon AIO (it is not), that would be pretty sad. But it wouldn't be the first

Java is pretty incomplete in this area, which should be addressed to a great degree in the new NIO API.

The core JVM isn't dependent on this stuff per se for performance, but it is critical to server side programs that have to deal with highly scalable IO systems, largely number of FDs, that go beyond the current expressiveness of select()/poll().

This is all standard fare in *any* kind of high performance networking application where some kind of high performance kernel/userspace event delivery system is needed, kqueue() principally.

Quoted 4 lines

time I've heard crap like that. There is propaganda out there telling people that 64-bit address spaces are needed for good java performance. Guess where that came from? (hint: they invented java and are in the buisness of selling 64-bit RISC processors)

What ? oh god. HotSpot is a pretty amazing compiler and it performs well. Swing does well now, but the lingering issue in Java is the shear size of it and possibly GC issues. It pretty clear that it's going to get larger, which is fine since memory is cheap.

bill

#13 lore raw

Re: aio

In reply to David S. Miller
Quoted 20 lines · David S. Miller

Date: Wed, 19 Dec 2001 19:06:29 -0800 (PST)
From: "David S. Miller" <[email protected]>
To: [email protected]
Cc: [email protected], [email protected], [email protected],
[email protected], [email protected]
Subject: Re: aio

From: John Heil <[email protected]>
Date: Wed, 19 Dec 2001 18:57:34 +0000 ( )

True for now, but if we want to expand linux into the enterprise and the desktop to a greater degree, then we need to support the Java community to draw them and their management in, rather than delaying beneficial features until their number on lkml reaches critical mass for a design discussion.

Firstly, you say this as if server java applets do not function at all or with acceptable performance today. That is not true for the vast majority of cases.

Quoted 7 lines

If java server applet performance in all cases is dependent upon AIO (it is not), that would be pretty sad. But it wouldn't be the first time I've heard crap like that. There is propaganda out there telling people that 64-bit address spaces are needed for good java performance. Guess where that came from? (hint: they invented java and are in the buisness of selling 64-bit RISC processors)

Agree. However, put your business hat for a minute. We want increased market share for linux and a lot of us, you included, live by it. If aio, the proposed implementation or some other, can provide an adequate performance boost for Java (yet to be seen), that at least allows the marketing folks one more argument to draw users to linux. Do think the trade mags etc don't watch what we do? A demonstrable advantage in Java performance is marketable and beneficial to all.

-
-----------------------------------------------------------------
John Heil
South Coast Software
Custom systems software for UNIX and IBM MVS mainframes 1-714-774-6952
[email protected]
http://www.sc-software.com
-----------------------------------------------------------------

#17 lore raw

Re: aio

In reply to Bill Huey
Quoted 5 lines · David S. Miller

I don't think AIO, because of it's non-trivial impact to the tree, is at all outside the scope of this list. This is in fact the place where major stuff like AIO is meant to be discussed, not some special list where only "AIO people" hang out, of course people on that list will be enthusiastic about AIO!

Well maybe yourself and others should make some comments about it then.

Quoted 4 lines

Frankly, on your other comments, I don't give a rats ass what BSD/OS people are doing about, nor how highly they rate, Java. That is neither here nor there. Java is going to be dead in a few years, and let's just agree to disagree about this particular point ok?

Who cares about Java? What about high performance LDAP servers or tux-like userspace performance? How about faster select and poll? An X server that doesn't have to make a syscall to find out that more data has arrived? What about nbd or iscsi servers that are in userspace and have all the benefits that their kernel side counterparts do?

-ben

Signature · 1 line
Fish.
#74 lore raw

Re: aio

In reply to Daniel Phillips
Quoted 5 lines · Dan Kegel

I know I volunteered to write some doc for your aio, and haven't delivered; thus I'm contributing to the problem. Mea culpa. But there are some small things that could be done. A freshmeat.net entry for the project, for instance. Shall I create one, or would you rather do it?
A home page for linux-aio would be great, too.

I've started writing some web pages, and will gratiously accept additional docs/text/questions. Freshmeat will get an entry right after I send this out.

-ben

Signature · 1 line
Fish.
#33 lore raw

Re: aio

In reply to Ben LaHaise

On Thu, Dec 20, 2001 at 12:07:21AM -0500, Pete Zaitcev <[email protected]> wrote:

| >[...]
| > However, heavily threaded apps regardless of language are hardly likely
| > to disappear; threads are the natural way to write many many things. And
| > if the kernel implements threads as on Linux, then the scheduler will
| > become much more important to good performance.
| 
| Cameron seems to be arguing with DaveM,

About the wrong things, but no matter.

| but subconsciously he
| only supports DaveM's point about AIO: Java cannot make use
| of AIO, so that's one (large or small, important or unimportant)
| group of applications down from the count.

You're sure? Java _authors_ can't make use of it, but Java _implementors_ probably have good reason to want it ...

| Just trying to keep on topic :)

Whatever for?
--
Cameron Simpson, DoD#743 [email protected] http://www.zip.com.au/~cs/

Reaching consensus in a group often is confused with finding the right
answer. - Norman Maier

#14 lore raw

Re: aio

In reply to John Heil

From: John Heil <[email protected]>
Date: Wed, 19 Dec 2001 19:30:13 +0000 ( )

Agree. However, put your business hat for a minute. We want increased market share for linux and a lot of us, you included, live by it.

Oh my buisness hat is certainly on, which is why I keep talking about the people who need this "paying for implementation and support of AIO for Linux". :-)

Make no mistake, I do agree with your points though in general.

But those things are not dependent upon "standard Linus Linux" having AIO first, this is what vendors do for differentiation by shipping feature X in their kernel before others.

#18 lore raw

Re: aio

In reply to Benjamin LaHaise

From: Benjamin LaHaise <[email protected]>
Date: Wed, 19 Dec 2001 22:47:17 -0500

Well maybe yourself and others should make some comments about it then.

Because, like I keep saying, it is totally uninteresting for most of us.

Who cares about Java?

The people telling me on this list how important AIO is for Linux :-)

What about high performance LDAP servers or tux-like userspace performance?

People have done "faster than TUX" userspace web service with the current kernel, that is without AIO. There is no reason you can't do a fast LDAP server with the current kernel either, any such claim is simply rubbish. Why do we need AIO again?

How about faster select and poll?

You don't need faster select and poll as demonstrated by the userspace "faster than TUX" example above.

An X server that doesn't have to make a syscall to find out that more data has arrived?

Who really needs this kind of performance improvement? Like anyone really cares if their window gets the keyboard focus or a pixel over a AF_UNIX socket a few nanoseconds faster. How many people do you think believe they have unacceptable X performance right now and that select()/poll() syscalls overhead is the cause? Please get real.

People who want graphics performance are not pushing their data through X over a filedescriptor, they are either using direct rendering in the app itself (ala OpenGL) or they are using shared memory for the bulk of the data (ala Xshm or Xv extensions).

What about nbd or iscsi servers that are in userspace and have all the benefits that their kernel side counterparts do?

I do not buy this claim that it is not possible the achieve the desired performance using existing facilities.

The only example of AIO benefitting performance I see right now are databases.

Franks a lot,
David S. Miller
[email protected]

#49 lore raw

Re: aio

In reply to David S. Miller
Quoted 5 lines · David S. Miller

Not precisely my thrust, which is that AIO is not important to any significant population of Linux users, it is "nook and cranny" in scope. And that those "nook and cranny" folks who really find it important can get paid implementation+support of AIO.

I disagree - we can probably make the aio by Ben quite important. Done right, it becomes a very natural way of doing event handling, and it could very well be rather useful for many things that use select loops right now.

So I actually like the thing as it stands now. What I don't like is how it's been handled, with people inside Oracle etc working with it, but _not_ people on the kernel mailing list. I don't worry about the code nearly as much as I worry about people starting to clique together.

Linus

#50 lore raw

Re: aio

From: Linus Torvalds <[email protected]>
Date: Wed, 19 Dec 2001 21:47:18 -0800 (PST)

it could very well be rather useful for many things that use select loops right now.

Then let us agree to disagree. :-) I think it's potential advantages, and how many things really "require it" for better performance, is being blown out of proportion.

#65 lore raw

Re: aio

In reply to Bill Huey
Quoted 4 lines · David S. Miller

How about faster select and poll?

You don't need faster select and poll as demonstrated by the userspace "faster than TUX" example above.

Step back for a moment. I know of phttpd and zeus. They both have a serious problem: they fall down when the load on the system exceeds the capabilities of the cpu. If you'd bother to take a look at the aio api I'm proposing, it has less overhead under heavy load as events get coalesced. Even then, the overhead under light load is less than signals or select or poll.

-ben

Signature · 1 line
Fish.
#58 lore raw

Re: aio

In reply to David S. Miller
Quoted 3 lines · David S. Miller

Then let us agree to disagree. :-) I think it's potential advantages, and how many things really "require it" for better performance, is being blown out of proportion.

Show me how to make a single process server that can handle 100000 or more open tcp sockets that doesn't collapse under load. I can do it with aio; can you do it without?

-ben

Signature · 1 line
Fish.
#66 lore raw

Re: aio

In reply to Benjamin LaHaise

From: Benjamin LaHaise <[email protected]>
Date: Thu, 20 Dec 2001 00:58:03 -0500

Step back for a moment. I know of phttpd and zeus. They both have a serious problem: they fall down when the load on the system exceeds the capabilities of the cpu. If you'd bother to take a look at the aio api I'm proposing, it has less overhead under heavy load as events get coalesced. Even then, the overhead under light load is less than signals or select or poll.

No I'm not talking about phttpd nor zeus, I'm talking about the guy who did the hacks where he'd put the http headers + content into a seperate file and just sendfile() that to the client.

I forget what his hacks were named, but there certainly was a longish thread on this list about it about 1 year ago if memory serves.

#59 lore raw

Re: aio

In reply to Benjamin LaHaise

From: Benjamin LaHaise <[email protected]>
Date: Thu, 20 Dec 2001 00:59:28 -0500

Show me how to make a single process server that can handle 100000 or more open tcp sockets that doesn't collapse under load. I can do it with aio; can you do it without?

Why are you limiting me to a single process? :-) Can I have at least 1 per cpu possibly? :-)))

#51 lore raw

Re: aio

In reply to Linus Torvalds
Quoted 2 lines · David S. Miller

Why are you limiting me to a single process? :-) Can I have at least 1 per cpu possibly? :-)))

1 process. 1 cpu machine. 1 gige card. As much ram as you want. No syscalls. Must exhibit a load curve similar to:

	y
	|  ...............
	| .
	|.
	+----------------x

Where x == requests per second sent to the machine and y is the number of resposes per second sent out of the machine. Hint: read the phttpd and /dev/poll papers for an idea of the breakdown that happens for larger values of x (make the cpu slower to cause the interesting points to move lower). For a third dimension to the graph, make the number of total connections the z axis.

-ben

Signature · 1 line
Fish.
#60 lore raw

Re: aio

In reply to David S. Miller

Could we get back on track, and possibly discuss the patches themselves, ok? We want _constructive_ criticism of the interfaces.

I think it's clear that many people do want to have aio support. At least as far as I'm concerned, that's not the reason I want to have public discussion. I want to make sure that the interfaces are good for aio users, and that the design isn't stupid.

If somebody can point to a better way of doing aio, and giving good arguments for that, more power to him. But let's not go down the path of "_I_ don't like aio, so _you_ must be stupid".

Linus

#52 lore raw

Re: aio

In reply to Benjamin LaHaise

From: Benjamin LaHaise <[email protected]>
Date: Thu, 20 Dec 2001 01:07:42 -0500

1 process. 1 cpu machine. 1 gige card. As much ram as you want. No syscalls. Must exhibit a load curve similar to:

y

   	|  ...............
   	| .
   	|.

+----------------x

Where x == requests per second sent to the machine and y is the number of resposes per second sent out of the machine. Hint: read the phttpd and /dev/poll papers for an idea of the breakdown that happens for larger values of x (make the cpu slower to cause the interesting points to move lower). For a third dimension to the graph, make the number of total connections the z axis.

Ok, TUX can do it. Now list for me some server that really matters other than web and ftp? If you say databases, then I agree with you but I will also reiterate how the people who need that level of database performance is "nook and cranny".

I think there is nothing wrong with doing a TUX module for situations where 1) the server is important for enough people and 2) scaling to the levels you are talking about is a real issue for that service.

#53 lore raw

Re: aio

Quoted 3 lines · David S. Miller

Ok, TUX can do it. Now list for me some server that really matters other than web and ftp?

Now now, that's unfair. We should be able to do it in user space.

I think the question you _should_ be lobbying at Ben and the other aio people is how the aio stuff could do zero-copy from disk cache to the network, ie do the things that Tux does internally where it does nonblocking reads from disk ad then sends them out non-blocking to the network without havign to copy the data _or_ have to use extremely expensive TLB mapping tricks to get at it..

Ie tie the "sendfile" and "aio" threads together, and ask Ben if we can do aio-sendfile and have thousands of asynchronous sendfiles going on at the same time, like Tux can do. And if not, then why not? Missing or bad interfaces?

Ben? Doing user-space IO is all well and good, but that extra copy and TLB stuff kills you. Tell us how to do it ;)

Linus

#20 lore raw

Re: aio

In reply to Benjamin LaHaise
Quoted 6 lines · David S. Miller

No I'm not talking about phttpd nor zeus, I'm talking about the guy who did the hacks where he'd put the http headers + content into a seperate file and just sendfile() that to the client.

I forget what his hacks were named, but there certainly was a longish thread on this list about it about 1 year ago if memory serves.

Would that be Fabio Riccardi's X15 stuff?

mrc

Signature · 3 lines
     Mike Castle      [email protected]      www.netcom.com/~dalgoda/
    We are all of us living in the shadow of Manhattan.  -- Watchmen
fatal ("You are in a maze of twisty compiler features, all different"); -- gcc
#21 lore raw

Re: aio

Quoted 9 lines · Mike Castle
Quoted 6 lines · David S. Miller

No I'm not talking about phttpd nor zeus, I'm talking about the guy who did the hacks where he'd put the http headers + content into a seperate file and just sendfile() that to the client.

I forget what his hacks were named, but there certainly was a longish thread on this list about it about 1 year ago if memory serves.

Would that be Fabio Riccardi's X15 stuff?

Yes. I was about to reply to this effect.

X15 was a userspace httpd that operated using the Tux-designed constructs -- sendfile and such. IIRC, Ingo actually pointed out some things Fabio did were non-RFC (sending the static headers may of been one of them, since the timestamp was wrong) and Fabio made a lot of changes. X15 seemed promising, especially since it trumpeted that Linux "worked" without sticking things in kernel-space, but I don't remember if we ever saw source (let alone a free license)?

Robert Love

#23 lore raw

Re: aio

In reply to Benjamin LaHaise
Quoted 3 lines · Robert Love

changes. X15 seemed promising, especially since it trumpeted that Linux "worked" without sticking things in kernel-space, but I don't remember if we ever saw source (let alone a free license)?

We did, but licensing was personal use only. It's all in the archives for those curious. Though following the URLs Fabio posted just got me to login screen.

mrc

Signature · 3 lines
     Mike Castle      [email protected]      www.netcom.com/~dalgoda/
    We are all of us living in the shadow of Manhattan.  -- Watchmen
fatal ("You are in a maze of twisty compiler features, all different"); -- gcc
#67 lore raw

Re: aio

In reply to Bill Huey
Quoted 10 lines · David S. Miller

From: Benjamin LaHaise <[email protected]>
Date: Wed, 19 Dec 2001 22:47:17 -0500
An X server that doesn't have to make a syscall to find out that more data has arrived?

Who really needs this kind of performance improvement? Like anyone really cares if their window gets the keyboard focus or a pixel over a AF_UNIX socket a few nanoseconds faster. How many people do you think believe they have unacceptable X performance right now and that select()/poll() syscalls overhead is the cause? Please get real.

I care, I always like faster graphics.

Quoted 4 lines

People who want graphics performance are not pushing their data through X over a filedescriptor, they are either using direct rendering in the app itself (ala OpenGL) or they are using shared memory for the bulk of the data (ala Xshm or Xv extensions).

You're probably overgeneralizing. Actually, I run games on my server and display the graphics on my laptop. It works. I'd be happy if it was faster.

I don't see right off how AIO would make that happen though. Ben, could you please enlighten me, what would be the mechanism? Are other OSes doing X with AIO?

--
Daniel

#54 lore raw

Re: aio

In reply to Linus Torvalds
Quoted 6 lines · Linus Torvalds

I think the question you _should_ be lobbying at Ben and the other aio people is how the aio stuff could do zero-copy from disk cache to the network, ie do the things that Tux does internally where it does nonblocking reads from disk ad then sends them out non-blocking to the network without havign to copy the data _or_ have to use extremely expensive TLB mapping tricks to get at it..

months ago i already offered Ben to port TUX to the aio interfaces once they are available in the kernel. Unfortunately right now i cant afford maintaining two separate TUX trees - so it's a chicken and egg thing in this context.

But once aio is available, i *will* do it, because one of Ben's goals fully state-machine-driven async block IO, which i'd like to use (and test, and finetune, and improve) very much. (right now TUX does async block IO via helper kernel threads. Async net-io is fully IRQ-driven.) I'd also like to prove that our aio interfaces are capable.

there are two possibilities i can think of:

1) lets get Ben's patch in but do *not* export the syscalls, yet.

2) find some nice way of doing 'experimental syscalls', which are not guaranteed to stay that way. (Perhaps this is a naive proposition, often there is nothing more permanent than temporary solutions.) Something like reserving 'temporary' syscalls at the end of the syscall space, which would be frequently moved/removed/renamed just to keep folks from relying on it. No interface is guaranteed. Perhaps some technical solution can be find to make these syscalls truly temporary.

i'm sure people will get excited about (ie. use) aio once it's in the kernel. Ben is very good at coding, perhaps not as good at PR, but should such level of PR really be a natural part of Linux development?

Quoted 4 lines

Ie tie the "sendfile" and "aio" threads together, and ask Ben if we can do aio-sendfile and have thousands of asynchronous sendfiles going on at the same time, like Tux can do. And if not, then why not? Missing or bad interfaces?

i'd love to find out. *If* it's guaranteed that some sort of sane aio will always be available from the point on it's introduced into the kernel then i'll switch TUX to it. (it will change TUX upside down, this is why i cannot maintain two separate TUX trees.) TUX doesnt need stable interfaces. While TUX might not be as important, usage-wise, it's certainly a good playing ground for such things.

Ingo

#19 lore raw

Re: aio

In reply to David S. Miller
Quoted 6 lines · Ingo Molnar

we need a sane interface that covers *all* sorts of IO, not just sockets. I used to have exactly the same optinion as you have now, but now i'd like to have a common async IO interface that will cover network IO, block IO [or graphics IO, or whatever comes up]. We should have something saner and more explicit than a side-branch of fcntl() handling the socket fasync code.

I second this wholeheartedly. And I believe there are still more motivations for providing asynchronous interfaces for all I/O in the realm of assisting the userland:

(1) It would simplify the ways applications have and the kernel overhead of responding to user input while I/O is in progress.

(2) It would provide a more efficient way to do M:N threading than watchdogs and nonblocking poll/select in itimers.

Cheers,
Bill

#61 lore raw

Re: aio

In reply to David S. Miller
Quoted 4 lines · Linus Torvalds

Could we get back on track, and possibly discuss the patches themselves, ok? We want _constructive_ criticism of the interfaces.

I think it's clear that many people do want to have aio support. At least

Yes, to add to the lot, even though you probably don't need any more proof :) we have at least 3 different products requiring this. Both for scalable communications aio (large no. of connections) as well as file/disk aio. In fact, one of the things that I've been worried about once 2.5 opened up and the bio changes started flowing in (much as I was delighted to see Jens' stuff finally getting integrated), was whether this would mean a longer timeframe before we can hope to see aio in a distribution (which is the question which I have to respond to our product groups about).

Quoted 4 lines

as far as I'm concerned, that's not the reason I want to have public discussion. I want to make sure that the interfaces are good for aio users, and that the design isn't stupid.

My feeling about this is that we shouldn't necessarily need to have the entire aio code in perfect shape and bring it in all in one shot. The thing I like about the design is that it is quite possible to split this up into smaller core patches and bring them in slowly. And I agree with Ingo that we should be able to start stabilizing the basic internal mechanisms or foundations on which aio is built before we freeze the external interfaces. In fact, these two things could happen parallely. So I was hoping that an evolutionary approach would work. In the current design, the aio path is handled separately from the normal i/o paths, so the impact on existing interfaces is less. It mainly affects aio users. So it ought to be possible to integrate it in a way that doesn't hurt regular operations, and it should be easier to change it once it is in without breaking too many things.

Existing aio users on other platforms that I have come across seem to use either POSIX aio (for file /disk aio) or completion port style interfaces (mainly for communications aio), both of which seem to be possible with Ben's implementation. One has to explicitly associate each i/o with the completion queue (ctx), rather than associate an fd as a whole with it so that all completion events on that fd come to the ctx. That should be OK. Besides with async poll support we can have per-file readiness notification as well. I was hoping for the async poll piece being available early to exercise the top half or event handling side of aio, so we have scalable select/poll support, so was focussing on that part to start with.

Your point about some critical discussion of the interfaces and the design is well-taken. We have had a few discussions on the aio mailing and more on irc about some aspects, but not quite a thorough out and out analysis of pros and cons of the whole design. I just started writing some points here, but then realized that it is going to take much longer, so decided to do that while working with Ben on the documentation, and discuss more after that.

Regards
Suparna

#64 lore raw

Re: aio

In reply to David S. Miller
Quoted 12 lines · David S. Miller

From: Bill Huey <[email protected]>
Date: Wed, 19 Dec 2001 19:07:16 -0800

And using lkml as a AIO forum is probably outside of the scope of this list and group.

This whole thread exists because Linus wants public general and technical discussion on lkml of new features to happen before he considers putting them into the tree, and the fact that they are not in the tree because he isn't seeing such enthusiastic discussions happening at all.

YES, and he is right doing so.

Quoted 5 lines

I don't think AIO, because of it's non-trivial impact to the tree, is at all outside the scope of this list. This is in fact the place where major stuff like AIO is meant to be discussed, not some special list where only "AIO people" hang out, of course people on that list will be enthusiastic about AIO!

agreed

Quoted 5 lines

Frankly, on your other comments, I don't give a rats ass what BSD/OS people are doing about, nor how highly they rate, Java. That is neither here nor there. Java is going to be dead in a few years, and let's just agree to disagree about this particular point ok?

mmhh, java will not be death untill a lot of commecial software will use it for graphical interfaces.
Infact it is simpler and cheaper for them to use java, and so we have to deal with this bad future, where a dead language will be keept alive by software houses.
That said, should we care about this? In my opinion, NO. and why we should? when there are no good technical reasons, political reasons should please disappear.

Luigi

#68 lore raw

Re: aio

In reply to Bill Huey
Quoted 6 lines · David S. Miller

If AIO was so relevant+sexy we'd be having threads of discussion about the AIO implementation instead of threads about how relevant it is or is not for the general populace. Wouldn't you concur? :-)

The people doing Java server applets are such a small fraction of the Linux user community.

People writing code for NT/Win2K/WinXP are being channelled into using AIO because that's the way to do things there (NT doesn't really support nonblocking I/O). Thus another valid economic reason AIO is important is to make it easier to port code from NT. I have received requests from NT folks for things like aio_recvfrom() (and have passed them on to Ben), so I'm not just guessing here.

As should be clear from my c10k page, I love nonblocking I/O, but I firmly believe that some form of AIO is vital.

- Dan

#69 lore raw

Re: aio

In reply to Bill Huey
Quoted 2 lines · David S. Miller

The people doing Java server applets are such a small fraction of the Linux user community.

The people doing Java Servlets are maybe a small fraction of the Linux Kernel Hackers community. Not the user community. They simply do not show up here because they don't care for Linux 2.5.0-rc1-prepatched.

Short head count: Who here is also a regular reader on Apache Jakarta lists?

Kernel hacking and Java most of the times doesn't mix. And Java folks are completly ambivalent to their OS: If Linux doesn't deliver, well. Windows, Solaris and BSD do. That's what Java is all about.

Regards
Henning

Signature · 5 lines
Dipl.-Inf. (Univ.) Henning P. Schmiedehausen       -- Geschaeftsfuehrer
INTERMETA - Gesellschaft fuer Mehrwertdienste mbH     [email protected]

Am Schwabachgrund 22  Fon.: 09131 / 50654-0   [email protected]
D-91054 Buckenhof     Fax.: 09131 / 50654-20   
#27 lore raw

Re: aio

In reply to Bill Huey
Quoted 2 lines · David S. Miller

neither here nor there. Java is going to be dead in a few years, and let's just agree to disagree about this particular point ok?

Care to point out why? Because of Sun or because of C#?

Regards
Henning

Signature · 5 lines
Dipl.-Inf. (Univ.) Henning P. Schmiedehausen       -- Geschaeftsfuehrer
INTERMETA - Gesellschaft fuer Mehrwertdienste mbH     [email protected]

Am Schwabachgrund 22  Fon.: 09131 / 50654-0   [email protected]
D-91054 Buckenhof     Fax.: 09131 / 50654-20   
#55 lore raw

Re: aio

In reply to Ingo Molnar
Quoted 3 lines · Ingo Molnar

there are two possibilities i can think of:

1) lets get Ben's patch in but do *not* export the syscalls, yet.

This is an excellent way to give aio the testing and exposure Linus wants without getting into the commitment / syscall mess.

Stick aio in the kernel, play with it via Tux, etc. The really interested can add temporary syscalls. aio (which I like, btw) will get testing and in time, once proven, we can add the syscalls.

Comments?

Robert Love

#28 lore raw

Re: aio

In reply to Henning Schmiedehausen
Quoted 6 lines · Henning Schmiedehausen
Quoted 2 lines · David S. Miller

neither here nor there. Java is going to be dead in a few years, and let's just agree to disagree about this particular point ok?

Care to point out why? Because of Sun or because of C#?

Because MSFT is bigger than SUNW :-) As that great American philosopher, Damon Runyon, once said, "The race is not always to the swift, nor the battle to the strong -- but that's the way to bet!"
--
M. Edward Borasky

[email protected]
http://www.borasky-research.net

If I had 40 billion dollars for every software monopoly that sells an unwieldy and hazardously complex development environment and is run by an arrogant college dropout with delusions of grandeur who treats his employees like serfs while he is acclaimed as a man of compelling vision, I'd be a wealthy man.

#56 lore raw

Re: aio

In reply to Ingo Molnar

On Thu, Dec 20, 2001 at 01:20:55PM -0500, Robert Love <[email protected]> wrote:

| On Thu, 2001-12-20 at 05:18, Ingo Molnar wrote:
| > there are two possibilities i can think of:
| > 1) lets get Ben's patch in but do *not* export the syscalls, yet.
| 
| This is an excellent way to give aio the testing and exposure Linus
| wants without getting into the commitment / syscall mess.
| Stick aio in the kernel, play with it via Tux, etc.  The really
| interested can add temporary syscalls.  aio (which I like, btw) will get
| testing and in time, once proven, we can add the syscalls.
| Comments?

Only that it would be hard for user space people to try it - does Ben's patch (with hypothetical syscalls) present the POSIX async interfaces out of the box? If not, testing with in-kernel things is sufficient. But if it does then it becomes more reasonable to transiently define some syscall numbers (high up, in some defined as "testing and like shifting sands" range) so user space can test the interface.

Thought: is there a meta-syscall in the kernel API for calling other syscalls? You could have such a beast taking negative numbers for experimental calls...

Signature · 3 lines
Cameron Simpson, DoD#743        [email protected]    http://www.zip.com.au/~cs/

Sometimes the only solution is to find a new problem.
#57 lore raw

Re: aio

In reply to Ingo Molnar
Quoted 3 lines · Cameron Simpson

Only that it would be hard for user space people to try it - does Ben's patch (with hypothetical syscalls) present the POSIX async interfaces out of the box?

No. POSIX aio does not have any concept of a completion queue. Completion in POSIX aio comes via a thread callback, signal delivery or polling, all of which are horrendously inefficient.

Quoted 4 lines

If not, testing with in-kernel things is sufficient. But
if it does then it becomes more reasonable to transiently define some syscall numbers (high up, in some defined as "testing and like shifting sands" range) so user space can test the interface.

Maybe. The unfortunate aspect to this is that you can't tell if a number matches the name you expect it to be, and invariably people end up running the wrong code on the wrong kernel. Or vendors start shipping patches to enable these new syscalls....

Quoted 3 lines

Thought: is there a meta-syscall in the kernel API for calling other syscalls? You could have such a beast taking negative numbers for experimental calls...

I'm working on something. Stay tuned.

-ben

Signature · 1 line
Fish.
#30 lore raw

Offtopic Java/C# [Re: aio]

In reply to Bill Huey
Quoted 2 lines · David S. Miller

That's a circular question, because C# exists due to Sun's mistakes with handling Java. So my answer is "both".

Well, they really serve different purposes and can't be compare. One is an unified object/class model for DCOM with more static typing stuff (boxing, parametric types), which the other is more CLish with class reflection built closely into the language runtime, threading and other self contained things within that system.

bill

#70 lore raw

Re: aio

In reply to Bill Huey
Quoted 8 lines · Dan Kegel
Quoted 6 lines · David S. Miller

If AIO was so relevant+sexy we'd be having threads of discussion about the AIO implementation instead of threads about how relevant it is or is not for the general populace. Wouldn't you concur? :-)

The people doing Java server applets are such a small fraction of the Linux user community.

Quoted 7 lines

reason AIO is important is to make it easier to port code from NT.

but I firmly believe that some form of AIO is vital.

- Dan

From the aio-0.3.1/README

section Current State

IPv4 TCP and UDP (rx only) sockets.

It is simply too early for sexy discussions. For me, the most appealing part of AIO is the socket handling. It seems a little bit broken in the current glibc emulation/implementation.
Recv and send operations are ordered when used on the same socket handle. Thus a recv must be finished before a subsequent send will happen. Good idea for files, bad for sockets.

SGI's implementation kaio, which works perfect for me, is widespread ignored and sufferes from the unreserved syscall problem like Ben's aio. I am sure there is a reason for ignoring SGI-kaio, i just do not remember.

With the current state of the different implementations it is difficult to have sex about or use them.
But i would really like tooooooooooooooooooo.

Gerold

-
The one-sig-perfd patch did not get much attention either. No one seems to use sockets these days.

#71 lore raw

Re: aio

Quoted 6 lines · Gerold Jury

It is simply too early for sexy discussions. For me, the most appealing part of AIO is the socket handling. It seems a little bit broken in the current glibc emulation/implementation. Recv and send operations are ordered when used on the same socket handle. Thus a recv must be finished before a subsequent send will happen. Good idea for files, bad for sockets.

is this a fundamental limitation expressed in the interface, or just an implementational limitation? On sockets this is indeed a big problem, HTTP pipelining wants completely separate receive/send queues.

Ingo

#72 lore raw

Re: aio

Quoted 8 lines · Ingo Molnar
Quoted 6 lines · Gerold Jury

It is simply too early for sexy discussions. For me, the most appealing part of AIO is the socket handling. It seems a little bit broken in the current glibc emulation/implementation. Recv and send operations are ordered when used on the same socket handle. Thus a recv must be finished before a subsequent send will happen. Good idea for files, bad for sockets.

Quoted 6 lines

is this a fundamental limitation expressed in the interface, or just an implementational limitation? On sockets this is indeed a big problem, HTTP pipelining wants completely separate receive/send queues.

Ingo

That is a very good question.

The Single UNIX ® Specification, Version 2 has the following to say.

If _POSIX_SYNCHRONIZED_IO is defined and synchronised I/O is enabled on the file associated with aiocbp->aio_fildes, the behaviour of this function is according to the definitions of synchronised I/O data integrity completion and synchronised I/O file integrity completion.

Maybe a was a little bit too fast in blaming glibc. I will go and look for more documentation about disabling synchronised I/O on a socket.

Dup()licating the socket handle is an easy workaround, but now i am convinced, a little bit man page digging will be lots of fun.

I hope the efforts of Benjamin LaHaise receive more attention and as soon as i know more about disabling synchronised I/O on sockets i will send an other email.

Gerold

--
I love AIO

#24 lore raw

Re: aio

In reply to Benjamin LaHaise
Quoted 2 lines

Who cares about Java? What about high performance LDAP servers or tux-like userspace performance? How about faster select and poll? An X server that

select/poll is a win - and Java recently discovered poll/select semantics 8)

#25 lore raw

Re: aio

In reply to Benjamin LaHaise
Quoted 1 line · Alan Cox

select/poll is a win - and Java recently discovered poll/select semantics 8)

Anything is a win over Java's threading model.

-ben

Signature · 1 line
Killer Attack Fish.
#42 lore raw

Re: aio

In reply to David S. Miller
Quoted 3 lines

Precisely, in fact. Anyone who can say that Java is going to be relevant in a few years time, with a straight face, is only kidding themselves.

Oh it'll be very relevant. Its leaking into all sorts of embedded uses, from Digital TV to smartcards. Its still useless for serious high end work an likely to stay so.

Quoted 2 lines

Java is not something to justify a new kernel feature, that is for certain.

There we agree. Things like the current asynch/thread mess in java are partly poor design of language and greatly stupid design of JVM.

#26 lore raw

Re: aio

In reply to Benjamin LaHaise
Quoted 6 lines · Benjamin LaHaise
Quoted 1 line · Alan Cox

select/poll is a win - and Java recently discovered poll/select semantics 8)

Anything is a win over Java's threading model.

-ben

Yeah, it's just another abstraction layer that lives on top of native threading model, so you don't have to worry about stuff like spinlock contention since it's been pushed down into the native threading implementation. It doesn't really add a tremendous amount of overhead given how delegates all of that to the native OS threading model.

Also, t would be nice to have some regular way of doing read-write lock without having to implement it in Java language itself, but it's not too critical since folks don't really push or use the JVM in that way just yet. It's certainly important in certain high contention systems in the kernel.

bill

#43 lore raw

Re: aio

In reply to David S. Miller
Quoted 8 lines · Alan Cox
Quoted 3 lines

Precisely, in fact. Anyone who can say that Java is going to be relevant in a few years time, with a straight face, is only kidding themselves.

Oh it'll be very relevant. Its leaking into all sorts of embedded uses, from Digital TV to smartcards. Its still useless for serious high end work an likely to stay so.

Quoted 5 lines
Quoted 2 lines

Java is not something to justify a new kernel feature, that is for certain.

There we agree. Things like the current asynch/thread mess in java are partly poor design of language and greatly stupid design of JVM.

It's not the fault of the JVM runtime nor the the language per se since both are excellent. The blame should instead be placed on the political process within Sun, which has created a lag in getting a decent IO event model/system available in the form of an API.

This newer system is suppose to be able to scale to tens of thousands of FDs and be able to handle heavy duty server side stuff in a more graceful manner. It's a reasonable system from what I saw, but the implementation of it is highly OS dependent and will be subject to those environmental constraints. Couple this and the HotSpot compiler (supposeablly competitive with gcc's -O3 from benchmarks) and it should be high useable for a broad range of of server side work when intelligently engineered.

bill

#44 lore raw

Re: aio

In reply to David S. Miller
Quoted 8 lines · Bill Huey
Quoted 2 lines

There we agree. Things like the current asynch/thread mess in java are partly poor design of language and greatly stupid design of JVM.

It's not the fault of the JVM runtime nor the the language per se since both are excellent. The blame should instead be placed on the political process within Sun, which has created a lag in getting a decent IO event model/system available in the form of an API.

Quoted 7 lines

This newer system is suppose to be able to scale to tens of thousands of FDs and be able to handle heavy duty server side stuff in a more graceful manner. It's a reasonable system from what I saw, but the implementation of it is highly OS dependent and will be subject to those environmental constraints. Couple this and the HotSpot compiler (supposeablly competitive with gcc's -O3 from benchmarks) and it should be high useable for a broad range of of server side work when intelligently engineered.

I served on JSR-51, the expert group that helped design the new I/O model. (The design was Sun's, but we had quite a bit of input.) For network I/O, there's a Selector object which is essentially a nice OO wrapper around the /dev/poll or kqueue/kevent abstraction. Selector does have a distinctly Unixy feel to it, but it can probably be implemented well on top of any reasonable OS; I'm quite sure it can be expressed fairly well in terms of Windows NT's asych I/O or Linux's rt signal stuff.

(I suspect the initial Linux implementations will just use poll(), but that's something the Blackdown team can fix. And heck, it ought to be easy to implement it on top of all the nifty poll replacements and choose between them at jvm startup time without any noticable overhead.)

- Dan

p.s. Davide, I didn't forget /dev/epoll, I just haven't had time to post Poller_devepoll yet!

#45 lore raw

Re: aio

Quoted 2 lines · Dan Kegel

p.s. Davide, I didn't forget /dev/epoll, I just haven't had time to post Poller_devepoll yet!

Yep, i just started feeling angry about this :)

- Davide

#73 lore raw

Re: aio

In reply to Ingo Molnar
Quoted 7 lines · Ingo Molnar
Quoted 5 lines

is this a fundamental limitation expressed in the interface, or just an implementational limitation? On sockets this is indeed a big problem, HTTP pipelining wants completely separate receive/send queues.

Ingo

I got the _POSIX_SYNCHRONIZED_IO completely wrong.
It has nothing to do with the ordering of the aio read/write requests.

Aio_read and aio_write work with absolute positions inside the file size. The order of requests is unspecified in SUSV V2.
SUSV V2 does neither prevent nor force the desired behaviour.

It seems like it is up to the implementation how to deal with the request order.
As mentioned earlier, SGI-kaio does the right thing with the same interface.

I want to add, that a combination of sigwaitinfo / sigtimedwait and aio is a very efficient way to deal with sockets. The accept may be handled with real time signals as well by using fcntl F_SETSIG and F_SETFL FASYNC.

Gerold

#46 lore raw

Java and Flam^H^H^H^H AIO (was: aio)

In reply to David S. Miller
Quoted 12 lines · Bill Huey
Quoted 11 lines · Alan Cox
Quoted 3 lines

Precisely, in fact. Anyone who can say that Java is going to be relevant in a few years time, with a straight face, is only kidding themselves.

Oh it'll be very relevant. Its leaking into all sorts of embedded uses, from Digital TV to smartcards. Its still useless for serious high end work an likely to stay so.

Quoted 2 lines

Java is not something to justify a new kernel feature, that is for certain.

Quoted 7 lines
Quoted 2 lines

There we agree. Things like the current asynch/thread mess in java are partly poor design of language and greatly stupid design of JVM.

It's not the fault of the JVM runtime nor the the language per se since both are excellent. The blame should instead be placed on the political process within Sun, which has created a lag in getting a decent IO event model/system available in the form of an API.

Hey wait, it can't be so. Sun apparently uses a boot camp system to guarantee that every project finishes on time, every time.

* daniel ducks and runs

--
Daniel

#16 lore raw

Re: aio

In reply to David S. Miller
Quoted 28 lines · Bill Huey
Quoted 7 lines · David S. Miller

Firstly, you say this as if server java applets do not function at all or with acceptable performance today. That is not true for the vast majority of cases.

If java server applet performance in all cases is dependent upon AIO (it is not), that would be pretty sad. But it wouldn't be the first

Java is pretty incomplete in this area, which should be addressed to a great degree in the new NIO API.

The core JVM isn't dependent on this stuff per se for performance, but it is critical to server side programs that have to deal with highly scalable IO systems, largely number of FDs, that go beyond the current expressiveness of select()/poll().

This is all standard fare in *any* kind of high performance networking application where some kind of high performance kernel/userspace event delivery system is needed, kqueue() principally.

Quoted 5 lines

time I've heard crap like that. There is propaganda out there telling people that 64-bit address spaces are needed for good java performance. Guess where that came from? (hint: they invented java and are in the buisness of selling 64-bit RISC processors)

Quoted 5 lines

What ? oh god. HotSpot is a pretty amazing compiler and it performs well. Swing does well now, but the lingering issue in Java is the shear size of it and possibly GC issues. It pretty clear that it's going to get larger, which is fine since memory is cheap.

I remind you: ORACLE 9i is requiring half a gig as a minimum just due to the use of the CRAPPY PIECE OF SHIT written in the Java, called, you guess it: Just the bloody damn Installer. Java is really condemned just due to the fact that both terms: speed and memmory usage are both allways only *relative* to other systems.

And yes GC's have only one problem - they try to give a general solution for problems which can be easly prooven to be mathmematically insolvable. The resulting undeterministic behaviour of applications is indeed the thing which is hurting most.