Syslets, "Threadlets", generic AIO support, v3
Subsystems (6)
Quoted 4 lines · Theodore Tso
I think what you are not hearing, and what everyone else is saying (INCLUDING Linus), is that for most programmers, state machines are much, much harder to program, understand, and debug compared to multi-threaded code. [...]
btw., another crutial thing that i think Evgeniy is missing is that threadlets /enable/ event loops to be used in practice! Right now the epoll/kevent programming model requires a total 100% avoidance of all context-switching in the 'main' event handler context while handling a request. If just 1% of all requests happen to block it might cause a /complete/ breakdown of an event loop's performance - it can easily cause a 10x drop in performance or worse!
So context-switching has to be avoided in 100% of the code that runs while handling requests, file descriptors have to be set to nonblocking (causing extra system calls), and all the syscalls that might return incomplete with either -EINVAL or with a short read/write have to be converted into a state machine. (or in the alternative, user-space threading has to be used, which opens up another hornet's nest)
/That/ is the main inhibiting factor of the measured use of event loops within Linux! It has zero integration capabilities with 'usual' coding techniques - driving the costs of its application up in the sky, and pushing event based servers into niches.
With threadlets the picture changes dramatically: all we have to concentrate on to get the performance of "100% event based servers" is to handle 'most' rescheduling events in the event loop. A 10-20% context switching ratio does not hurt at all. (it causes ~1% of throughput loss.)
Furthermore, even if a particular configuration or module of the server (say Apache) happens to trigger a high rate of scheduling, the performance breakdown model of threadlets is /vastly/ superior to event based servers. The measurements so far have shown that the absolute worst-case threading server performance is at around 60% of that of non-context-switching servers - and even that level is reached gradually, leaving time for action for the server owner. While with fully event based servers there are mostly only two modes of
performance: 100% performance and near-0% performance: total breakdown.
Ingo
Quoted 6 lines · Ingo Molnar
based servers. The measurements so far have shown that the absolute worst-case threading server performance is at around 60% of that of non-context-switching servers - and even that level is reached gradually, leaving time for action for the server owner. While with fully event based servers there are mostly only two modes of
performance: 100% performance and near-0% performance: total breakdown.
Let's live in piece! :)
I always agreed that they should be used together - event-based rings of IO requests, if request happens to block (which should be avaided as much as possible), then it continues on behalf of sleeping thread.
Quoted 1 line
Ingo
Signature · 1 line
Evgeniy Polyakov
Quoted 13 lines · Ingo Molnar
Quoted 2 lines · Theodore Tso
I think what you are not hearing, and what everyone else is saying (INCLUDING Linus), is that for most programmers, state machines are much, much harder to program, understand, and debug compared to multi-threaded code. [...]
btw., another crutial thing that i think Evgeniy is missing is that threadlets /enable/ event loops to be used in practice! Right now the epoll/kevent programming model requires a total 100% avoidance of all context-switching in the 'main' event handler context while handling a request. If just 1% of all requests happen to block it might cause a /complete/ breakdown of an event loop's performance - it can easily cause a 10x drop in performance or worse!
So context-switching has to be avoided in 100% of the code that runs while handling requests, file descriptors have to be set to nonblocking (causing extra system calls), and all the syscalls that might return incomplete with either
-EINVALor with a short read/write have to be converted into a state machine. (or in the alternative, user-space threading has to be used, which opens up another hornet's nest)/That/ is the main inhibiting factor of the measured use of event loops within Linux! It has zero integration capabilities with 'usual' coding techniques - driving the costs of its application up in the sky, and pushing event based servers into niches.
Having written such a niche event based server, I can 100% confirm what Ingo is saying here. We had a single process drive I/O to the kernel through an event model (based on kernel aio extended with IO_CMD_POLL), and user level threads managed by a custom scheduler that managed I/O, timeouts, and thread scheduling.
We once considered dropping from a user-level thread model to a state machine model, but the effort was astronomical and we wouldn't see the rewards until it was all done, so naturally we didn't do it.
Quoted 4 lines
With threadlets the picture changes dramatically: all we have to concentrate on to get the performance of "100% event based servers" is to handle 'most' rescheduling events in the event loop. A 10-20% context switching ratio does not hurt at all. (it causes ~1% of throughput loss.)
Furthermore, even if a particular configuration or module of the server (say Apache) happens to trigger a high rate of scheduling, the performance breakdown model of threadlets is /vastly/ superior to event based servers. The measurements so far have shown that the absolute worst-case threading server performance is at around 60% of that of non-context-switching servers - and even that level is reached gradually, leaving time for action for the server owner. While with fully event based servers there are mostly only two modes of performance: 100% performance and near-0% performance: total breakdown.
Yes. Threadlets as the default aio solution (easy to use, acceptable performance even in worst cases), with specialized solutions where applicable (epoll for networking, aio for O_DIRECT disk) look like a good mix of performance and sanity.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 10 lines · Theodore Tso
I think what you are not hearing, and what everyone else is saying (INCLUDING Linus), is that for most programmers, state machines are much, much harder to program, understand, and debug compared to multi-threaded code. You may disagree (were you a MacOS 9 programmer in another life?), and it may not even be true for you if you happen to be one of those folks more at home with Scheme continuations, for example. But it is true that for most kernel programmers, threaded programming is much easier to understand, and we need to engineer the kernel for what will be maintainable for the majority of the kernel development community.
State machines are much harder to write without going through a real on-paper design phase first. But multi-threaded code is much harder for a team of average working coders to write correctly, judging from the numerous train wrecks that I've been called in to salvage over the last ten years or so.
The typical 50-250KLoC multi-threaded C/C++/Java application, even if it's been shipping to customers for several years, is littered with locking constructs yet routinely corrupts shared data structures. Change the number of threads in a pool, adjust the thread priorities, or move a couple of lines of code around, and you're very likely to get an unexplained deadlock. God help you if you try to use a debugger on it -- hundreds of latent race conditions will crop up that didn't happen to trigger before because the thread didn't get preempted there.
The only programming languages that I have seen widely used in US industry (so Lisps and self-consciously functional languages are out) in which mere mortals write passable multi-threaded applications are Visual Basic and Python. That's partly because programmers in these languages are not in the habit of throwing pointers around; but if that were all there was to it, Java programmers would be a lot more successful than they are at actually writing threaded programs rather than nibbling cautiously around the edges with EJB. It also helps a lot that strings are immutable; but again, Java shares this property. No, the big difference is that VB and Python dicts and arrays are thread-safed by the language runtime, and Java collections are not. So while there may be all sorts of pointless and dangerous mis-locking, it's "protecting" somewhat higher-level data structures.
What does this have to do with the kernel? Well, if you're going to create Yet Another Micro^WLightweight-Threading Construct for AIO, it would be mighty nice not to be slinging bare pointers around until the IO is actually complete and the kernel isn't going to be touching the data buffer any more. It would also be mighty nice to have a thread-safe "request pool" data structure on which actions like bulk cancellation and iteration over a subset can operate. (The iterator returned by, say, a three-sided query on a RCU priority queue may contain _stale_ information, but never _inconsistent_ information.)
I recognize that this is more object-oriented snake oil than kernel programmers usually tolerate, but it would really help AIO-threaded programs suck less. It is also very much in the Unix tradition -- what are file descriptors and fd_sets if not object-oriented design? And if following the socket model was good enough for epoll and netlink, why not for threadlet pools?
In the best of all possible worlds, AIO would look just like the good old socket-bind-listen-accept model, except that I/O is transacted on the "listen" socket as long as it can be serviced from cache, and accept() only gets a new connection when a delayed I/O arrives. The object hiding inside the fd returned by socket() would be the "threadlet pool", and the object hiding inside each fd returned by accept() would be a threadlet. Only this time you do it right and make errno(fd) be a vsyscall that returns a threadlet-local error state, and you assign reasonable semantics to operations on an fd that has already encountered an exception. Much like IEEE 754, actually.
Anyway, like I said, good threaded code is quite rare. On the other hand, I have seen plenty of reasonably successful event-loop programming in C and C++, mostly in MacOS and Windows and PalmOS GUIs where the OS deals with event queues and event handler registration. It's not the world's most CPU-efficient strategy because of all those function pointers and virtual methods, but those costs are dwarfed by the GUI bounding boxes and repaints and things anyway. More to the point, writing an event-loop framework for other people to use involves extensive APIs that are stable in the tiniest details and extensively documented. Not, perhaps, Plan A for the Linux kernel community. :-)
Happily, a largely event-driven userspace framework can easily be stacked on top of a threaded kernel -- as long as they're the right kind of threads. The right kind of threads do not proliferate malloc arenas by allowing preemption in mid-malloc. (They may need to malloc(), and that may be a preemption point relative to _real_ threads, but you shouldn't switch or cancel threadlets there.) The right kind of threads do not leak locking primitives when cancelled, because they don't have to take a lock in order to update the right kind of data structure. The right kind of threads can use floating point safely as long as they don't expect FPU state to be preserved across a syscall.
The right kind of threads, in short, work like coroutines or MacOS/PalmOS "event handlers", with the added convenience of being able to write them as if they were normal sequential code, with normal access to a persistent stack frame and to process globals. And if you do them right, they're cheap to migrate, easy and harmless to throttle and cancel in bulk, and easy to punt out to an "I/O coprocessor" in the future. The key is to move data into and out of the "I/O registers" at well-defined points and not to break the encapsulation in between. Delay the extraction of results from the "I/O registers" as long as possible, and the hypothetical AIO coprocessor can go chew on them in parallel with the main "integer" code flow, which only stalls when it can't go any further without the I/O result.
If you've got some time to kill, you can even analyze an existing, well-documented flavor of I/O strings (I like James Antill's Vstr library myself) and define a set of "AIO opcodes" that manipulate AIO fds and AIO strings as primitive types, just like the FPU manipulates floating-point numbers as primitive types. Pick a CPU architecture with a good range of unused trap opcodes (PPC, for instance) and actually move the I/O strings into kernel space, mapping the AIO operations and the I/O string API onto the free trap opcodes (really no different from syscalls, except it's easier to inspect the assembly that the compiler produces and see what's going on). For extra credit, implement most of the AIO opcodes in Verilog, bolt them onto the PPC core inside a Virtex4 FX FPGA, refine the semantics to permit efficient pipelining, and collect your Turing award. :-)
Cheers,
- Michael
Quoted 6 lines · Michael K. Edwards
State machines are much harder to write without going through a real on-paper design phase first. But multi-threaded code is much harder for a team of average working coders to write correctly, judging from the numerous train wrecks that I've been called in to salvage over the last ten years or so.
130 lines skipped...
I have only one question - wasn't it too lazy to write all that? :)
Quoted 2 lines
Cheers,
- Michael
Signature · 1 line
Evgeniy Polyakov
Quoted 1 line · Evgeniy Polyakov
130 lines skipped...
Yeah, I edited it down a lot before sending it. :-)
Quoted 1 line
I have only one question - wasn't it too lazy to write all that? :)
I'm pretty lazy all right. But occasionally an interesting problem (and revamping AIO is very interesting) makes me think, and what little thinking I do is always accompanied by writing. Once I've thought something through to the point that I think I understand the problem, I've even been known to attempt a solution. Not always, though; more often, I find a new interesting problem, or else I am forcibly reminded that I should be spending my little store of insight on revenue-producing activity.
In this instance, there didn't seem to be any harm in sending my thoughts to LKML as I wrote them, on the off chance that Ingo or Davide would get some value out of them in this design cycle (which any code I eventually get around to producing will miss). So far, I've gotten some rather dismissive pushback from Ingo and Alan (who seem to have no interest outside x86 and less understanding than I would have thought of what real userspace code looks like), a "why preach to people who know more than you do" from Davide, a brief aside on the dominance of x86 from Oleg, and one off-list "keep up the good work". Not a very rich harvest from (IMHO) pretty good seeds.
In short, so far the "Linux kernel community" is upholding its reputation for insularity, arrogance, coding without prior design, lack of interest in userspace problems, and inability to learn from the mistakes of others. (None of these characterizations depends on there being any real insight in anything I have written.) Linus himself has a very different reputation -- plenty of arrogance all right, but genuine brilliance and hard work, and sincere (if cranky) efforts to explain the "theory of operations" underlying central design choices. So far he hasn't commented directly on anything I have had to say; it will be interesting to see whether he tells me to stop annoying the pros and to go away until I have some code to contribute.
Happy hacking,
- Michael
P. S. I do think "threadlets" are brilliant, though, and reading Ingo's patches gave me a much better idea of what would be involved in prototyping Asynchronously Executed I/O Unit opcodes.
Discussing LKML community [OT from the Re: [patch 00/13] Syslets, "Threadlets", generic AIO support, v3]
Quoted 4 lines
From: "Michael K. Edwards" <[email protected]>
Newsgroups: gmane.linux.kernel
Subject: Re: [patch 00/13] Syslets, "Threadlets", generic AIO support, v3
Date: Wed, 28 Feb 2007 09:01:07 -0800
Michael,
[]
Quoted 8 lines
In this instance, there didn't seem to be any harm in sending my thoughts to LKML as I wrote them, on the off chance that Ingo or Davide would get some value out of them in this design cycle (which any code I eventually get around to producing will miss). So far, I've gotten some rather dismissive pushback from Ingo and Alan (who seem to have no interest outside x86 and less understanding than I would have thought of what real userspace code looks like), a "why preach to people who know more than you do" from Davide,
this may be sad, unless you've spent time and effort to make a Patch, i.e. read source, understand why it's written so, why it's being used now that way, and why it has to be updated on new cycle of kernel development.
Quoted 1 line
a brief aside on the dominance of x86 from Oleg,
I didn't have a chance, and probably i will not have one, to communicate with people like you to learn from your wisdom personally. That's why i've replied to your, after you've mentioned transputers. And i've got rather different opinion, than i expected. That shows my test-tube being, little experience etc. As discussion was about CPUs, it was technical, thus on-topic for LKML.
Quoted 2 lines
and one off-list "keep up the good work". Not a very rich harvest from (IMHO) pretty good seeds.
Offlist message was my share of view about things, that were offtopic, and clarifying about lkml thing, and it wasn't on-topic for LKML.
I'm pretty sure, that there libraries of books, written on every single bit of things Linux currently *implements* in asm/C.
(1) Thus, `return -ENOPATCH', man, regardless what you are saying in lkml. That's why prominent people, you've joined me with (:, replied in go-to-kernelnewbie style.
Quoted 5 lines
In short, so far the "Linux kernel community" is upholding its reputation for insularity, arrogance, coding without prior design, lack of interest in userspace problems, and inability to learn from the mistakes of others. (None of these characterizations depends on there being any real insight in anything I have written.)
You, as a person, who have right to be personally wrong, may think that way. But do not forget, as i've wrote you offlist and in (1), this is development community, sometimes development of development one, etc; educated, enthusiastic, wise, Open Source, poor on time (and money :).
Quoted 2 lines
Happy hacking,
- Michael
And you too. LKML *can* (sometimes may) show how useful this hacking is.
Quoted 3 lines
P. S. I do think "threadlets" are brilliant, though, and reading Ingo's patches gave me a much better idea of what would be involved in prototyping Asynchronously Executed I/O Unit opcodes.
You are discussing on-topic thing in the P.S. And this is IMHO wrong approach.
Also, note, that i've changed subject, stripped cc list, please note, that i can be young and naive boy barking up the wrong tree.
Kind regards.
--
-o--=O`C /. .\
#oo'L O o
<___=E M ^-- (Wuuf)
Quoted 5 lines · Michael K. Edwards
State machines are much harder to write without going through a real on-paper design phase first. But multi-threaded code is much harder for a team of average working coders to write correctly, judging from the numerous train wrecks that I've been called in to salvage over the last ten years or so.
I have to agree; state machines are harder to design and read, but multithreaded programs are harder to write and debug _correctly_.
Another way of putting it is that the threadlet approach is easier to do, but harder to do _right_.
Quoted 12 lines · Ingo Molnar
Quoted 8 lines · Ulrich Drepper
Quoted 2 lines · Ingo Molnar
in terms of AIO, the best queueing model is i think what the kernel uses
internally: freely ordered, with barrier support.Speaking of AIO, how do you imagine
lio_listiois implemented? If there is no asynchronous syscall it would mean creating a threadlet for each request but this means either waiting or creating several/many threads.
Quoted 9 lines
my current thinking is that special-purpose (non-programmable, static) APIs like aio_*() and lio_*(), where every last cycle of performance matters, should be implemented using syslets - even if it is quite tricky to write syslets (which they no doubt are - just compare the size of
syslet-test.ctothreadlet-test.c). So i'd move syslets into the same category as raw syscalls: pieces of the raw infrastructure between the kernel and glibc, not an exposed API to apps. [and even if we keep them in that category they still need quite a bit of API work, to clean up the 32/64-bit issues, etc.]
Now that chains of syscalls can be way more easily handled with clets^wthreadlets, why would we need the whole syslets crud inside?
Why can't aio_* be implemented with *simple* (or parallel/unrelated) syscall submit w/out the burden of a complex, limiting and heavy API (I won't list all the points against syslets, because I already did it enough times)? The compat layer only is so bad to not be even funny. Look at the code. Only removing the syslets crud would prolly cut 40% of it. And we did not even touch the compat code yet.
- Davide
Quoted 10 lines · Davide Libenzi
Quoted 9 lines
my current thinking is that special-purpose (non-programmable, static) APIs like aio_*() and lio_*(), where every last cycle of performance matters, should be implemented using syslets - even if it is quite tricky to write syslets (which they no doubt are - just compare the size of
syslet-test.ctothreadlet-test.c). So i'd move syslets into the same category as raw syscalls: pieces of the raw infrastructure between the kernel and glibc, not an exposed API to apps. [and even if we keep them in that category they still need quite a bit of API work, to clean up the 32/64-bit issues, etc.]
Quoted 2 lines
Why can't aio_* be implemented with *simple* (or parallel/unrelated) syscall submit w/out the burden of a complex, limiting and heavy API
there are so many variants of what people think 'asynchronous IO' should look like - i'd not like to limit them. I agree that once a particular syslet script becomes really popular, it might (and should) in fact be pushed into a separate system call.
But i also agree that a one-shot-syscall sys_async() syscall could be done too - for those uses where only a single system call is needed and where the fetching of a single uatom would be small but nevertheless unnecessary overhead. A one-shot async syscall needs to get /8/ parameters (the syscall nr is the seventh parameter and the return code of the nested syscall is the eighth). So at least two parameters will have to be passed in indirectly and validated, and 32/64-bit compat conversions added, etc. anyway!
The copy_uatom() assembly code i did is really fast so i doubt there would be much measurable performance difference between the two solutions. Plus, putting the uatom into user memory allows the caching of uatoms - further dilluting the advantage of passing in the values per register. The whole difference should be on the order of 10 cycles, so this really isnt a high prio item in my view.
Quoted 2 lines
Now that chains of syscalls can be way more easily handled with clets^wthreadlets, why would we need the whole syslets crud inside?
no, threadlets dont really solve the basic issue of people wanting to 'combine' syscalls, avoid the syscall entry overhead (even if that is small), and the desire to rely on kthread->kthread context switching which is even faster than uthread->uthread context-switching, etc. Furthermore, syslets dont really cause any new problem. They are almost totally orthogonal, isolated, and cause no wide infrastructure needs.
as long as syslets remain a syscall-level API, for the measured use of the likes of glibc and libaio (and not exposed in a programmable manner to user-space), i see no big problem with them at all. They can also be used without them having any classic pthread user-state (without linking to libpthread). Think of it like the raw use of clone(): possible and useful in some cases, but not something that a typical application would do. This is a 'raw syscall plugins' thing, to be used by those user-space entities that use raw syscalls: infrastructure libraries. Raw syscalls themselves are tied to the platform, are not easily used in some cases, thus almost no application uses them directly, but uses the generic functions glibc exposes.
in the long run, sys_syslet_exec(), were it not to establish itself as a widely used interface, could be implemented purely from user-space too (say from the VDSO, at much worse performance, but the kernel would stay backwards compatible with the syscall), so there's almost no risk here. You dont like it => dont use it. Meanwhile, i'll happily take any suggestion to make the syslet API more digestable.
Ingo
Quoted 11 lines · Ingo Molnar
Quoted 2 lines · Davide Libenzi
Why can't aio_* be implemented with *simple* (or parallel/unrelated) syscall submit w/out the burden of a complex, limiting and heavy API
there are so many variants of what people think 'asynchronous IO' should look like - i'd not like to limit them. I agree that once a particular syslet script becomes really popular, it might (and should) in fact be pushed into a separate system call.
Quoted 8 lines
But i also agree that a one-shot-syscall
sys_async()syscall could be done too - for those uses where only a single system call is needed and where the fetching of a single uatom would be small but nevertheless unnecessary overhead. A one-shot async syscall needs to get /8/ parameters (the syscall nr is the seventh parameter and the return code of the nested syscall is the eighth). So at least two parameters will have to be passed in indirectly and validated, and 32/64-bit compat conversions added, etc. anyway!
At this point, given how threadlets can be easily/effectively dispatched from userspace, I'd argue the presence of either single/parallel or syslet submission altogether. Threadlets allows you to code chains *way* more naturally than syslets, and since they basically are like functions calls in the fast path, they can be used even for single/parallel submissions. No compat code required (ok, besides the trivial async_wait). My point is, the syslet infrastructure is expensive for the kernel in terms of compat, and extra code added to handle the cond/jumps/etc. Is also non-trivial to use from userspace. Are those big performance advantages there to justify its existence? I doubt that the price of a sysenter is a lot bigger than a atom decoding, but I'm looking forward in being proven wrong by real life performance numbers ;)
- Davide
Quoted 6 lines · Davide Libenzi
At this point, given how threadlets can be easily/effectively dispatched from userspace, I'd argue the presence of either single/parallel or syslet submission altogether. Threadlets allows you to code chains *way* more naturally than syslets, and since they basically are like functions calls in the fast path, they can be used even for single/parallel submissions.
Well, I agree, except for one thing:
- user space execution is *inherently* more expensive.
Why? Stack. Stack. Stack.
If you support threadlets with user space code, it means that you need a separate user-space stack for each threadlet. That's a potentially *big* cost to bear, both from a setup standpoint and from simply a memory allocation standpoint.
Quite frankly, I think threadlets are a great idea, but I think the lack of user-level footprint is *also* a great idea, and you should support both.
In short - the only thing I *don't* think is a great idea are those linked lists of atoms. I still think it's a pretty horrible interface, and I still don't think it really buys us very much. The only way it would buy us a lot is to change the linked lists dynamically (ie add new events at the end while old events are still executing), but quite frankly, that just makes the whole interface *even*worse* and just makes me have debugging nightmares (I'm also not even convinced it really would help
us: we might avoid some costs of adding new events, but it would only avoid them for serial execution, and if the whole point of this is to execute things in parallel, that's a stupid thing to do).
So I would repeat my call for getting rid of the atoms, and instead just do a "single submission" at a time. Do the linking by running a threadlet that has user space code (and the stack overhead), which is MUCH more flexible. And do nonlinked single system calls without *either* atoms *or* a user-space stack footprint.
Please?
What am I missing?
Linus
Quoted 3 lines · Linus Torvalds
[...] The only way it would buy us a lot is to change the linked lists dynamically (ie add new events at the end while old events are still executing), [...]
that's quite close to what Jens' FIO plugin for syslets (engines/syslet-rw.c) does currently: it builds lists of syslets as IO gets submitted, batches them up for some time and then sends them off. It is a natural next step to do this for in-flight syslets as well.
Ingo
Quoted 11 lines · Linus Torvalds
Quoted 6 lines · Davide Libenzi
At this point, given how threadlets can be easily/effectively dispatched from userspace, I'd argue the presence of either single/parallel or syslet submission altogether. Threadlets allows you to code chains *way* more naturally than syslets, and since they basically are like functions calls in the fast path, they can be used even for single/parallel submissions.
Well, I agree, except for one thing:
- user space execution is *inherently* more expensive.
Quoted 6 lines
Why? Stack. Stack. Stack.
If you support threadlets with user space code, it means that you need a separate user-space stack for each threadlet. That's a potentially *big* cost to bear, both from a setup standpoint and from simply a memory allocation standpoint.
Right, point taken.
Quoted 11 lines
In short - the only thing I *don't* think is a great idea are those linked lists of atoms. I still think it's a pretty horrible interface, and I still don't think it really buys us very much. The only way it would buy us a lot is to change the linked lists dynamically (ie add new events at the end while old events are still executing), but quite frankly, that just makes the whole interface *even*worse* and just makes me have debugging nightmares (I'm also not even convinced it really would help
us: we might avoid some costs of adding new events, but it would only avoid them for serial execution, and if the whole point of this is to execute things in parallel, that's a stupid thing to do).
Quoted 5 lines
So I would repeat my call for getting rid of the atoms, and instead just do a "single submission" at a time. Do the linking by running a threadlet that has user space code (and the stack overhead), which is MUCH more flexible. And do nonlinked single system calls without *either* atoms *or* a user-space stack footprint.
Here we very much agree. The way I'd like it:
struct async_syscall {
unsigned long nr_sysc;
unsigned long params[8];
long result;
};int async_exec(struct async_syscall *a, int n);
or:
int async_exec(struct async_syscall **a, int n);
At this point I'm ok even with the userspace ring buffer, returning back pointers to "struct async_syscall".
- Davide
Quoted 8 lines · Davide Libenzi
Here we very much agree. The way I'd like it:
struct async_syscall { unsigned long nr_sysc; unsigned long params[8]; long result; };
No, the "result" needs to go somewhere else. The caller may be totally uninterested in keeping the system call number or parameters around until the operation completes, but if you put them in the same structure with the result, you obviously cannot sanely get rid of them.
I also don't much like read-write interfaces (which the above would be:
the kernel would read most of the structure, and then write one member of the structure).
It's entirely possible, for example, that the operation we submit is some legacy "aio_read()", which has soem other structure layout than the new one (but one field will be the result code).
Linus
Quoted 15 lines · Linus Torvalds
Quoted 8 lines · Davide Libenzi
Here we very much agree. The way I'd like it:
struct async_syscall { unsigned long nr_sysc; unsigned long params[8]; long result; };No, the "result" needs to go somewhere else. The caller may be totally uninterested in keeping the system call number or parameters around until the operation completes, but if you put them in the same structure with the result, you obviously cannot sanely get rid of them.
Quoted 7 lines
I also don't much like read-write interfaces (which the above would be:
the kernel would read most of the structure, and then write one member of the structure).It's entirely possible, for example, that the operation we submit is some legacy "
aio_read()", which has soem other structure layout than the new one (but one field will be the result code).
Ok, makes sense. Something like this then?
struct async_syscall {
unsigned long nr_sysc;
unsigned long params[8];
long *result;
};And what would async_wait() return bak? Pointers to "struct async_syscall" or pointers to "result"?
- Davide
Quoted 8 lines · Davide Libenzi
struct async_syscall { unsigned long nr_sysc; unsigned long params[8]; long *result; };And what would
async_wait()return bak? Pointers to "structasync_syscall" or pointers to "result"?
Either one has downsides. Pointer to struct async_syscall requires that the caller keep the struct around. Pointer to result requires that the caller always reserve a location for the result.
Does the kernel care about the (possibly rare) case of callers that don't want to pay attention to result? If so, what about adding some kind of caller-specified handle to struct async_syscall, and having async_wait() return the handle? In the case where the caller does care about the result, the handle could just be the address of result.
Chris
Quoted 15 lines · Chris Friesen
Quoted 8 lines · Davide Libenzi
struct async_syscall { unsigned long nr_sysc; unsigned long params[8]; long *result; };And what would
async_wait()return bak? Pointers to "structasync_syscall" or pointers to "result"?Either one has downsides. Pointer to struct
async_syscallrequires that the caller keep the struct around. Pointer to result requires that the caller always reserve a location for the result.
Quoted 5 lines
Does the kernel care about the (possibly rare) case of callers that don't want to pay attention to result? If so, what about adding some kind of caller-specified handle to struct
async_syscall, and havingasync_wait()return the handle? In the case where the caller does care about the result, the handle could just be the address of result.
Something like this (with async_wait() returning asynid's)?
struct async_syscall {
long *result;
unsigned long asynid;
unsigned long nr_sysc;
unsigned long params[8];
};- Davide
Quoted 23 lines · Davide Libenzi
Quoted 15 lines · Chris Friesen
Quoted 8 lines · Davide Libenzi
struct async_syscall { unsigned long nr_sysc; unsigned long params[8]; long *result; };And what would
async_wait()return bak? Pointers to "structasync_syscall" or pointers to "result"?Either one has downsides. Pointer to struct
async_syscallrequires that the caller keep the struct around. Pointer to result requires that the caller always reserve a location for the result.Quoted 5 lines
Does the kernel care about the (possibly rare) case of callers that don't want to pay attention to result? If so, what about adding some kind of caller-specified handle to struct
async_syscall, and havingasync_wait()return the handle? In the case where the caller does care about the result, the handle could just be the address of result.
Quoted 8 lines
Something like this (with
async_wait()returning asynid's)?struct async_syscall { long *result; unsigned long asynid; unsigned long nr_sysc; unsigned long params[8]; };
Having result pointer as NULL might imply that result is not interested in and thus it can be discarded and event async syscall will not be returned through aync_wait().
More flexible is having request flags field in the structure.
Quoted 2 lines
- Davide
Signature · 1 line
Evgeniy Polyakov
Quoted 11 lines · Evgeniy Polyakov
Quoted 6 lines
struct async_syscall { long *result; unsigned long asynid; unsigned long nr_sysc; unsigned long params[8]; };Having result pointer as NULL might imply that result is not interested in and thus it can be discarded and event async syscall will not be returned through
aync_wait().
More flexible is having request flags field in the structure.
Ugh, that is alrasdy implemented in v5.
Signature · 1 line
Evgeniy Polyakov
Quoted 5 lines · Linus Torvalds
So I would repeat my call for getting rid of the atoms, and instead just do a "single submission" at a time. Do the linking by running a threadlet that has user space code (and the stack overhead), which is MUCH more flexible. And do nonlinked single system calls without *either* atoms *or* a user-space stack footprint.
I agree that threadlets are much more flexible - and they might in fact win in the long run due to that.
i'll add a one-shot syscall API in v6 and then we'll be able to see them side by side. (wanted to do that in v5 but it got delayed by x86_64 issues, x86_64's entry code is certainly ... tricky wrt. ptregs saving)
wrt. one-shot syscalls, the user-space stack footprint would still probably be there, because even async contexts that only do single-shot processing need to drop out of kernel mode to handle signals. We could probably hack the signal routing code to never deliver to such threads (but bounce it over to the head context, which is always available) but i think that would be a bit messy. (i dont exclude it though)
I think syslets might also act as a prototyping platform for new system calls. If any particular syslet atom string comes up more frequently (and we could even automate the profiling of that within the kernel), then it's a good candidate for a standalone syscall. Currently we dont have such information in any structured way: the connection between streams of syscalls done by applications is totally opaque to the kernel.
Also, i genuinely believe that to be competitive (performance-wise) with fully in-kernel queueing solutions, we need syslets - the syslet NULL overhead is 20 cycles (this includes copying, engine overhead, etc.), the syscall NULL overhead is 280-300 cycles. It could probably be made more capable by providing more special system calls like sys_upcall() to execute a user-space function. (that way a syslet could still execute user-space code without having to exit out of kernel mode too frequently) Or perhaps a sys_x86_bytecode() call, that would execute a pre-verified, kernel-stored sequence of simplified x86 bytecode, using the kernel stack.
My fear is that if we force all these things over to one-shot syscalls or threadlets then this will become another second-tier mechanism. By providing syslets we give the message: "sure, come on and play within the kernel if you want to, but it's not easy".
Ingo
Quoted 6 lines · Ingo Molnar
more capable by providing more special system calls like
sys_upcall()to execute a user-space function. (that way a syslet could still execute user-space code without having to exit out of kernel mode too frequently) Or perhaps asys_x86_bytecode()call, that would execute a pre-verified, kernel-stored sequence of simplified x86 bytecode, using the kernel stack.
Which means the userspace code would then run with kernel privilege level somehow (after security verifier, whatever). You remember I think it's a plain crazy idea...
I don't want to argue about syslets, threadlets, whatever async or syscall-merging mechanism here, I'm just focusing on the idea of yours of running userland code in kernel space somehow (I hoped you given up on it by now). Fixing the greatest syslets limitation is going to open a can of worms as far as security is concerned.
The fact that userland code must not run with kernel privilege level, is the reason why syslets aren't very useful (but again: focusing on the syslets vs async-syscalls isn't my interest).
Frankly I think this idea of running userland code with kernel privileges fits in the same category of porting linux to segmentation to avoid the cost of pagetables to gain some bit of performance despite losing in many other areas. Nobody in real life will want to make that trade, for such an incredibly small performance improvement.
For things that can be frequently combined, it's much simpler and cheaper to created a "merged" syscall (i.e. sys_spawn = sys_fork+sys_exec) than to invent a way to upload userland generated bytecodes to kernel space to do that.
Quoted 11 lines · Andrea Arcangeli
Quoted 6 lines · Ingo Molnar
more capable by providing more special system calls like
sys_upcall()to execute a user-space function. (that way a syslet could still execute user-space code without having to exit out of kernel mode too frequently) Or perhaps asys_x86_bytecode()call, that would execute a pre-verified, kernel-stored sequence of simplified x86 bytecode, using the kernel stack.Which means the userspace code would then run with kernel privilege level somehow (after security verifier, whatever). You remember I think it's a plain crazy idea...
Syslets/threadlets do not execute userspace code in kernel - they behave similar to threads. sys_upcall() would be a wrapper for quite complex threadlet machinery.
Signature · 1 line
Evgeniy Polyakov
Quoted 4 lines · Ingo Molnar
wrt. one-shot syscalls, the user-space stack footprint would still probably be there, because even async contexts that only do single-shot processing need to drop out of kernel mode to handle signals.
Why?
The easiest thing to do with signals is to just not pick them up. If the signal was to that *particular* threadlet (ie a "cancel"), then we just want to kill the threadlet. And if the signal was to the thread group, there is no reason why the threadlet should pick it up.
In neither case is there *any* reason to handle the signal in the threadlet, afaik.
And having to have a stack allocation for each threadlet certainly means that you complicate things a lot. Suddenly you have allocations that can't just go away. Again, I'm pointing to the problems I already pointed out with the allocations of the atom structures - quite often you do *not* want to keep track of anything specific for completion time, and that means that you MUST NOT have to de-allocate anythign either.
Again, think aio_read(). With the *exact* current binary interface. PLEASE. If you cannot emulate that with threadlets, then threadlets are *pointless*. On eof the major reasons for the whole exercise was to get rid of the special code in fs/aio.c.
So I repeat: if you cannot do that, and remain binary compatible, don't even bother.
Linus
Quoted 2 lines · Davide Libenzi
My point is, the syslet infrastructure is expensive for the kernel in terms of compat, [...]
it is not. Today i've implemented 64-bit syslets on x86_64 and 32-bit-on-64-bit compat syslets. Both the 64-bit and the 32-bit syslet (and threadlet) binaries work just fine on a 64-bit kernel, and they share 99% of the infrastructure. There's only a single #ifdef CONFIG_COMPAT in kernel/async.c:
#ifdef CONFIG_COMPAT
asmlinkage struct syslet_uatom __user * compat_sys_async_exec(struct syslet_uatom __user *uatom,
struct async_head_user __user *ahu)
{
return __sys_async_exec(uatom, ahu, &compat_sys_call_table,
compat_NR_syscalls);
}
#endif
Even mixed-mode syslets should work (although i havent specifically tested them), where the head switches between 64-bit and 32-bit mode and submits syslets from both 64-bit and from 32-bit mode, and at the same time there might be both 64-bit and 32-bit syslets 'in flight'.
But i'm happy to change the syslet API in any sane way, and did so based on feedback from Jens who is actually using them.
Ingo
Quoted 22 lines · Ingo Molnar
Quoted 2 lines · Davide Libenzi
My point is, the syslet infrastructure is expensive for the kernel in terms of compat, [...]
it is not. Today i've implemented 64-bit syslets on
x86_64and 32-bit-on-64-bit compat syslets. Both the 64-bit and the 32-bit syslet (and threadlet) binaries work just fine on a 64-bit kernel, and they share 99% of the infrastructure. There's only a single #ifdefCONFIG_COMPATinkernel/async.c:#ifdef
CONFIG_COMPATasmlinkage struct
syslet_uatom__user* compat_sys_async_exec(structsyslet_uatom__user*uatom,
structasync_head_user__user*ahu)
{
return __sys_async_exec(uatom, ahu,&compat_sys_call_table,
compat_NR_syscalls);
}
Quoted 1 line
#endif
Did you hide all the complexity of the userspace atom decoding inside another function? :)
How much code would go away, in case we pick a simple/parallel sys_async_exec engine? Atoms decoding, special userspace variable access for loops, jumps/cond/... VM engine.
Quoted 7 lines
Even mixed-mode syslets should work (although i havent specifically tested them), where the head switches between 64-bit and 32-bit mode and submits syslets from both 64-bit and from 32-bit mode, and at the same time there might be both 64-bit and 32-bit syslets 'in flight'.
But i'm happy to change the syslet API in any sane way, and did so based on feedback from Jens who is actually using them.
Wouldn't you agree on a simple/parallel execution engine like me and Linus are proposing (and threadlets, of course)?
- Davide
Quoted 24 lines · Davide Libenzi
Quoted 22 lines · Ingo Molnar
Quoted 2 lines · Davide Libenzi
My point is, the syslet infrastructure is expensive for the kernel in terms of compat, [...]
it is not. Today i've implemented 64-bit syslets on
x86_64and 32-bit-on-64-bit compat syslets. Both the 64-bit and the 32-bit syslet (and threadlet) binaries work just fine on a 64-bit kernel, and they share 99% of the infrastructure. There's only a single #ifdefCONFIG_COMPATinkernel/async.c:#ifdef
CONFIG_COMPATasmlinkage struct
syslet_uatom__user* compat_sys_async_exec(structsyslet_uatom__user*uatom,
structasync_head_user__user*ahu)
{
return __sys_async_exec(uatom, ahu,&compat_sys_call_table,
compat_NR_syscalls);
}
Quoted 4 lines
Quoted 1 line
#endif
Did you hide all the complexity of the userspace atom decoding inside another function? :)
no, i made the 64-bit and 32-bit structures layout-compatible. This makes the 32-bit structure as large as the 64-bit ones, but that's not a big issue, compared to the simplifications it brings.
Quoted 4 lines
Quoted 2 lines
But i'm happy to change the syslet API in any sane way, and did so based on feedback from Jens who is actually using them.
Wouldn't you agree on a simple/parallel execution engine [...]
the thing is, there's almost zero overhead from having those basic things like conditions and the ->next link, and they make it so much more capable. As usual my biggest problem is that you are not trying to use syslets at all - you are only trying to get rid of them ;-) My purpose with syslets is to enable a syslet to do almost anything that user-space could do too, as simply as possible. Syslets could even allocate user-space memory and then use it (i dont think we actually want to do that though). That doesnt mean arbitrary complex code /should/ be done via syslets, or that it wont be significantly slower than what user-space can do, but i'd not like to artificially dumb the engine down. I'm totally willing to simplify/shrink the vectoring of arguments and just about anything else, but your proposals so far (such as your return-value-embedded-in-atom suggestion) all kill important aspects of the engine.
All the existing syslet features were purpose-driven: i actually coded up a sample syslet, trying to do something that makes sense, and added these features based on that. The engine core takes up maybe 50 lines of code.
Ingo
Quoted 8 lines · Ingo Molnar
Quoted 2 lines · Davide Libenzi
Did you hide all the complexity of the userspace atom decoding inside another function? :)
no, i made the 64-bit and 32-bit structures layout-compatible. This makes the 32-bit structure as large as the 64-bit ones, but that's not a big issue, compared to the simplifications it brings.
Do you have a new version to review?
Quoted 5 lines
Quoted 4 lines
Quoted 2 lines
But i'm happy to change the syslet API in any sane way, and did so based on feedback from Jens who is actually using them.
Wouldn't you agree on a simple/parallel execution engine [...]
Quoted 14 lines
the thing is, there's almost zero overhead from having those basic things like conditions and the ->next link, and they make it so much more capable. As usual my biggest problem is that you are not trying to use syslets at all - you are only trying to get rid of them ;-) My purpose with syslets is to enable a syslet to do almost anything that user-space could do too, as simply as possible. Syslets could even allocate user-space memory and then use it (i dont think we actually want to do that though). That doesnt mean arbitrary complex code /should/ be done via syslets, or that it wont be significantly slower than what user-space can do, but i'd not like to artificially dumb the engine down. I'm totally willing to simplify/shrink the vectoring of arguments and just about anything else, but your proposals so far (such as your return-value-embedded-in-atom suggestion) all kill important aspects of the engine.
Ok, we're past the error code in the atom, as Linus pointed out ;) How about this, with async_wait returning asynid's back to a userspace ring buffer?
struct syslet_utaom {
long *result;
unsigned long asynid;
unsigned long nr_sysc;
unsigned long params[8];
};My problem with the syslets in their current form is, do we have a real use for them that justify the extra complexity inside the kernel? Or with a simple/parellel async submission, coupled with threadlets, we can cover a pretty broad range of real life use cases?
- Davide
Quoted 4 lines · Davide Libenzi
Quoted 2 lines · Ingo Molnar
* Davide Libenzi <[email protected]> wrote:
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
Did you hide all the complexity of the userspace atom decoding inside another function? :)
no, i made the 64-bit and 32-bit structures layout-compatible. This makes the 32-bit structure as large as the 64-bit ones, but that's not a big issue, compared to the simplifications it brings.
Do you have a new version to review?
yep, i've just released -v5.
Quoted 9 lines
How about this, with
async_waitreturning asynid's back to a userspace ring buffer?struct syslet_utaom { long *result; unsigned long asynid; unsigned long nr_sysc; unsigned long params[8]; };
we talked about the parameters at length: if they are pointers the layout is significantly more flexible and more capable. It's a pretty similar argument to the return-pointer thing. For example take a look at how the IO syslet atoms in Jens' FIO engine share the same fd. Even if there's 20000 of them. And they are fully cacheable in constructed state. The same goes for the webserving examples i've got in the async-test userspace sample code. I can pick up a cached request and only update req->fd, i dont have to reinit the atoms at all. It stays nicely in the cache, is not re-dirtied, etc.
furthermore, having the parameters as pointers is also an optimization:
look at the copy_uatom() x86 assembly code i did - it can do a simple jump out of the parameter fetching code. I actually tried /both/ of these variants in assembly (as i mentioned it in a previous reply, in the v1 thread) and the speed difference between a pointer and non-pointer variant was negligible. (even with 6 parameters filled in)
but yes ... another two more small changes and your layout will be awfully similar to the current uatom layout =B-)
Quoted 2 lines
My problem with the syslets in their current form is, do we have a real use for them that justify the extra complexity inside the kernel?
i call bullshit. really. I have just gone out and wasted some time cutting & pasting all the syslet engine code: it is 153 lines total, plus 51 lines of comments. The total patchset in comparison is:
35 files changed, 1890 insertions(+), 71 deletions(-)
(and this over-estimates it because if this got removed then we'd still have to add an async execution syscall.) And the code is pretty compact and self-contained. Threadlets share much of the infrastructure with
syslets: for example the completion ring code is _100%_ shared, the async execution code is 98% shared.
You are free to not like it though, and i'm willing to change any aspect of the API to make it more intuitive and more useful, but calling it 'complexity' at this point is just handwaving. And believe it or not, a good number of people actually find syslets pretty cool.
Quoted 2 lines
Or with a simple/parellel async submission, coupled with threadlets, we can cover a pretty broad range of real life use cases?
sure, if we debate its virtualization driven market penetration via self promoting technologies that also drive customer satisfaction, then we'll be able to increase shareholder value by improving the user experience and we'll also succeed in turning this vision into a supply/demand marketplace. Or not?
Ingo
Quoted 8 lines · Ingo Molnar
Quoted 2 lines
Or with a simple/parellel async submission, coupled with threadlets, we can cover a pretty broad range of real life use cases?
sure, if we debate its virtualization driven market penetration via self promoting technologies that also drive customer satisfaction, then we'll be able to increase shareholder value by improving the user experience and we'll also succeed in turning this vision into a supply/demand marketplace. Or not?
Okkey then, I guess it's good to go as is :)
- Davide
Quoted 7 lines
Plus there's the fundamental killer that KAIO is a /lot/ harder to implement (and to maintain) on the kernel side: it has to be implemented for every IO discipline, and even for the IO disciplines it supports at the moment, it is not truly asynchronous for things like metadata blocking or VFS blocking. To handle things like metadata blocking it has to resort to non-statemachine techniques like retries - which are bad for performance.
Yes, yes, yes.
As one of the poor suckers who has been fixing bugs in fs/aio.c and fs/direct-io.c, I really want everyone to read Ingo's paragraph a few times. Have it printed on a t-shirt.
Look at the number of man-years that have gone into fs/aio.c and fs/direct-io.c. After all that effort it *barely* supports non-blocking O_DIRECT IO.
The maintenance overhead of those two files, above all else, is what pushed me to finally try that nutty fibril attempt.
Quoted 4 lines
Syslets/threadlets on the other hand, once the core is implemented, have near zero ongoing maintainance cost (compared to KAIO pushed into every IO subsystem) and cover all IO disciplines and API variants immediately, and they are as perfectly asynchronous as it gets.
Amen.
As an experiment, I'm working on backing the sys_io_*() calls with syslets. It's looking very promising so far.
Quoted 3 lines
So all in one, i used to think that AIO state-machines have a long-term place within the kernel, but with syslets i think i've proven myself embarrasingly wrong =B-)
Welcome to the party :).
- z
Quoted 3 lines · Zach Brown
As one of the poor suckers who has been fixing bugs in
fs/aio.candfs/direct-io.c, I really want everyone to read Ingo's paragraph a few times. Have it printed on a t-shirt.
direct-io.c is evil. Ridiculously.
Quoted 4 lines
Amen.
As an experiment, I'm working on backing the sys_io_*() calls with syslets. It's looking very promising so far.
Great, I'd love to see the comparisons.
Quoted 6 lines
Quoted 4 lines
So all in one, i used to think that AIO state-machines have a long- term
place within the kernel, but with syslets i think i've proven myself embarrasingly wrong =B-)Welcome to the party :).
Well, there are always the 2.4 patches which are properly state drive and reasonably simple. Retry was born out of a need to come up with a mechanism that had less impact on the core kernel code, and yes, it seems to be a failure and in dire need of replacement.
One other implementation to consider is actually using kernel threads compared to how syslets perform. Direct IO for one always blocks, so there shouldn't be much of a performance difference compared to syslets, with the bonus that no arch specific code is needed.
-ben
Signature · 2 lines
"Time is of no importance, Mr. President, only life is important." Don't Email: <[email protected]>.
Quoted 1 line
direct-io.cis evil. Ridiculously.
You will have a hard time finding someone to defend it, I predict :).
There is good news on that front, too. Chris (Mason) is making progress on getting rid of the worst of the Magical Locking that makes buffered races with O_DIRECT ops so awful.
I'm not holding my breath for a page cache so fine grained that it could pin and reference 512B granular user regions and build bios from them, though that sure would be nice :).
Quoted 4 lines
Quoted 2 lines
As an experiment, I'm working on backing the sys_io_*() calls with syslets. It's looking very promising so far.
Great, I'd love to see the comparisons.
I'm out for data. If it sucks, well, we'll know just how much. I'm pretty hopeful that it won't :).
Quoted 4 lines
One other implementation to consider is actually using kernel threads compared to how syslets perform. Direct IO for one always blocks, so there shouldn't be much of a performance difference compared to syslets, with the bonus that no arch specific code is needed.
Yeah, I'm starting with raw kernel threads so we can get some numbers before moving to syslets.
One of the benefits syslets bring is the ability to start processing the next pending request the moment current request processing blocks. In the concurrent O_DIRECT write case that avoids releasing a ton of kernel threads which all just run to serialize on i_mutex (potentially bouncing it around cache domains) as the O_DIRECT ops are built and sent.
-z
Quoted 8 lines · Ingo Molnar
threadlets (and syslets) are parallel contexts and they behave so - queuing and execution semantics are then ontop of that, implemented either by glibc, or implemented by the application. There is no 'pipeline' of requests imposed - the structure of pending requests is totally free-form. For example in
threadlet-test.ci've in essence implemented a 'set of requests' with the submission site only interested in whether all requests are done or not - but any stricter (or even looser) semantics and ordering can be used too.
In short, you have a dataflow model with infinite parallelism, implemented using threads of control mapped willy-nilly onto the underlying hardware. This has not proven to be a successful model in the past.
Quoted 5 lines
in terms of AIO, the best queueing model is i think what the kernel uses
internally: freely ordered, with barrier support. (That is equivalent to a "queue of sets", where the queue are the barriers, and the sets are the requests within barriers. If there is no barrier pending then there's just one large freely-ordered set of requests.)
That's a big part of why Linux scales poorly for workloads that involve a large volume of in-flight I/O transactions. Unless you essentially lock one application thread to each CPU core, with a complete understanding of its cache sharing and latency relationships to all the other cores, and do your own userspace I/O scheduling and dispatching state machine -- which is what all industrial-strength databases and other sorts of transaction engines currently do -- you get the same old best-effort context-thrashing scheduler we've had since Solaris 2.0.
Let's do something genuinely better this time, OK?
Cheers,
- Michael
Quoted 7 lines · Michael K. Edwards
[...] Unless you essentially lock one application thread to each CPU core, with a complete understanding of its cache sharing and latency relationships to all the other cores, and do your own userspace I/O scheduling and dispatching state machine -- which is what all industrial-strength databases and other sorts of transaction engines currently do -- you get the same old best-effort context-thrashing scheduler we've had since Solaris 2.0.
the syslet/threadlet framework has been derived from Tux, which one can accuse of may things, but which i definitely can not accuse of being slow. It has no relationship whatsoever to Solaris 2.0 or later.
Your other mail showed that you have very basic misunderstandings about how threadlets work, on which you based a string of firm but incorrect conclusions. In this discussion i'm mostly interested in specific feedback about syslets/threadlets - thankfully we are past the years of "unless Linux does generic technique X it will stay a hobby OS forever" type of time-wasting discussions.
Ingo
Quoted 3 lines · Ingo Molnar
the syslet/threadlet framework has been derived from Tux, which one can accuse of may things, but which i definitely can not accuse of being slow. It has no relationship whatsoever to Solaris 2.0 or later.
So how well does Tux fare on a NUMA box? The Solaris 2.0 reference was not about the origins of the code, it was about the era when SMP first became relevant to the UNIX application programmer. I remember an emphasis on scheduler scalability then, too. It took them quite a while to figure out that having an efficient scheduler is of little use if you are scheduling the wrong things on the wrong CPUs in the wrong order and thereby thrashing continuously. By that time we had given up and gone back to message passing via the network stack, which was the one kernel component that could figure out how to get state from one CPU to another without taking all of its clothes off and changing underwear in between. Sound familiar?
Quoted 6 lines
Your other mail showed that you have very basic misunderstandings about how threadlets work, on which you based a string of firm but incorrect conclusions. In this discussion i'm mostly interested in specific feedback about syslets/threadlets - thankfully we are past the years of "unless Linux does generic technique X it will stay a hobby OS forever" type of time-wasting discussions.
Well, maybe I'll let someone else weigh in about whether I understood threadlets well enough to provide feedback worth reading. As for the "hobby OS forever" bit, that's an utter misrepresentation of my comments and criticism. Linux is now good enough for Oracle to have more or less abandoned Sun for Linux. That's as good as it needs to be, as far as Oracle and IBM are concerned. The question is now whether it will ever get substantially _better_, so that you can do something constructive with a NUMA box or a 64-core MIPS without having the resources of an Oracle or a Google to build an OS-atop-the-OS.
Cheers,
- Michael
Quoted 84 lines · Ingo Molnar
this is the v3 release of the syslet/threadlet subsystem:
http://redhat.com/~mingo/syslet-patches/
This release came a few days later than i originally wanted, because i've implemented many fundamental changes to the code. The biggest highlights of v3 are:
- "Threadlets": the introduction of the 'threadlet' execution concept.
- syslets: multiple rings support with no kernel-side footprint, the elimination of
mlock()pinning, no async_register/unregister() calls needed anymore and more."Threadlets" are basically the user-space equivalent of syslets: small functions of execution that the kernel attempts to execute without scheduling. If the threadlet blocks, the kernel creates a real thread from it, and execution continues in that thread. The 'head' context (the context that never blocks) returns to the original function that called the threadlet. Threadlets are very easy to use:
long
my_threadlet_fn(void *data){ char *name = data; int fd; fd = open(name, O_RDONLY); if (fd < 0) goto out; fstat(fd, &stat); read(fd, buf, count) ...out:
return threadlet_complete(); }
main(){ done = threadlet_exec(threadlet_fn, new_stack, &user_head); if (!done) reqs_queued++; }There is no limitation whatsoever about how a threadlet function can look like: it can use arbitrary system-calls and all execution will be procedural. There is no 'registration' needed when running threadlets
either: the kernel will take care of all the details, user-space just runs a threadlet without any preparation and that's it.Completion of async threadlets can be done from user-space via any of the existing APIs: in
threadlet-test.c(see the async-test-v3.tar.gz user-space examples at the URL above) i've for example used a futex between the head and the async threads to do threadlet notification. Butselect(),poll()or signals can be used too - whichever is most convenient to the application writer.Threadlets can also be thought of as 'optional threads': they execute in the original context as long as they do not block, but once they block, they are moved off into their separate thread context - and the original context can continue execution.
Threadlets can also be thought of as 'on-demand parallelism': user-space does not have to worry about setting up, sizing and feeding a thread pool - the kernel will execute the workload in a single-threaded manner as long as it makes sense, but once the context blocks, a parallel context is created. So parallelism inside applications is utilized in a natural way. (The best place to do this is in the kernel - user-space has no idea about what level of parallelism is best for any given moment.)
I believe this threadlet concept is what user-space will want to use for programmable parallelism.
[ Note that right now there's a pair of system-calls:
sys_threadlet_on()
andsys_threadlet_off()that demarks the beginning and the end of a syslet function, which enter the kernel even in the 'cached' case - but my plan is to do these two system calls via a vsyscall, without having to enter the kernel at all. That will reduce cached threadlet execution NULL-overhead to around 10 nsecs - making it essentially zero. ]
Quoted 7 lines
Threadlets share much of the scheduling infrastructure with syslets.
Syslets (small, kernel-side, scripted "syscall plugins") are still supported - they are (much...) harder to program than threadlets but they allow the highest performance. Core infrastructure libraries like glibc/libaio are expected to use syslets. Jens Axboe's FIO tool already includes support for v2 syslets, and the following patch updates FIO to
Ah, glad to see that - I was wondering if it was worthwhile to try adding syslet support to aio-stress to be able to perform some comparisons. Hopefully FIO should be able to generate a similar workload, but I haven't tried it yet so am not sure. Are you planning to upload some results (so I can compare it with patterns I am familiar with) ?
Regards
Suparna
Quoted 25 lines
the v3 API:
http://redhat.com/~mingo/syslet-patches/fio-syslet-v3.patch
Furthermore, the syslet code and API has been significantly enhanced as well:
- support for multiple completion rings has been added
- there is no more mlock()ing of the completion ring(s)
- sys_async_register()/unregister() has been removed as it is not needed anymore.
sys_async_exec()can be called straight away.- there is no kernel-side resource used up by async completion rings at all (all the state is in user-space), so an arbitrary number of completion rings are supported.
plus lots of bugs were fixed and a good number of cleanups were done as well. The v3 code is ABI-incompatible with v2, due to these fundamental changes.
As always, comments, suggestions, reports are welcome.
Ingo
Signature · 3 lines
Suparna Bhattacharya ([email protected]) Linux Technology Center IBM Software Lab, India
Quoted 9 lines · Suparna Bhattacharya
Quoted 8 lines
Threadlets share much of the scheduling infrastructure with syslets.
Syslets (small, kernel-side, scripted "syscall plugins") are still supported - they are (much...) harder to program than threadlets but they allow the highest performance. Core infrastructure libraries like glibc/libaio are expected to use syslets. Jens Axboe's FIO tool already includes support for v2 syslets, and the following patch updates FIO to
Quoted 3 lines
Ah, glad to see that - I was wondering if it was worthwhile to try adding syslet support to aio-stress to be able to perform some comparisons. [...]
i think it would definitely be worth it.
Quoted 3 lines
[...] Hopefully FIO should be able to generate a similar workload, but I haven't tried it yet so am not sure. Are you planning to upload some results (so I can compare it with patterns I am familiar with) ?
i had no time yet to do careful benchmarks. Right now my impression from quick testing is that libaio performance can be exceeded via syslets. So it would be very interesting if you could try this too, independently of me.
Ingo
A quick fio test (was Re: [patch 00/13] Syslets, "Threadlets", generic AIO support, v3)
In reply to Ingo MolnarQuoted 3 lines · Ingo Molnar
this is the v3 release of the syslet/threadlet subsystem:
[snip]
Ingo, some testing of the experimental syslet queueing stuff, in the syslet-testing branch of fio.
Fio job file:
[global]
bs=8k
size=1g
direct=0
ioengine=syslet-rw
iodepth=32
rw=read
[file]
filename=/ramfs/testfile
Only changes between runs was changing ioengine and iodepth as indicated in the table below.
Results:
Engine Depth Bw (MiB/sec)
--------------------------------------------
libaio 1 441 syslet 1 574 sync 1 589 libaio 32 613 syslet 32 681
Results are stable to within +/- 1MiB/sec. So you can see that syslet are still a bit slower than sync for depth 1, but beats the pants off libaio for equal depths. Note that this is buffered IO, I'll be out for the weekend, but I'll hack some direct IO testing up next week to compare "real" queuing.
Just a quick microbenchmark to gauge current overhead...
Signature · 1 line
Jens Axboe
Re: A quick fio test (was Re: [patch 00/13] Syslets, "Threadlets", generic AIO support, v3)
Quoted 36 lines · Jens Axboe
Quoted 3 lines · Ingo Molnar
this is the v3 release of the syslet/threadlet subsystem:
[snip]
Ingo, some testing of the experimental syslet queueing stuff, in the syslet-testing branch of fio.
Fio job file:
[global]
bs=8k
size=1g
direct=0
ioengine=syslet-rw
iodepth=32
rw=read[file]
filename=/ramfs/testfileOnly changes between runs was changing ioengine and iodepth as indicated in the table below.
Results:
Engine Depth Bw (MiB/sec)
--------------------------------------------libaio 1 441 syslet 1 574 sync 1 589 libaio 32 613 syslet 32 681
Quoted 7 lines
Results are stable to within +/- 1MiB/sec. So you can see that syslet are still a bit slower than sync for depth 1, but beats the pants off libaio for equal depths. Note that this is buffered IO, I'll be out for the weekend, but I'll hack some direct IO testing up next week to compare "real" queuing.
Just a quick microbenchmark to gauge current overhead...
This is just ramfs, to gauge pure overheads, is that correct ?
BTW, I'm not surprised at Ingo's initial results of syslet vs libaio overheads, for aio-stress/fio type streaming io runs, because these cases do not involve large numbers of outstanding ios. So the overhead of thread creation with syslets is amortized across the entire run of io submissions because of the reuse of already created async threads. While in the libaio case there is a setup and teardown of kiocbs per request.
What I have been concerned about instead in the past when considering thread based AIO implementations is the resource(memory) consumption impact on overall system performance and adaptability to varying loads. It is nice that we can avoid that for the cached cases, but for the general blocking cases, it is still not clear to me whether we have addressed this well enough yet. I used to think that even the kiocb was too heavyweight for its purpose ... especially in terms of scaling to larger loads.
As a really crude (and not very realistic) example of the potential impact of large numbers of outstanding IOs, I tried some quick direct IO comparisons using fio:
[global]
ioengine=syslet-rw
buffered=0
rw=randread
bs=64k
size=1024m
iodepth=64
Engine Depth Bw (MiB/sec)
libaio 64 17.323
syslet 64 17.524
libaio 20000 15.226
syslet 20000 11.015
Regards
Suparna
Signature · 3 lines
Suparna Bhattacharya ([email protected]) Linux Technology Center IBM Software Lab, India
Re: A quick fio test (was Re: [patch 00/13] Syslets, "Threadlets", generic AIO support, v3)
Quoted 11 lines · Suparna Bhattacharya
As a really crude (and not very realistic) example of the potential impact of large numbers of outstanding IOs, I tried some quick direct IO comparisons using fio:
[global]
ioengine=syslet-rw
buffered=0
rw=randread
bs=64k
size=1024m
iodepth=64
could you please try those iodepth=20000 tests with the latest fio-testing branch of fio as well? Jens wrote a new, smarter syslet plugin for FIO. You'll need the v3 syslet kernel plus:
git-clone git://git.kernel.dk/data/git/fio.git
cd fio
git-checkout syslet-testing
my expectation is that it should behave better with iodepth=20000 (although i havent tried that yet).
Ingo
Re: A quick fio test (was Re: [patch 00/13] Syslets, "Threadlets", generic AIO support, v3)
Quoted 19 lines · Ingo Molnar
Quoted 11 lines · Suparna Bhattacharya
As a really crude (and not very realistic) example of the potential impact of large numbers of outstanding IOs, I tried some quick direct IO comparisons using fio:
[global]
ioengine=syslet-rw
buffered=0
rw=randread
bs=64k
size=1024m
iodepth=64could you please try those iodepth=20000 tests with the latest fio-testing branch of fio as well? Jens wrote a new, smarter syslet plugin for FIO. You'll need the v3 syslet kernel plus:
Quoted 6 lines
git-clone
git://git.kernel.dk/data/git/fio.git
cd fio
git-checkout syslet-testingmy expectation is that it should behave better with iodepth=20000 (although i havent tried that yet).
I picked up the fio snapshot from 22nd Feb (fio-git-20070222212513.tar.gz) and used the v3 syslet patches from your web-site.
Do I still need to get something more recent ?
Regards
Suparna
Quoted 2 lines
Ingo
Signature · 3 lines
Suparna Bhattacharya ([email protected]) Linux Technology Center IBM Software Lab, India
Re: A quick fio test (was Re: [patch 00/13] Syslets, "Threadlets", generic AIO support, v3)
Quoted 24 lines · Suparna Bhattacharya
Quoted 23 lines · Ingo Molnar
Quoted 11 lines · Suparna Bhattacharya
As a really crude (and not very realistic) example of the potential impact of large numbers of outstanding IOs, I tried some quick direct IO comparisons using fio:
[global]
ioengine=syslet-rw
buffered=0
rw=randread
bs=64k
size=1024m
iodepth=64could you please try those iodepth=20000 tests with the latest fio-testing branch of fio as well? Jens wrote a new, smarter syslet plugin for FIO. You'll need the v3 syslet kernel plus:
git-clone
git://git.kernel.dk/data/git/fio.git
cd fio
git-checkout syslet-testing
Quoted 7 lines
Quoted 2 lines
my expectation is that it should behave better with iodepth=20000 (although i havent tried that yet).
I picked up the fio snapshot from 22nd Feb (fio-git-20070222212513.tar.gz) and used the v3 syslet patches from your web-site.
Do I still need to get something more recent ?
Yes, you need to test the syslet+testing branch that Ingo referenced. Your test above is not totally fair right now, since you are doing significantly less system calls with libaio. So to compare apples with apples, try the syslet-testing branch. If you can't get it because of firewall problems, check http://brick.kernel.dk/snaps/ for the latest fio snapshot. If it has the syslet-testing branch, then that is recent enough.
Signature · 1 line
Jens Axboe
Re: A quick fio test (was Re: [patch 00/13] Syslets, "Threadlets", generic AIO support, v3)
Quoted 33 lines · Jens Axboe
Quoted 24 lines · Suparna Bhattacharya
Quoted 23 lines · Ingo Molnar
Quoted 11 lines · Suparna Bhattacharya
As a really crude (and not very realistic) example of the potential impact of large numbers of outstanding IOs, I tried some quick direct IO comparisons using fio:
[global]
ioengine=syslet-rw
buffered=0
rw=randread
bs=64k
size=1024m
iodepth=64could you please try those iodepth=20000 tests with the latest fio-testing branch of fio as well? Jens wrote a new, smarter syslet plugin for FIO. You'll need the v3 syslet kernel plus:
git-clone
git://git.kernel.dk/data/git/fio.git
cd fio
git-checkout syslet-testingQuoted 7 lines
Quoted 2 lines
my expectation is that it should behave better with iodepth=20000 (although i havent tried that yet).
I picked up the fio snapshot from 22nd Feb (fio-git-20070222212513.tar.gz) and used the v3 syslet patches from your web-site.
Do I still need to get something more recent ?
Quoted 7 lines
Yes, you need to test the syslet+testing branch that Ingo referenced. Your test above is not totally fair right now, since you are doing significantly less system calls with libaio. So to compare apples with apples, try the syslet-testing branch. If you can't get it because of firewall problems, check http://brick.kernel.dk/snaps/ for the latest fio snapshot. If it has the syslet-testing branch, then that is recent enough.
I have a feeling this is getting to be a little more bleeding edge than I had anticipated :), so will just hold off for a bit until this crystallizes a bit.
Regards
Suparna
Quoted 3 lines
Signature · 2 lines
-- Jens Axboe
Signature · 3 lines
Suparna Bhattacharya ([email protected]) Linux Technology Center IBM Software Lab, India
Re: A quick fio test (was Re: [patch 00/13] Syslets, "Threadlets", generic AIO support, v3)
Quoted 42 lines · Suparna Bhattacharya
Quoted 33 lines · Jens Axboe
Quoted 24 lines · Suparna Bhattacharya
Quoted 23 lines · Ingo Molnar
Quoted 11 lines · Suparna Bhattacharya
As a really crude (and not very realistic) example of the potential impact of large numbers of outstanding IOs, I tried some quick direct IO comparisons using fio:
[global]
ioengine=syslet-rw
buffered=0
rw=randread
bs=64k
size=1024m
iodepth=64could you please try those iodepth=20000 tests with the latest fio-testing branch of fio as well? Jens wrote a new, smarter syslet plugin for FIO. You'll need the v3 syslet kernel plus:
git-clone
git://git.kernel.dk/data/git/fio.git
cd fio
git-checkout syslet-testingQuoted 7 lines
Quoted 2 lines
my expectation is that it should behave better with iodepth=20000 (although i havent tried that yet).
I picked up the fio snapshot from 22nd Feb (fio-git-20070222212513.tar.gz) and used the v3 syslet patches from your web-site.
Do I still need to get something more recent ?
Quoted 7 lines
Yes, you need to test the syslet+testing branch that Ingo referenced. Your test above is not totally fair right now, since you are doing significantly less system calls with libaio. So to compare apples with apples, try the syslet-testing branch. If you can't get it because of firewall problems, check http://brick.kernel.dk/snaps/ for the latest fio snapshot. If it has the syslet-testing branch, then that is recent enough.
Quoted 3 lines
I have a feeling this is getting to be a little more bleeding edge than I had anticipated :), so will just hold off for a bit until this crystallizes a bit.
Fair enough, I'll try your test with a huge number of pending requests and see how it fares.
Signature · 1 line
Jens Axboe
Re: A quick fio test (was Re: [patch 00/13] Syslets, "Threadlets", generic AIO support, v3)
In reply to Suparna BhattacharyaSome more results, using a larger number of processes and io depths. A repeat of the tests from friday, with added depth 20000 for syslet and libaio:
Engine Depth Processes Bw (MiB/sec)
----------------------------------------------------
libaio 1 1 602 syslet 1 1 759 sync 1 1 776 libaio 32 1 832 syslet 32 1 898 libaio 20000 1 581 syslet 20000 1 609
syslet still on top. Measuring O_DIRECT reads (of 4kb size) on ramfs with 100 processes each with a depth of 200, reading a per-process private file of 10mb (need to fit in my ram...) 10 times each. IOW, doing 10,000MiB of IO in total:
Engine Depth Processes Bw (MiB/sec)
----------------------------------------------------
libaio 200 100 1488 syslet 200 100 1714
Results are stable to within approx +/- 10MiB/sec. The syslet case completes a whole second faster than libaio (~6 vs ~7 seconds). Testing was done with fio HEAD eb7c8ae27bc301b77490b3586dd5ccab7c95880a, and it uses the v4 patch series.
Engine Depth Processes Bw (MiB/sec)
----------------------------------------------------
libaio 200 100 1488 syslet 200 100 1714 sync 200 100 1843
Signature · 1 line
Jens Axboe
Re: A quick fio test (was Re: [patch 00/13] Syslets, "Threadlets", generic AIO support, v3)
Quoted 15 lines · Jens Axboe
Some more results, using a larger number of processes and io depths. A repeat of the tests from friday, with added depth 20000 for syslet and libaio:
Engine Depth Processes Bw (MiB/sec)
----------------------------------------------------libaio 1 1 602 syslet 1 1 759 sync 1 1 776 libaio 32 1 832 syslet 32 1 898 libaio 20000 1 581 syslet 20000 1 609
Quoted 4 lines
syslet still on top. Measuring
O_DIRECTreads (of 4kb size) on ramfs with 100 processes each with a depth of 200, reading a per-process private file of 10mb (need to fit in my ram...) 10 times each. IOW, doing 10,000MiB of IO in total:
But, why ramfs ? Don't we want to exercise the case where O_DIRECT actually blocks ? Or am I missing something here ?
Regards
Suparna
Quoted 19 lines
Engine Depth Processes Bw (MiB/sec)
----------------------------------------------------libaio 200 100 1488 syslet 200 100 1714Results are stable to within approx +/- 10MiB/sec. The syslet case completes a whole second faster than libaio (~6 vs ~7 seconds). Testing was done with fio HEAD
eb7c8ae27bc301b77490b3586dd5ccab7c95880a, and it uses the v4 patch series.Engine Depth Processes Bw (MiB/sec)
----------------------------------------------------libaio 200 100 1488 syslet 200 100 1714 sync 200 100 1843Signature · 1 line
Jens Axboe
Signature · 3 lines
Suparna Bhattacharya ([email protected]) Linux Technology Center IBM Software Lab, India
Re: A quick fio test (was Re: [patch 00/13] Syslets, "Threadlets", generic AIO support, v3)
Quoted 7 lines · Suparna Bhattacharya
Quoted 4 lines
syslet still on top. Measuring
O_DIRECTreads (of 4kb size) on ramfs with 100 processes each with a depth of 200, reading a per-process private file of 10mb (need to fit in my ram...) 10 times each. IOW, doing 10,000MiB of IO in total:But, why ramfs ? Don't we want to exercise the case where
O_DIRECTactually blocks ? Or am I missing something here ?
ramfs is just the easiest way to measure the pure CPU overhead of a workload without real IO delays (and resulting idle time) getting in the way. It's certainly not the same thing, but still it's pretty useful most of the time. I used a different method, loopback block device, and got similar results. [ Real IO shows similar results as well, but is a lot more noisy and hence harder to interpret (and thus easier to get wrong). ]
Ingo
Re: A quick fio test (was Re: [patch 00/13] Syslets, "Threadlets", generic AIO support, v3)
In reply to Suparna BhattacharyaQuoted 16 lines · Suparna Bhattacharya
Quoted 15 lines · Jens Axboe
Some more results, using a larger number of processes and io depths. A repeat of the tests from friday, with added depth 20000 for syslet and libaio:
Engine Depth Processes Bw (MiB/sec)
----------------------------------------------------libaio 1 1 602 syslet 1 1 759 sync 1 1 776 libaio 32 1 832 syslet 32 1 898 libaio 20000 1 581 syslet 20000 1 609
Quoted 7 lines
Quoted 4 lines
syslet still on top. Measuring
O_DIRECTreads (of 4kb size) on ramfs with 100 processes each with a depth of 200, reading a per-process private file of 10mb (need to fit in my ram...) 10 times each. IOW, doing 10,000MiB of IO in total:But, why ramfs ? Don't we want to exercise the case where
O_DIRECTactually blocks ? Or am I missing something here ?
Just overhead numbers for that test case, lets try something like your described job.
Test case is doing random reads from /dev/sdb, in chunks of 64kb:
Engine Depth Processes Bw (KiB/sec)
----------------------------------------------------
libaio 200 100 2813 syslet 200 100 3944 libaio 20000 1 2793 syslet 20000 1 3854 sync (*) 20000 1 2866
deadline was used for IO scheduling, to minimize impact. Not sure why syslet actually does so much better here, looing at vmstat the rate is steady and all runs are basically 50/50 idle/wait. One difference is that the submission itself takes a long time on libaio, since the io_submit() will block on request allocation. The generated IO pattern from each process is the same for all runs. The drive is a lousy sata that doesn't even do queuing, FWIW.
[*] Just for comparison, the depth is obviously really 1 at the kernel
side since it's sync.
Signature · 1 line
Jens Axboe
Re: A quick fio test (was Re: [patch 00/13] Syslets, "Threadlets", generic AIO support, v3)
Quoted 38 lines · Jens Axboe
Quoted 16 lines · Suparna Bhattacharya
Quoted 15 lines · Jens Axboe
Some more results, using a larger number of processes and io depths. A repeat of the tests from friday, with added depth 20000 for syslet and libaio:
Engine Depth Processes Bw (MiB/sec)
----------------------------------------------------libaio 1 1 602 syslet 1 1 759 sync 1 1 776 libaio 32 1 832 syslet 32 1 898 libaio 20000 1 581 syslet 20000 1 609Quoted 7 lines
Quoted 4 lines
syslet still on top. Measuring
O_DIRECTreads (of 4kb size) on ramfs with 100 processes each with a depth of 200, reading a per-process private file of 10mb (need to fit in my ram...) 10 times each. IOW, doing 10,000MiB of IO in total:But, why ramfs ? Don't we want to exercise the case where
O_DIRECTactually blocks ? Or am I missing something here ?Just overhead numbers for that test case, lets try something like your described job.
Test case is doing random reads from
/dev/sdb, in chunks of 64kb:Engine Depth Processes Bw (KiB/sec)
----------------------------------------------------libaio 200 100 2813 syslet 200 100 3944 libaio 20000 1 2793 syslet 20000 1 3854 sync (*) 20000 1 2866
Quoted 7 lines
deadline was used for IO scheduling, to minimize impact. Not sure why syslet actually does so much better here, looing at vmstat the rate is steady and all runs are basically 50/50 idle/wait. One difference is that the submission itself takes a long time on libaio, since the
io_submit()will block on request allocation. The generated IO pattern from each process is the same for all runs. The drive is a lousy sata that doesn't even do queuing, FWIW.
I tried the latest fio code with syslet v4, and my results are a little different - have yet to figure out why or what to make of it. I hope I have all the right pieces now.
This is an ext2 filesystem, SCSI AIC7xxx.
I used an iodepth_batch size of 8 to limit the number of ios in a single io_submit (thanks for adding that parameter to fio !), like we did in aio-stress.
Engine Depth Batch Bw (KiB/sec)
----------------------------------------------------
libaio 64 8 17,226
syslet 64 8 17,620
libaio 20000 8 18,552
syslet 20000 8 14,935
Which is not bad, actually.
If I do not specify the iodepth_batch (i.e. default to depth), then the difference becomes more pronounced at higher depths. However, I doubt whether anyone would be using such high batch sizes in practice ...
Engine Depth Batch Bw (KiB/sec)
----------------------------------------------------
libaio 64 default 17,429
syslet 64 default 16,155
libaio 20000 default 15,494
syslet 20000 default 7,971
Often times it is the application tuning that makes all the difference, so am not really sure how much to read into these results. That's always been the hard part of async io ...
Regards
Suparna
Quoted 6 lines
[*] Just for comparison, the depth is obviously really 1 at the kernel
side since it's sync.Signature · 1 line
Jens Axboe
Signature · 3 lines
Suparna Bhattacharya ([email protected]) Linux Technology Center IBM Software Lab, India
Re: A quick fio test (was Re: [patch 00/13] Syslets, "Threadlets", generic AIO support, v3)
Quoted 66 lines · Suparna Bhattacharya
Quoted 38 lines · Jens Axboe
Quoted 16 lines · Suparna Bhattacharya
Quoted 15 lines · Jens Axboe
Some more results, using a larger number of processes and io depths. A repeat of the tests from friday, with added depth 20000 for syslet and libaio:
Engine Depth Processes Bw (MiB/sec)
----------------------------------------------------libaio 1 1 602 syslet 1 1 759 sync 1 1 776 libaio 32 1 832 syslet 32 1 898 libaio 20000 1 581 syslet 20000 1 609Quoted 7 lines
Quoted 4 lines
syslet still on top. Measuring
O_DIRECTreads (of 4kb size) on ramfs with 100 processes each with a depth of 200, reading a per-process private file of 10mb (need to fit in my ram...) 10 times each. IOW, doing 10,000MiB of IO in total:But, why ramfs ? Don't we want to exercise the case where
O_DIRECTactually blocks ? Or am I missing something here ?Just overhead numbers for that test case, lets try something like your described job.
Test case is doing random reads from
/dev/sdb, in chunks of 64kb:Engine Depth Processes Bw (KiB/sec)
----------------------------------------------------libaio 200 100 2813 syslet 200 100 3944 libaio 20000 1 2793 syslet 20000 1 3854 sync (*) 20000 1 2866Quoted 7 lines
deadline was used for IO scheduling, to minimize impact. Not sure why syslet actually does so much better here, looing at vmstat the rate is steady and all runs are basically 50/50 idle/wait. One difference is that the submission itself takes a long time on libaio, since the
io_submit()will block on request allocation. The generated IO pattern from each process is the same for all runs. The drive is a lousy sata that doesn't even do queuing, FWIW.I tried the latest fio code with syslet v4, and my results are a little different - have yet to figure out why or what to make of it. I hope I have all the right pieces now.
This is an ext2 filesystem, SCSI AIC7xxx.
I used an
iodepth_batchsize of 8 to limit the number of ios in a singleio_submit(thanks for adding that parameter to fio !), like we did in aio-stress.Engine Depth Batch Bw (KiB/sec)
----------------------------------------------------
libaio 64 8 17,226
syslet 64 8 17,620
libaio 20000 8 18,552
syslet 20000 8 14,935
Quoted 1 line
Which is not bad, actually.
It's not bad for such a high depth/batch setting, but I still wonder why are results are so different. I'll look around for an x86 box with some TCQ/NCQ enabled storage attached for testing. Can you pass me your command line or job file (whatever you use) so we are on the same page?
Quoted 11 lines
If I do not specify the
iodepth_batch(i.e. default to depth), then the difference becomes more pronounced at higher depths. However, I doubt whether anyone would be using such high batch sizes in practice ...Engine Depth Batch Bw (KiB/sec)
----------------------------------------------------
libaio 64 default 17,429
syslet 64 default 16,155
libaio 20000 default 15,494
syslet 20000 default 7,971
If iodepth_batch isn't set, the syslet queued io will be serialized and not take advantage of queueing. How does the job file perform with ioengine=sync?
Quoted 3 lines
Often times it is the application tuning that makes all the difference, so am not really sure how much to read into these results. That's always been the hard part of async io ...
Yes I agree, it's handy to get an overview though.
Signature · 1 line
Jens Axboe
Re: A quick fio test (was Re: [patch 00/13] Syslets, "Threadlets", generic AIO support, v3)
My two coins:
# cat job [global] bs=8k size=1g direct=0 ioengine=sync iodepth=32 rw=read
[file]
filename=/home/user/test
sync:
READ: io=1,024MiB, aggrb=39,329KiB/s, minb=39,329KiB/s, maxb=39,329KiB/s, mint=27301msec, maxt=27301msec
libaio:
READ: io=1,024MiB, aggrb=39,435KiB/s, minb=39,435KiB/s, maxb=39,435KiB/s, mint=27228msec, maxt=27228msec
syslet-rw:
READ: io=1,024MiB, aggrb=29,567KiB/s, minb=29,567KiB/s, maxb=29,567KiB/s, mint=36315msec, maxt=36315msec
During syslet-rw test about 9500 async schduledes happend. I use fio-git-20070226150114.tar.gz
P.S. Jens, fio_latest.tar.gz has wrong permissions, it can not be opened.
Signature · 1 line
Evgeniy Polyakov
Re: A quick fio test (was Re: [patch 00/13] Syslets, "Threadlets", generic AIO support, v3)
Quoted 21 lines · Evgeniy Polyakov
My two coins:
# cat job [global] bs=8k size=1g direct=0 ioengine=sync iodepth=32 rw=read[file]
filename=/home/user/testsync:
READ: io=1,024MiB, aggrb=39,329KiB/s, minb=39,329KiB/s, maxb=39,329KiB/s, mint=27301msec, maxt=27301mseclibaio:
READ: io=1,024MiB, aggrb=39,435KiB/s, minb=39,435KiB/s, maxb=39,435KiB/s, mint=27228msec, maxt=27228msec
Quoted 6 lines
syslet-rw:
READ: io=1,024MiB, aggrb=29,567KiB/s, minb=29,567KiB/s, maxb=29,567KiB/s, mint=36315msec, maxt=36315msecDuring syslet-rw test about 9500 async schduledes happend. I use fio-git-20070226150114.tar.gz
That looks pretty pathetic :-). What IO scheduler did you use? syslets will confuse CFQ currently, so you want to compare with using eg deadline or as. That is one of the downsides of this approach.
I'll try your test as soon as this bisect series is done.
Quoted 2 lines
P.S. Jens, fio_latest.tar.gz has wrong permissions, it can not be opened.
Oh thanks, indeed. It was disabled symlinks that broke it. Fixed now.
Signature · 1 line
Jens Axboe
Re: A quick fio test (was Re: [patch 00/13] Syslets, "Threadlets", generic AIO support, v3)
Quoted 26 lines · Jens Axboe
Quoted 25 lines · Evgeniy Polyakov
My two coins:
# cat job [global] bs=8k size=1g direct=0 ioengine=sync iodepth=32 rw=read[file]
filename=/home/user/testsync:
READ: io=1,024MiB, aggrb=39,329KiB/s, minb=39,329KiB/s, maxb=39,329KiB/s, mint=27301msec, maxt=27301mseclibaio:
READ: io=1,024MiB, aggrb=39,435KiB/s, minb=39,435KiB/s, maxb=39,435KiB/s, mint=27228msec, maxt=27228msecsyslet-rw:
READ: io=1,024MiB, aggrb=29,567KiB/s, minb=29,567KiB/s, maxb=29,567KiB/s, mint=36315msec, maxt=36315msec
Quoted 6 lines
Quoted 2 lines
During syslet-rw test about 9500 async schduledes happend. I use fio-git-20070226150114.tar.gz
That looks pretty pathetic :-). What IO scheduler did you use? syslets will confuse CFQ currently, so you want to compare with using eg deadline or as. That is one of the downsides of this approach.
Deadline shows this:
sync:
READ: io=1,024MiB, aggrb=38,212KiB/s, minb=38,212KiB/s, maxb=38,212KiB/s, mint=28099msec, maxt=28099msec
libaio:
READ: io=1,024MiB, aggrb=37,933KiB/s, minb=37,933KiB/s, maxb=37,933KiB/s, mint=28306msec, maxt=28306msec
syslet-rw:
READ: io=1,024MiB, aggrb=34,759KiB/s, minb=34,759KiB/s, maxb=34,759KiB/s, mint=30891msec, maxt=30891msec
There were about 10k async schedulings.
Signature · 1 line
Evgeniy Polyakov
Re: A quick fio test (was Re: [patch 00/13] Syslets, "Threadlets", generic AIO support, v3)
Quoted 44 lines · Evgeniy Polyakov
Quoted 26 lines · Jens Axboe
Quoted 25 lines · Evgeniy Polyakov
My two coins:
# cat job [global] bs=8k size=1g direct=0 ioengine=sync iodepth=32 rw=read[file]
filename=/home/user/testsync:
READ: io=1,024MiB, aggrb=39,329KiB/s, minb=39,329KiB/s, maxb=39,329KiB/s, mint=27301msec, maxt=27301mseclibaio:
READ: io=1,024MiB, aggrb=39,435KiB/s, minb=39,435KiB/s, maxb=39,435KiB/s, mint=27228msec, maxt=27228msecsyslet-rw:
READ: io=1,024MiB, aggrb=29,567KiB/s, minb=29,567KiB/s, maxb=29,567KiB/s, mint=36315msec, maxt=36315msecQuoted 6 lines
Quoted 2 lines
During syslet-rw test about 9500 async schduledes happend. I use fio-git-20070226150114.tar.gz
That looks pretty pathetic :-). What IO scheduler did you use? syslets will confuse CFQ currently, so you want to compare with using eg deadline or as. That is one of the downsides of this approach.
Deadline shows this:
sync:
READ: io=1,024MiB, aggrb=38,212KiB/s, minb=38,212KiB/s, maxb=38,212KiB/s, mint=28099msec, maxt=28099mseclibaio:
READ: io=1,024MiB, aggrb=37,933KiB/s, minb=37,933KiB/s, maxb=37,933KiB/s, mint=28306msec, maxt=28306msec
Quoted 5 lines
syslet-rw:
READ: io=1,024MiB, aggrb=34,759KiB/s, minb=34,759KiB/s, maxb=34,759KiB/s, mint=30891msec, maxt=30891msecThere were about 10k async schedulings.
I think the issue here is pretty simple - when fio gets a queue full like condition (it reaches the depth you set, 32), it commits them and starts queuing again. Since that'll likely block, it'll get issued by another process. So you suddenly have a nice sequence of reads from one process (pending, only one is actually committed since it's serialized), and then a read further down the line that goes behind those you already committed. Then result is seeky, where it should have been sequential.
Do you get expected results if you set iodepth_low=1? That'll make fio drain the queue before building it up again, should get you a sequential access pattern with syslets.
Signature · 1 line
Jens Axboe
Re: A quick fio test (was Re: [patch 00/13] Syslets, "Threadlets", generic AIO support, v3)
Quoted 24 lines · Jens Axboe
Quoted 10 lines
Deadline shows this:
sync:
READ: io=1,024MiB, aggrb=38,212KiB/s, minb=38,212KiB/s, maxb=38,212KiB/s, mint=28099msec, maxt=28099mseclibaio:
READ: io=1,024MiB, aggrb=37,933KiB/s, minb=37,933KiB/s, maxb=37,933KiB/s, mint=28306msec, maxt=28306msecQuoted 5 lines
syslet-rw:
READ: io=1,024MiB, aggrb=34,759KiB/s, minb=34,759KiB/s, maxb=34,759KiB/s, mint=30891msec, maxt=30891msecThere were about 10k async schedulings.
I think the issue here is pretty simple - when fio gets a queue full like condition (it reaches the depth you set, 32), it commits them and starts queuing again. Since that'll likely block, it'll get issued by another process. So you suddenly have a nice sequence of reads from one process (pending, only one is actually committed since it's serialized), and then a read further down the line that goes behind those you already committed. Then result is seeky, where it should have been sequential.
Quoted 3 lines
Do you get expected results if you set iodepth_low=1? That'll make fio drain the queue before building it up again, should get you a sequential access pattern with syslets.
With such a change results should be better - not only because seek is removed with sequential read, but also number of working threads decreases with time - until queue is filled again.
So, syslet-rw has increased to 37mb/sec out of 39/sync and 38/libaio, the latter two did not changed.
With iodepth of 10k, I get the same performance for
libaio and syslets - about 36mb/sec, it does not depend on iodepth_low being set to 1 or default (full).
So syslets have small problems with small number of iodepth - its performance is about 34mb/sec and then increases to 36 with iodepth grow. While libaio decreases from 38 down to 36 mb/sec.
iodepth_low=1 helps syslets to have 37mb/sec with iodepth=32, with 3200 and 10k it does not play any role.
Quoted 2 lines
Signature · 1 line
Jens Axboe
Signature · 1 line
Evgeniy Polyakov
Re: A quick fio test (was Re: [patch 00/13] Syslets, "Threadlets", generic AIO support, v3)
Quoted 25 lines · Evgeniy Polyakov
Quoted 24 lines · Jens Axboe
Quoted 10 lines
Deadline shows this:
sync:
READ: io=1,024MiB, aggrb=38,212KiB/s, minb=38,212KiB/s, maxb=38,212KiB/s, mint=28099msec, maxt=28099mseclibaio:
READ: io=1,024MiB, aggrb=37,933KiB/s, minb=37,933KiB/s, maxb=37,933KiB/s, mint=28306msec, maxt=28306msecQuoted 5 lines
syslet-rw:
READ: io=1,024MiB, aggrb=34,759KiB/s, minb=34,759KiB/s, maxb=34,759KiB/s, mint=30891msec, maxt=30891msecThere were about 10k async schedulings.
I think the issue here is pretty simple - when fio gets a queue full like condition (it reaches the depth you set, 32), it commits them and starts queuing again. Since that'll likely block, it'll get issued by another process. So you suddenly have a nice sequence of reads from one process (pending, only one is actually committed since it's serialized), and then a read further down the line that goes behind those you already committed. Then result is seeky, where it should have been sequential.
Quoted 7 lines
Quoted 3 lines
Do you get expected results if you set iodepth_low=1? That'll make fio drain the queue before building it up again, should get you a sequential access pattern with syslets.
With such a change results should be better - not only because seek is removed with sequential read, but also number of working threads decreases with time - until queue is filled again.
Yep, although it probably doesn't matter for such a low bandwidth test anyway.
Quoted 2 lines
So, syslet-rw has increased to 37mb/sec out of 39/sync and 38/libaio, the latter two did not changed.
I wonder why all three aren't doing 39mb/sec flat here, it's a pretty trivial case...
Quoted 3 lines
With iodepth of 10k, I get the same performance for
libaio and syslets - about 36mb/sec, it does not depend oniodepth_lowbeing set to 1 or default (full).
Yep, the larger the iodepth, the less costly a seek on new queue buildup gets. So that is as expected.
Quoted 3 lines
So syslets have small problems with small number of iodepth - its performance is about 34mb/sec and then increases to 36 with iodepth grow. While libaio decreases from 38 down to 36 mb/sec.
Using your job file and fio HEAD (forces iodepth_low=1 for syslet if iodepth_low isn't specified), I get:
Engine Depth Bw (kb/sec)
-----------------------------------
syslet 1 37163 syslet 32 37197 syslet 10000 36577 libaio 1 37144 libaio 32 37159 libaio 10000 36463 sync 1 37154
Results are highly stable. Note that this test case isn't totally fair, since libaio isn't really async when you do buffered file IO.
Signature · 1 line
Jens Axboe
Re: A quick fio test (was Re: [patch 00/13] Syslets, "Threadlets", generic AIO support, v3)
In reply to Jens AxboeQuoted 67 lines · Jens Axboe
Quoted 66 lines · Suparna Bhattacharya
Quoted 38 lines · Jens Axboe
Quoted 16 lines · Suparna Bhattacharya
Quoted 15 lines · Jens Axboe
Some more results, using a larger number of processes and io depths. A repeat of the tests from friday, with added depth 20000 for syslet and libaio:
Engine Depth Processes Bw (MiB/sec)
----------------------------------------------------libaio 1 1 602 syslet 1 1 759 sync 1 1 776 libaio 32 1 832 syslet 32 1 898 libaio 20000 1 581 syslet 20000 1 609Quoted 7 lines
Quoted 4 lines
syslet still on top. Measuring
O_DIRECTreads (of 4kb size) on ramfs with 100 processes each with a depth of 200, reading a per-process private file of 10mb (need to fit in my ram...) 10 times each. IOW, doing 10,000MiB of IO in total:But, why ramfs ? Don't we want to exercise the case where
O_DIRECTactually blocks ? Or am I missing something here ?Just overhead numbers for that test case, lets try something like your described job.
Test case is doing random reads from
/dev/sdb, in chunks of 64kb:Engine Depth Processes Bw (KiB/sec)
----------------------------------------------------libaio 200 100 2813 syslet 200 100 3944 libaio 20000 1 2793 syslet 20000 1 3854 sync (*) 20000 1 2866Quoted 7 lines
deadline was used for IO scheduling, to minimize impact. Not sure why syslet actually does so much better here, looing at vmstat the rate is steady and all runs are basically 50/50 idle/wait. One difference is that the submission itself takes a long time on libaio, since the
io_submit()will block on request allocation. The generated IO pattern from each process is the same for all runs. The drive is a lousy sata that doesn't even do queuing, FWIW.I tried the latest fio code with syslet v4, and my results are a little different - have yet to figure out why or what to make of it. I hope I have all the right pieces now.
This is an ext2 filesystem, SCSI AIC7xxx.
I used an
iodepth_batchsize of 8 to limit the number of ios in a singleio_submit(thanks for adding that parameter to fio !), like we did in aio-stress.Engine Depth Batch Bw (KiB/sec)
----------------------------------------------------
libaio 64 8 17,226
syslet 64 8 17,620
libaio 20000 8 18,552
syslet 20000 8 14,935
Quoted 6 lines
Quoted 1 line
Which is not bad, actually.
It's not bad for such a high depth/batch setting, but I still wonder why are results are so different. I'll look around for an x86 box with some TCQ/NCQ enabled storage attached for testing. Can you pass me your command line or job file (whatever you use) so we are on the same page?
Sure - I used variations of the following job file (e.g. engine=syslet-rw, iodepth=20000).
Also the io scheduler on my system is set to Anticipatory by default. FWIW it is a 4 way SMP (PIII, 700MHz)
; aio-stress -l -O -o3 <1GB file>
[global]
ioengine=libaio
buffered=0
rw=randread
bs=64k
size=1024m
directory=/kdump/suparna
[testfile2]
iodepth=64
iodepth_batch=8
Quoted 5 lines
Quoted 5 lines
If I do not specify the
iodepth_batch(i.e. default to depth), then the difference becomes more pronounced at higher depths. However, I doubt whether anyone would be using such high batch sizes in practice ...
Quoted 8 lines
Quoted 7 lines
Engine Depth Batch Bw (KiB/sec)
----------------------------------------------------
libaio 64 default 17,429
syslet 64 default 16,155
libaio 20000 default 15,494
syslet 20000 default 7,971If
iodepth_batchisn't set, the syslet queued io will be serialized and
I see, so then this particular setting is not very meaningful
Quoted 2 lines
not take advantage of queueing. How does the job file perform with ioengine=sync?
Just tried it now : 9,027KiB/s
Quoted 6 lines
Quoted 4 lines
Often times it is the application tuning that makes all the difference, so am not really sure how much to read into these results. That's always been the hard part of async io ...
Yes I agree, it's handy to get an overview though.
True, at least some of this helps us gain a little more understanding about the boundaries and how to tune it to be most effective.
Regards
Suparna
Quoted 3 lines
Signature · 2 lines
-- Jens Axboe
Signature · 3 lines
Suparna Bhattacharya ([email protected]) Linux Technology Center IBM Software Lab, India
Re: A quick fio test (was Re: [patch 00/13] Syslets, "Threadlets", generic AIO support, v3)
Quoted 20 lines · Suparna Bhattacharya
Quoted 4 lines
It's not bad for such a high depth/batch setting, but I still wonder why are results are so different. I'll look around for an x86 box with some TCQ/NCQ enabled storage attached for testing. Can you pass me your command line or job file (whatever you use) so we are on the same page?
Sure - I used variations of the following job file (e.g. engine=syslet-rw, iodepth=20000).
Also the io scheduler on my system is set to Anticipatory by default. FWIW it is a 4 way SMP (PIII, 700MHz)
; aio-stress -l -O -o3 <1GB file>
[global]
ioengine=libaio
buffered=0
rw=randread
bs=64k
size=1024m
directory=/kdump/suparna
Quoted 3 lines
[testfile2]
iodepth=64
iodepth_batch=8
Ok, now that I can run this on more than x86, I gave it a spin on a box with a little more potent storage. This is a core 2 quad, disks are 7200rpm sata (with NCQ) and a 15krpm SCSI disk. IO scheduler is deadline.
SATA disk:
Engine Depth Batch Bw (KiB/sec)
----------------------------------------------------
libaio 64 8 17,486
syslet 64 8 17,357
libaio 20000 8 17,625
syslet 20000 8 16,526
sync 1 1 7,529
SCSI disk:
Engine Depth Batch Bw (KiB/sec)
----------------------------------------------------
libaio 64 8 20,723
syslet 64 8 20,742
libaio 20000 8 21,125
syslet 20000 8 19,610
sync 1 1 16,659
Quoted 10 lines
Quoted 8 lines
Quoted 7 lines
Engine Depth Batch Bw (KiB/sec)
----------------------------------------------------
libaio 64 default 17,429
syslet 64 default 16,155
libaio 20000 default 15,494
syslet 20000 default 7,971If
iodepth_batchisn't set, the syslet queued io will be serialized andI see, so then this particular setting is not very meaningful
Not if you want to take advantage of hw queuing, as in this random workload. fio being a test tool, it's important to be able to control as many aspects of what happens as possible. That means you can also do things that you do not want to do in real life, having a pending list of 20000 serialized requests is indeed one of them. It also means you pretty much have to know what you are doing, when testing little details like this.
Signature · 1 line
Jens Axboe
Re: A quick fio test (was Re: [patch 00/13] Syslets, "Threadlets", generic AIO support, v3)
Quoted 3 lines · Jens Axboe
Engine Depth Batch Bw (KiB/sec)
libaio 20000 8 21,125
syslet 20000 8 19,610
i'd like to do something more about this to be more in line with libaio
- if nothing else then for the bragging rights ;-) It seems to me that a drop of ~7% in throughput cannot be explained with any CPU overhead, it must be some sort of queueing + IO scheduling effect - right?
Ingo
Re: A quick fio test (was Re: [patch 00/13] Syslets, "Threadlets", generic AIO support, v3)
Quoted 11 lines · Ingo Molnar
Quoted 3 lines · Jens Axboe
Engine Depth Batch Bw (KiB/sec)
libaio 20000 8 21,125
syslet 20000 8 19,610i'd like to do something more about this to be more in line with libaio
- if nothing else then for the bragging rights ;-) It seems to me that a drop of ~7% in throughput cannot be explained with any CPU overhead, it must be some sort of queueing + IO scheduling effect - right?
syslet shows a slightly higher overhead, but nothing that will account for any bandwidth change in this test. The box is obviously mostly idle when running this test, it's not very CPU consuming. The IO pattern issued is not the same, since libaio would commit IO [0..7], then [8..15] and so on, where syslet would expose [0,8,16,24,32,40,48,56] and then [1,9,17,25,33,41,49,57] etc. If iodepth_batch is set to 1 you'd get a closer match wrt io pattern, but at a higher cost (increased system calls, and 8 times as many pending async threads). That gets it to 20,253KiB/s here with ~1000 as many context switches.
So in short, it's harder to compare with real storage, as access patterns don't translate very easily
Signature · 1 line
Jens Axboe
Re: A quick fio test (was Re: [patch 00/13] Syslets, "Threadlets", generic AIO support, v3)
In reply to Suparna BhattacharyaQuoted 24 lines · Suparna Bhattacharya
I tried the latest fio code with syslet v4, and my results are a little different - have yet to figure out why or what to make of it. I hope I have all the right pieces now.
This is an ext2 filesystem, SCSI AIC7xxx.
I used an
iodepth_batchsize of 8 to limit the number of ios in a singleio_submit(thanks for adding that parameter to fio !), like we did in aio-stress.Engine Depth Batch Bw (KiB/sec)
----------------------------------------------------
libaio 64 8 17,226
syslet 64 8 17,620
libaio 20000 8 18,552
syslet 20000 8 14,935Which is not bad, actually.
If I do not specify the
iodepth_batch(i.e. default to depth), then the difference becomes more pronounced at higher depths. However, I doubt whether anyone would be using such high batch sizes in practice ...
Quoted 7 lines
Engine Depth Batch Bw (KiB/sec)
----------------------------------------------------
libaio 64 default 17,429
syslet 64 default 16,155
libaio 20000 default 15,494
syslet 20000 default 7,971
But what about cpu usage? At these low levels, the cpu is probably underutilized. It would be interesting to measure cpu time per I/O request (or, alternatively, use an I/O subsystem that can saturate the processors).
Signature · 1 line
error compiling committee.c: too many arguments to function
Re: A quick fio test (was Re: [patch 00/13] Syslets, "Threadlets", generic AIO support, v3)
Quoted 4 lines · Avi Kivity
But what about cpu usage? At these low levels, the cpu is probably underutilized. It would be interesting to measure cpu time per I/O request (or, alternatively, use an I/O subsystem that can saturate the processors).
yeah - that's what testing on ramdisk (Jens') or on a loopback block device (mine) approximates to a certain degree.
Ingo
Re: A quick fio test (was Re: [patch 00/13] Syslets, "Threadlets", generic AIO support, v3)
Quoted 9 lines · Ingo Molnar
Quoted 2 lines · Avi Kivity
But what about cpu usage? At these low levels, the cpu is probably underutilized. It would be interesting to measure cpu time per I/O request (or, alternatively, use an I/O subsystem that can saturate the processors).
yeah - that's what testing on ramdisk (Jens') or on a loopback block device (mine) approximates to a certain degree.
Ramdisks or fully cached loopback return immediately, so cache thrashing effects don't show up.
Maybe a device mapper delay target or nbd + O_DIRECT can insert delays to make the workload more disk-like.
Signature · 1 line
error compiling committee.c: too many arguments to function
Re: A quick fio test (was Re: [patch 00/13] Syslets, "Threadlets", generic AIO support, v3)
Quoted 5 lines · Avi Kivity
Quoted 2 lines
yeah - that's what testing on ramdisk (Jens') or on a loopback block device (mine) approximates to a certain degree.
Ramdisks or fully cached loopback return immediately, so cache thrashing effects don't show up.
even fully cached loopback schedules the loopback kernel thread - but i agree that it's inaccurate: hence the 'approximates to a certain degree'.
Quoted 2 lines
Maybe a device mapper delay target or nbd +
O_DIRECTcan insert delays to make the workload more disk-like.
yeah. I'll hack a small timeout into loopback requests i think. But then real disk-platter effects are left out ... so it all comes down to eventually having to try it on real disks too :)
Ingo
Re: A quick fio test (was Re: [patch 00/13] Syslets, "Threadlets", generic AIO support, v3)
Quoted 6 lines · Ingo Molnar
Quoted 3 lines
Maybe a device mapper delay target or nbd +
O_DIRECTcan insert delays to make the workload more disk-like.yeah. I'll hack a small timeout into loopback requests i think. But then real disk-platter effects are left out ... so it all comes down to eventually having to try it on real disks too :)
Having a random component in the timeout may increase realism.
hundred-disk boxes are noisy, though the blinkenlights are nice :)
Signature · 1 line
error compiling committee.c: too many arguments to function
Re: A quick fio test (was Re: [patch 00/13] Syslets, "Threadlets", generic AIO support, v3)
In reply to Avi KivityQuoted 15 lines · Avi Kivity
Quoted 13 lines · Ingo Molnar
Quoted 5 lines · Avi Kivity
But what about cpu usage? At these low levels, the cpu is probably underutilized. It would be interesting to measure cpu time per I/O request (or, alternatively, use an I/O subsystem that can saturate the processors).
yeah - that's what testing on ramdisk (Jens') or on a loopback block device (mine) approximates to a certain degree.
Quoted 5 lines
Ramdisks or fully cached loopback return immediately, so cache thrashing effects don't show up.
Maybe a device mapper delay target or nbd +
O_DIRECTcan insert delays to make the workload more disk-like.
Take a look at scsi-debug, it can do at least some of that.
Signature · 1 line
Jens Axboe
Re: A quick fio test (was Re: [patch 00/13] Syslets, "Threadlets", generic AIO support, v3)
In reply to Jens AxboeQuoted 7 lines · Jens Axboe
Some more results, using a larger number of processes and io depths. A repeat of the tests from friday, with added depth 20000 for syslet and libaio:
Engine Depth Processes Bw (MiB/sec)
----------------------------------------------------
Quoted 7 lines
libaio 1 1 602 syslet 1 1 759 sync 1 1 776 libaio 32 1 832 syslet 32 1 898 libaio 20000 1 581 syslet 20000 1 609
That looks great! IMO there may be a little higher cost associated with the syslets thread switches, that we currently do not perform 100% cleanly, but results look nevertheless awesome.
- Davide
Re: A quick fio test (was Re: [patch 00/13] Syslets, "Threadlets", generic AIO support, v3)
In reply to Suparna BhattacharyaQuoted 8 lines · Suparna Bhattacharya
Quoted 2 lines
my expectation is that it should behave better with iodepth=20000 (although i havent tried that yet).
I picked up the fio snapshot from 22nd Feb (fio-git-20070222212513.tar.gz) and used the v3 syslet patches from your web-site.
Do I still need to get something more recent ?
yeah, there's something more recent. Please do this:
yum install git
git-clone git://git.kernel.dk/data/git/fio.git
cd fio
git-branch syslet-testing
git-checkout
this should give you the latest version of the v3 based FIO code. It's one generation newer than the one you tried. I mean the snapshot you used is meanwhile a /whole/ day old, so it's truly ancient stuff! ;-)
Ingo
Re: A quick fio test (was Re: [patch 00/13] Syslets, "Threadlets", generic AIO support, v3)
In reply to Jens AxboeQuoted 9 lines · Jens Axboe
Results:
Engine Depth Bw (MiB/sec)
--------------------------------------------libaio 1 441 syslet 1 574 sync 1 589 libaio 32 613 syslet 32 681
Can we get runs with large I/Os, large I/O depths, and most
importantly tons of processes? I can absolutely believe that syslets would compete well with one process on the system. But with 1000 processes doing 1000s of blocking I/Os, I'd really be interested to see how that plays out.
Joel
Signature · 9 lines
Joel's Second Law: If a code change requires additional user setup, it is wrong. Joel Becker Principal Software Developer Oracle E-mail: [email protected] Phone: (650) 506-8127
Re: A quick fio test (was Re: [patch 00/13] Syslets, "Threadlets", generic AIO support, v3)
Quoted 11 lines · Joel Becker
Quoted 9 lines · Jens Axboe
Results:
Engine Depth Bw (MiB/sec)
--------------------------------------------libaio 1 441 syslet 1 574 sync 1 589 libaio 32 613 syslet 32 681
Quoted 5 lines
Can we get runs with large I/Os, large I/O depths, and most
importantly tons of processes? I can absolutely believe that syslets would compete well with one process on the system. But with 1000 processes doing 1000s of blocking I/Os, I'd really be interested to see how that plays out.
Sure, I'll add this to the testing list for monday.
Signature · 1 line
Jens Axboe
PATCH 1/1 Syslets/threadlets, generic AIO support, v4
In reply to Ingo Molnarthis is the v4 release of the syslet/threadlet subsystem:
http://redhat.com/~mingo/syslet-patches/
v4 is a smaller update than v3 (so i wont send out the full queue to lkml - see the broken out queue in the patches-v4 directory at the URL above). Changes since v3:
- the threadlet API changed: the sys_async_threadlet() syscall now takes a 'completion event' pointer, and auto-completes it into the completion ring. I've updated the test-threadlet.c code to make use of it. So completion of threadlets and syslets is quite similar now - sharing even more infrastructure. (To get true pthread compatibility a sys_exit() driven CLEARTID completion method will be added too in the future).
- a small performance fix for syslet rescheduling. The syslet ABI has not changed.
- test-threadlet.c fixes a thread stack leak, and the other tests too have a number of small fixes and cleanups.
Ingo
Threadlet/syslet v4 bug report.
In reply to Ingo MolnarHi Ingo.
I tried to create web server application for threadlets to show (will not write what I wanted to show, but you might guess :)
I started threadlet-test from async-test-v4 and got following bug.
To compile v4 and v3 I need to apply a patch (sent in previous e-mail), which removes %xgs assignment since it is never used and actually does not even exist it pt_regs.
It is 2.6.20-rc1 tree (patch failed in Makefile (no need to export your kernel extraversion) and in asm/i386/kernel/process (i387 initialization)), both are trivially fixable by hands.
Machine is via epia, fc5 distro.
I'm currently compiling tree fwith debug support to decode where bug lives if it will be reproduced.
[ 894.597934] general protection fault: 0000 [#1] [ 894.597959] PREEMPT SMP [ 894.597975] Modules linked in: dm_mod nvram ehci_hcd uhci_hcd
snd_via82xx snd_ac97_codec ac97_bus snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_pcm_oss snd_mixer_oss snd_pcm snd_timer snd_page_alloc snd_mpu401_uart snd_rawmidi snd_seq_device snd generic soundcore via_rhine i2c_viapro i2c_core mii ext3 jbd ide_disk
[ 894.598131] CPU: 0 [ 894.598135] EIP: 0060:[<c012ab99>] Not tainted VLI [ 894.598141] EFLAGS: 00010282 (2.6.21-rc1 #2) [ 894.598179] EIP is at cachemiss_thread+0xa/0xc6 [ 894.598200] eax: cdfa8ef4 ebx: cdfa8ef4 ecx: cdfa8ef4 edx: cdfa8ef4 [ 894.598222] esi: 00000000 edi: 00000000 ebp: 00000000 esp: ca6aff94 [ 894.598245] ds: 007b es: 007b fs: 0000 gs: 0033 ss: 0068 [ 894.598268] Process threadlet-test (pid: 12030, ti=ca6ae000 task=cdf9a790 task.ti=ca6ae000) [ 894.598290] Stack: cd28a080 cdfa8db8 c0114536 cdfa8c10 c012ab8f 00000000 00000000 c0103af8 [ 894.598339] cdfa8ef4 c012ab8f 00000000 cdfa8ef4 00000000 00000000 00000000 00000000 [ 894.598385] 0000007b 0000007b 00000000 ffffffff c0103af0 00000060 00000286 00000000 [ 894.598437] Call Trace: [ 894.598463] [<c0114536>] schedule_tail+0x2d/0x80 [ 894.598497] [<c012ab8f>] cachemiss_thread+0x0/0xc6 [ 894.598526] [<c0103af8>] async_thread_helper+0x8/0x1c [ 894.598561] [<c012ab8f>] cachemiss_thread+0x0/0xc6 [ 894.598598] [<c0103af0>] async_thread_helper+0x0/0x1c [ 894.598631] ======================= [ 894.598648] Code: b8 1f 00 00 89 42 38 8b 53 04 8b 46 0c 81 c2 b8 1f 00 00 89 42 2c 83 c8 ff 83 c4 10 5b 5e 5f 5d c3 57 89 c1 56 53 89 c3 83 ec 10 <64> 8b 35 08 00 00 00 8d be c4 02 00 00 89 f0 89 fa e8 ab f8 ff [ 894.598875] EIP: [<c012ab99>] cachemiss_thread+0xa/0xc6 SS:ESP 0068:ca6aff94
Signature · 1 line
Evgeniy Polyakov
Re: Threadlet/syslet v4 bug report.
Quoted 10 lines · Evgeniy Polyakov
Hi Ingo.
I tried to create web server application for threadlets to show (will not write what I wanted to show, but you might guess :)
I started threadlet-test from async-test-v4 and got following bug.
To compile v4 and v3 I need to apply a patch (sent in previous e-mail), which removes %xgs assignment since it is never used and actually does not even exist it
pt_regs.
If I uncomment xgs and recompile - kernel crashes early on init...
Btw, could you create a git tree to pull from?
That would be much easier to track problems and to test for others.
Signature · 1 line
Evgeniy Polyakov
Quoted 3 lines · Ingo Molnar
this is the v3 release of the syslet/threadlet subsystem:
There is no %xgs.
arch/i386/kernel/process.c renamed from arch/i386/kernel/process.c~ +0 -1
@@ -426,7 +426,6 @@ regs.xds = __USER_DS; regs.xes = __USER_DS; - regs.xgs = __KERNEL_PDA; regs.orig_eax =-1 ; regs.eip = (unsigned long ) async_thread_helper; regs.xcs = __KERNEL_CS |get_kernel_rpl ();
Signature · 1 line
Evgeniy Polyakov
Quoted 10 lines · Evgeniy Polyakov
Quoted 3 lines · Ingo Molnar
this is the v3 release of the syslet/threadlet subsystem:
There is no %xgs.
arch/i386/kernel/process.c line 429
Quoted 4 lines
arch/i386/kernel/process.c renamed from arch/i386/kernel/process.c~ +0
-1@@ -426,7 +426,6 @@ regs.xds = __USER_DS; regs.xes = __USER_DS; - regs.xgs = __KERNEL_PDA;
hm, what tree are you using as a base? The syslet patches are against v2.6.20 at the moment. (the x86 PDA changes will probably interfere with it on v2.6.21-rc1-ish kernels) Note that otherwise the syslet/threadlet patches are for x86 only at the moment (as i mentioned in the announcement), and the generic code itself contains some occasional x86-ishms as well. (None of the concepts are x86-specific though - multi-stack architectures should work just as well as RISC-ish CPUs.)
if you create a threadlet based test-webserver, could you please do a comparable kevents implementation as well? I.e. same HTTP parser (or non-parser, as usually the case is with prototypes ;). Best would be something that one could trigger between threadlet and kevent mode, using the same binary :-)
Ingo
Quoted 14 lines · Ingo Molnar
Quoted 11 lines · Evgeniy Polyakov
Quoted 3 lines · Ingo Molnar
this is the v3 release of the syslet/threadlet subsystem:
There is no %xgs.
arch/i386/kernel/process.c renamed from arch/i386/kernel/process.c~ +0
-0@@ -426,7 +426,6 @@
Quoted 11 lines
arch/i386/kernel/process.c line 429
Quoted 3 lines
arch/i386/kernel/process.c renamed from arch/i386/kernel/process.c~ +0
-1regs.xds = __USER_DS; regs.xes = __USER_DS; - regs.xgs = __KERNEL_PDA;hm, what tree are you using as a base? The syslet patches are against v2.6.20 at the moment. (the x86 PDA changes will probably interfere with it on v2.6.21-rc1-ish kernels) Note that otherwise the syslet/threadlet patches are for x86 only at the moment (as i mentioned in the announcement), and the generic code itself contains some occasional x86-ishms as well. (None of the concepts are x86-specific though - multi-stack architectures should work just as well as RISC-ish CPUs.)
It is rc1 - and crashes.
I test on i386 via epia (the only machine which runs x86 right now).
If there will not be any new patches, I will create 2.6.20 test tree tomorrow.
Quoted 5 lines
if you create a threadlet based test-webserver, could you please do a comparable kevents implementation as well? I.e. same HTTP parser (or non-parser, as usually the case is with prototypes ;). Best would be something that one could trigger between threadlet and kevent mode, using the same binary :-)
Ok, I will create such a monster tomorrow :)
I will use the same base for threadlet as for kevent/epoll - there is no parser, just sendfile() of the static file which contains http header and actual page.
threadlet1 {
accept()
create threadlet2 {
send data
}
}Is above scheme correct for threadlet scenario?
But note, that on my athlon64 3500 test machine kevent is about 7900 requests per second compared to 4000+ epoll, so expect a challenge. lighhtpd is about the same 4000 requests per second though, since it can not be easily optimized for kevents.
Quoted 1 line
Ingo
Signature · 1 line
Evgeniy Polyakov
Quoted 10 lines · Evgeniy Polyakov
Quoted 8 lines
hm, what tree are you using as a base? The syslet patches are against v2.6.20 at the moment. (the x86 PDA changes will probably interfere with it on v2.6.21-rc1-ish kernels) Note that otherwise the syslet/threadlet patches are for x86 only at the moment (as i mentioned in the announcement), and the generic code itself contains some occasional x86-ishms as well. (None of the concepts are x86-specific though - multi-stack architectures should work just as well as RISC-ish CPUs.)
It is rc1 - and crashes.
yeah. I'm not surprised. The PDA is not set up in create_async_thread() for example.
Quoted 12 lines
Quoted 5 lines
if you create a threadlet based test-webserver, could you please do a comparable kevents implementation as well? I.e. same HTTP parser (or non-parser, as usually the case is with prototypes ;). Best would be something that one could trigger between threadlet and kevent mode, using the same binary :-)
Ok, I will create such a monster tomorrow :)
I will use the same base for threadlet as for kevent/epoll - there is no parser, just
sendfile()of the static file which contains http header and actual page.
Quoted 8 lines
threadlet1 { accept() create threadlet2 { send data } }Is above scheme correct for threadlet scenario?
yep, this is a good first cut. Doing this after the listen() is useful:
int one = 1;
ret = setsockopt(listen_sock_fd, SOL_SOCKET, SO_REUSEADDR,
(char *)&one, sizeof(int));and i'd suggest to do this after every accept()-ed socket:
int flag = 1;
setsockopt(sock_fd, IPPROTO_TCP, TCP_NODELAY,
(char *) &flag, sizeof(int));Do you have any link where i could check the type of HTTP parsing and send transport you are (or will be) using? What type of http client are you using to measure, with precisely what options?
Quoted 2 lines
But note, that on my athlon64 3500 test machine kevent is about 7900 requests per second compared to 4000+ epoll, so expect a challenge.
single-core CPU i suspect?
Quoted 2 lines
lighhtpd is about the same 4000 requests per second though, since it can not be easily optimized for kevents.
mean question: do you promise to post the results even if they are not unfavorable to threadlets? ;-)
if i want to test kevents on a v2.6.20 kernel base, do you have an URL for me to try?
Ingo
Quoted 12 lines · Ingo Molnar
Quoted 9 lines · Evgeniy Polyakov
Quoted 8 lines
hm, what tree are you using as a base? The syslet patches are against v2.6.20 at the moment. (the x86 PDA changes will probably interfere with it on v2.6.21-rc1-ish kernels) Note that otherwise the syslet/threadlet patches are for x86 only at the moment (as i mentioned in the announcement), and the generic code itself contains some occasional x86-ishms as well. (None of the concepts are x86-specific though - multi-stack architectures should work just as well as RISC-ish CPUs.)
Quoted 4 lines
Quoted 1 line
It is rc1 - and crashes.
yeah. I'm not surprised. The PDA is not set up in
create_async_thread()for example.
Ok, I will roll back to vanilla 2.6.20 tomorrow.
Quoted 30 lines
Quoted 12 lines
Quoted 5 lines
if you create a threadlet based test-webserver, could you please do a comparable kevents implementation as well? I.e. same HTTP parser (or non-parser, as usually the case is with prototypes ;). Best would be something that one could trigger between threadlet and kevent mode, using the same binary :-)
Ok, I will create such a monster tomorrow :)
I will use the same base for threadlet as for kevent/epoll - there is no parser, just
sendfile()of the static file which contains http header and actual page.Quoted 8 lines
threadlet1 { accept() create threadlet2 { send data } }Is above scheme correct for threadlet scenario?
yep, this is a good first cut. Doing this after the
listen()is useful:int one = 1; ret = setsockopt(listen_sock_fd, SOL_SOCKET, SO_REUSEADDR, (char *)&one, sizeof(int));and i'd suggest to do this after every accept()-ed socket:
Quoted 8 lines
int flag = 1; setsockopt(sock_fd, IPPROTO_TCP, TCP_NODELAY, (char *) &flag, sizeof(int));Do you have any link where i could check the type of HTTP parsing and send transport you are (or will be) using? What type of http client are you using to measure, with precisely what options?
For example this ones (essentially the same, except that epoll and kevent are used):
http://tservice.net.ru/~s0mbre/archive/kevent/evserver_kevent.c
http://tservice.net.ru/~s0mbre/archive/kevent/evserver_epoll.c
Quoted 4 lines
Quoted 2 lines
But note, that on my athlon64 3500 test machine kevent is about 7900 requests per second compared to 4000+ epoll, so expect a challenge.
single-core CPU i suspect?
Yep.
Quoted 5 lines
Quoted 2 lines
lighhtpd is about the same 4000 requests per second though, since it can not be easily optimized for kevents.
mean question: do you promise to post the results even if they are not unfavorable to threadlets? ;-)
If they are too good, I will start searching for bugs and tune my code first, but eventually of course yes.
In my blog I will post them in 'real-time' even if kevent will unbelieveably suck.
Quoted 2 lines
if i want to test kevents on a v2.6.20 kernel base, do you have an URL for me to try?
I have a git tree at (based on rc1 as requested by Andrew Morton):
http://tservice.net.ru/~s0mbre/archive/kevent/kevent.git/
Or patches at kevent homepage:
http://tservice.net.ru/~s0mbre/old/?section=projects&item=kevent
Direct link to the latest patchset:
http://tservice.net.ru/~s0mbre/archive/kevent/kevent-37/
(order is insignificant as far as I recall, except 'compile-fix', whcih must be the latest).
Quoted 1 line
Ingo
Signature · 1 line
Evgeniy Polyakov
Quoted 8 lines · Evgeniy Polyakov
Quoted 3 lines
Do you have any link where i could check the type of HTTP parsing and send transport you are (or will be) using? What type of http client are you using to measure, with precisely what options?
For example this ones (essentially the same, except that epoll and kevent are used):
http://tservice.net.ru/~s0mbre/archive/kevent/evserver_kevent.c
http://tservice.net.ru/~s0mbre/archive/kevent/evserver_epoll.c
thx - i guess i should just run them without any options and they bind themselves to port 80? What 'ab' options are you using typically to measure them?
Ingo
Quoted 4 lines · Ingo Molnar
Quoted 4 lines
Quoted 3 lines
Do you have any link where i could check the type of HTTP parsing and send transport you are (or will be) using? What type of http client are you using to measure, with precisely what options?
Quoted 8 lines
Quoted 4 lines
For example this ones (essentially the same, except that epoll and kevent are used):
http://tservice.net.ru/~s0mbre/archive/kevent/evserver_kevent.c
http://tservice.net.ru/~s0mbre/archive/kevent/evserver_epoll.cthx - i guess i should just run them without any options and they bind themselves to port 80? What 'ab' options are you using typically to measure them?
Yes, but they require /tmp/index.html to have http header and actual data page. They do not parse http request :)
For athlon 3500 I used
ab -c8000 -n80000 $url
for via epia likely two/three times less.
Quoted 1 line
Ingo
Signature · 1 line
Evgeniy Polyakov
Quoted 6 lines · Evgeniy Polyakov
Quoted 3 lines
thx - i guess i should just run them without any options and they bind themselves to port 80? What 'ab' options are you using typically to measure them?
Yes, but they require
/tmp/index.htmlto have http header and actual data page. They do not parse http request :)
ok. When i connect to the epoll server via "telnet mysever 80", and enter a 'request', i get back the content - but the socket connection is not closed. Every time i type enter i get a new content back. Why is that so - the code seems to contain a close(fd).
Ingo
Quoted 7 lines · Ingo Molnar
Quoted 4 lines · Evgeniy Polyakov
Quoted 3 lines
thx - i guess i should just run them without any options and they bind themselves to port 80? What 'ab' options are you using typically to measure them?
Quoted 8 lines
Quoted 2 lines
Yes, but they require
/tmp/index.htmlto have http header and actual data page. They do not parse http request :)ok. When i connect to the epoll server via "telnet mysever 80", and enter a 'request', i get back the content - but the socket connection is not closed. Every time i type enter i get a new content back. Why is that so - the code seems to contain a close(fd).
I'd say a close(s); is missing just before return 0; in evtest_callback_client() ?
Regards,
Frederik
Quoted 9 lines · Evgeniy Polyakov
Quoted 3 lines
thx - i guess i should just run them without any options and they bind themselves to port 80? What 'ab' options are you using typically to measure them?
Yes, but they require
/tmp/index.htmlto have http header and actual data page. They do not parse http request :)For athlon 3500 I used
ab -c8000 -n80000 $url
how do the header portions of your /tmp/index.html data page look like?
Ingo
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
threadlets as 'naive pool of threads', epoll, some measurements
Quoted 7 lines · Davide Libenzi
Quoted 5 lines
i dont understand - this confuses the client because there's no Content-Length field. Did you insert a Content-Length field manually? What i'm trying to figure out, are you relying on a keepalive client or not? I.e. is there a -k option to 'ab' as well, besides the ones you mentioned?
You don't need a Content-Length if you're closing the connection ;)
yeah. But there's apparently a bug in the code because it doesnt close 's'.
Quoted 3 lines
In any case, Evgeniy test "servers" so not handle properly the delivery of the content, where it is assumed that an open+sendfile is a continuos non-blocking operation. [...]
i have tried the one Evgeniy provided in the URL:
http://tservice.net.ru/~s0mbre/archive/kevent/evserver_epoll.c
and 'ab -k -c8000 -n80000' almost always aborts with:
apr_socket_recv: Connection reset by peer (104)
in the few cases it finishes, i got the following epoll result, over gigabit ethernet, on an UP Athlon64 box:
eserver_epoll: 7800 reqs/sec
the same with the most naive implementation of the same, using threadlets:
eserver_threadlet: 5800 reqs/sec
Extrapolating from Evgeniy's numbers i'd expect kevents to do around 14000 reqs/sec.
i've attached eserver_threadlet.c [compile it in the async-test/ directory of the syslet userspace testcode, and change MAX_PENDING to 9000 in sys.h] - it uses the request handling function from eserver_epoll - no tuning or anything.
while keeping in mind that these are raw, preliminary numbers with little analysis of them, i'm surprised that 8000 async threads perform so well. Obviously this testcase and implementation just degrades threadlets down to a pool of 8000 user-space threads, so there's no win at all from context caching, no intelligent queueing back to the head context, etc.
Ingo
---------{ eserver_threadlet.c }--------------->
#include <unistd.h>
#include <stdio.h>
#include <stdlib.h>
#include <signal.h>
#include <sys/wait.h>
#include <linux/futex.h>
#include <sys/time.h>
#include <sys/types.h>
#include <linux/unistd.h>
#include <netinet/tcp.h>
#include <stdio.h>
#include <netinet/in.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <sched.h>
#include <sys/time.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <arpa/inet.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <errno.h>
#include <sys/poll.h>
#include <sys/sendfile.h>
#include <sys/epoll.h>
#define DEBUG 0
#include "syslet.h"
#include "sys.h"
#include "threadlet.h"
struct request {
struct request *next_free;
/*
* The threadlet stack is part of the request structure
* and is thus reused as threadlets complete:
*/
unsigned long threadlet_stack;
/*
* These are all the request-specific parameters:
*/
long sock;
};//#define ulog(f, a...) fprintf(stderr, f, ##a)
#define ulog(f, a...)
#define ulog_err(f, a...) printf(f ": %s [%d].\n", ##a, strerror(errno), errno)
static long handle_request(void *__req)
{
struct request *req = __req;
int s = req->sock, err, fd;
off_t offset;
int count;
char path[] = "/tmp/index.html";
char buf[4096];
struct timeval tm;read_again:
count = 40960;
offset = 0;
err = recv(s, buf, sizeof(buf), 0);
if (err < 0) {
ulog_err("Failed to read data from s=%d", s);
goto err_out_remove;
}
if (err == 0) {
gettimeofday(&tm, NULL);
ulog("%08lu:%06lu: Client exited: fd=%d.\n", tm.tv_sec, tm.tv_usec, s);
goto err_out_remove;
}
fd = open(path, O_RDONLY);
if (fd == -1) {
ulog_err("Failed to open '%s'", path);
err = -1;
goto err_out_remove;
}
#if 0
do {
err = read(fd, buf, sizeof(buf));
if (err <= 0)
break;
err = send(s, buf, err, 0);
if (err <= 0)
break;
} while (1);
#endif
err = sendfile(s, fd, &offset, count);
{
int on = 0;
setsockopt(s, SOL_TCP, TCP_CORK, &on, sizeof(on));
}
close(fd);
if (err < 0) {
ulog_err("Failed send %d bytes: fd=%d.\n", count, s);
goto err_out_remove;
}
gettimeofday(&tm, NULL);
ulog("%08lu:%06lu: %d bytes has been sent to client fd=%d.\n", tm.tv_sec, tm.tv_usec, err, s);
goto read_again;err_out_remove:
printf("blah\n");
close(s);
return complete_threadlet_fn(req, &async_head);
}/*
* Freelist to recycle requests:
*/
static struct request *freelist;
/*
* Allocate a request and set up its syslet atoms:
*/
static struct request *alloc_req(void)
{
struct request *req;
/*
* Occasionally we have to refill the new-thread stack
* entry:
*/
if (!async_head.new_thread_stack) {
async_head.new_thread_stack = thread_stack_alloc();
pr("allocated new thread stack: %08lx\n",
async_head.new_thread_stack);
}
if (freelist) {
req = freelist;
pr("reusing req %p, threadlet stack %08lx\n",
req, req->threadlet_stack);
freelist = freelist->next_free;
req->next_free = NULL;
return req;
}
req = calloc(1, sizeof(struct request));
pr("allocated req %p\n", req);
req->threadlet_stack = thread_stack_alloc();
pr("allocated thread stack %08lx\n", req->threadlet_stack);
return req;
}/*
* Check whether there are any completions queued for user-space
* to finish up:
*/
static unsigned long complete(void)
{
unsigned long completed = 0;
struct request *req;
for (;;) {
req = (void *)completion_ring[async_head.user_ring_idx];
if (!req)
return completed;
completed++;
pr("completed req %p (threadlet stack %08lx)\n",
req, req->threadlet_stack);
req->next_free = freelist;
freelist = req;
/*
* Clear the completion pointer. To make sure the
* kernel never stomps upon still unhandled completions
* in the ring the kernel only writes to a NULL entry,
* so user-space has to clear it explicitly:
*/
completion_ring[async_head.user_ring_idx] = NULL;
async_head.user_ring_idx++;
if (async_head.user_ring_idx == MAX_PENDING)
async_head.user_ring_idx = 0;
}
}static unsigned int pending_requests;
/*
* Handle a request that has just been submitted (either it has
* already been executed, or we have to account it as pending):
*/
static void handle_submitted_request(struct request *req, long done)
{
unsigned int nr;
if (done) {
/*
* This is the cached case - free the request:
*/
pr("cache completed req %p (threadlet stack %08lx)\n",
req, req->threadlet_stack);
req->next_free = freelist;
freelist = req;
return;
}
/*
* 'cachemiss' case - the syslet is not finished
* yet. We will be notified about its completion
* via the completion ring:
*/
assert(pending_requests < MAX_PENDING-1);
pending_requests++;
pr("req %p is pending. %d reqs pending.\n", req, pending_requests);
/*
* Attempt to complete requests - this is a fast
* check if there's no completions:
*/
nr = complete();
pending_requests -= nr;
/*
* If the ring is full then wait a bit:
*/
while (pending_requests == MAX_PENDING-1) {
pr("sys_async_wait()");
/*
* Wait for 4 events - to batch things a bit:
*/
sys_async_wait(4, async_head.user_ring_idx, &async_head);
nr = complete();
pending_requests -= nr;
pr("after wait: completed %d requests - still pending: %d\n",
nr, pending_requests);
}
}static void webserver(int l_sock)
{
struct sockaddr addr;
struct request *req;
socklen_t addrlen;
long done;
int sock;
async_head_init();
/*
* We dont use syslets for the first request (or for oversized
* and/or erroneous requests):
*/
for (;;) {
req = alloc_req();
if (!req)
break;
sock = accept(l_sock, &addr, &addrlen);
if (sock < 0)
break;
req->sock = sock;
done = threadlet_exec(handle_request, req,
req->threadlet_stack, &async_head);
handle_submitted_request(req, done);
}
}static int tcp_listen_socket(int *l_socket)
{
int ret;
int one = 1;
struct sockaddr_in ad;
/*
* Setup for server socket
*/
memset(&ad, 0, sizeof(ad));
ad.sin_family = AF_INET;
ad.sin_addr.s_addr = htonl(INADDR_ANY);
ad.sin_port = htons(2222);
*l_socket = socket(PF_INET, SOCK_STREAM, IPPROTO_TCP);
if (*l_socket < 0) {
perror("socket (server)");
return *l_socket;
}
ret = setsockopt(*l_socket, SOL_SOCKET, SO_REUSEADDR, (char *)&one,
sizeof(int));
if (ret < 0) {
perror("setsockopt");
return ret;
}
ret = bind(*l_socket, (struct sockaddr *)&ad, sizeof(ad));
if (ret < 0) {
perror("bind");
return ret;
}
ret = listen(*l_socket, 1000);
if (ret < 0) {
perror("listen");
return ret;
}
return 0;
}int main(int argc, char *argv[])
{
int ret;
int l_sock;
struct sched_param p = { sched_priority: 0 };
sched_setscheduler(getpid(), 3 /* SCHED_BATCH */, &p);
ret = tcp_listen_socket(&l_sock);
if (ret)
exit(ret);
printf("listening on port 2222\n");
printf("using threadlets\n");
webserver(l_sock);
async_head_exit();
exit(0);
}Re: threadlets as 'naive pool of threads', epoll, some measurements
update:
Quoted 4 lines
i have tried the one Evgeniy provided in the URL:
http://tservice.net.ru/~s0mbre/archive/kevent/evserver_epoll.c
Quoted 8 lines
and 'ab -k -c8000 -n80000' almost always aborts with:
apr_socket_recv: Connection reset by peer (104)in the few cases it finishes, i got the following epoll result, over gigabit ethernet, on an UP Athlon64 box:
eserver_epoll: 7800 reqs/sec
eserver_epoll.c had a number of bugs. The most serious one was the apparently buggy use of "EPOLLET" (edge-triggered events). Removing that and moving epoll to level-triggered (which is slower but does not result in missed events) gives:
eserver_epoll: 9400 reqs/sec
Quoted 4 lines
the same with the most naive implementation of the same, using threadlets:
eserver_threadlet: 5800 reqs/sec
eserver_epoll_threadlet: 9400 reqs/sec
as expected, the level of extra blocking triggered by this is low - even if the full request function runs without nonblock assumptions.
Ingo
Re: threadlets as 'naive pool of threads', epoll, some measurements
yet another performance update - with the fixed 'heaps of stupid threads' evserver_threadlet.c code attached below i got:
Quoted 2 lines
evserver_epoll: 9400 reqs/sec
evserver_epoll_threadlet: 9400 reqs/sec
evserver_threadlet: 9000 reqs/sec
so the overhead, instead of the 10x slowdown Evgeniy predicted/feared, is 4% for this particular, very event-centric workload.
why? because Evgeniy still overlooks what i've mentioned so many times:
that there is lots of inherent 'caching' possible even in this particular '8000 clients' workload, which even the most stupid threadlet queueing model is able to take advantage of. The maximum level of parallelism that i've measured during this test was 161 threads.
Ingo
-----------{ evserver_threadlet.c }---------------> #include <unistd.h>
#include <stdio.h>
#include <stdlib.h>
#include <signal.h>
#include <sys/wait.h>
#include <linux/futex.h>
#include <sys/time.h>
#include <sys/types.h>
#include <linux/unistd.h>
#include <netinet/tcp.h>
#include <stdio.h>
#include <netinet/in.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <sched.h>
#include <sys/time.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <arpa/inet.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <errno.h>
#include <sys/poll.h>
#include <sys/sendfile.h>
#include <sys/epoll.h>
#define DEBUG 0
#include "syslet.h"
#include "sys.h"
#include "threadlet.h"
struct request {
struct request *next_free;
/*
* The threadlet stack is part of the request structure
* and is thus reused as threadlets complete:
*/
unsigned long threadlet_stack;
/*
* These are all the request-specific parameters:
*/
long sock;
};//#define ulog(f, a...) fprintf(stderr, f, ##a)
#define ulog(f, a...)
#define ulog_err(f, a...) printf(f ": %s [%d].\n", ##a, strerror(errno), errno)
static long handle_request(void *__req)
{
struct request *req = __req;
int s = req->sock, err, fd;
off_t offset;
int count;
char path[] = "/tmp/index.html";
char buf[4096];
struct timeval tm;
count = 40960;
offset = 0;
err = recv(s, buf, sizeof(buf), 0);
if (err < 0) {
ulog_err("Failed to read data from s=%d", s);
goto err_out_remove;
}
if (err == 0) {
gettimeofday(&tm, NULL);
ulog("%08lu:%06lu: Client exited: fd=%d.\n", tm.tv_sec, tm.tv_usec, s);
goto err_out_remove;
}
fd = open(path, O_RDONLY);
if (fd == -1) {
ulog_err("Failed to open '%s'", path);
err = -1;
goto err_out_remove;
}
#if 0
do {
err = read(fd, buf, sizeof(buf));
if (err <= 0)
break;
err = send(s, buf, err, 0);
if (err <= 0)
break;
} while (1);
#endif
err = sendfile(s, fd, &offset, count);
{
int on = 0;
setsockopt(s, SOL_TCP, TCP_CORK, &on, sizeof(on));
}
close(fd);
if (err < 0) {
ulog_err("Failed send %d bytes: fd=%d.\n", count, s);
goto err_out_remove;
}
close(s);
gettimeofday(&tm, NULL);
ulog("%08lu:%06lu: %d bytes has been sent to client fd=%d.\n", tm.tv_sec, tm.tv_usec, err, s);
return complete_threadlet_fn(req, &async_head);err_out_remove:
printf("blah\n");
close(s);
return complete_threadlet_fn(req, &async_head);
}/*
* Freelist to recycle requests:
*/
static struct request *freelist;
/*
* Allocate a request and set up its syslet atoms:
*/
static struct request *alloc_req(void)
{
struct request *req;
/*
* Occasionally we have to refill the new-thread stack
* entry:
*/
if (!async_head.new_thread_stack) {
async_head.new_thread_stack = thread_stack_alloc();
pr("allocated new thread stack: %08lx\n",
async_head.new_thread_stack);
}
if (freelist) {
req = freelist;
pr("reusing req %p, threadlet stack %08lx\n",
req, req->threadlet_stack);
freelist = freelist->next_free;
req->next_free = NULL;
return req;
}
req = calloc(1, sizeof(struct request));
pr("allocated req %p\n", req);
req->threadlet_stack = thread_stack_alloc();
pr("allocated thread stack %08lx\n", req->threadlet_stack);
return req;
}/*
* Check whether there are any completions queued for user-space
* to finish up:
*/
static unsigned long complete(void)
{
unsigned long completed = 0;
struct request *req;
for (;;) {
req = (void *)completion_ring[async_head.user_ring_idx];
if (!req)
return completed;
completed++;
pr("completed req %p (threadlet stack %08lx)\n",
req, req->threadlet_stack);
req->next_free = freelist;
freelist = req;
/*
* Clear the completion pointer. To make sure the
* kernel never stomps upon still unhandled completions
* in the ring the kernel only writes to a NULL entry,
* so user-space has to clear it explicitly:
*/
completion_ring[async_head.user_ring_idx] = NULL;
async_head.user_ring_idx++;
if (async_head.user_ring_idx == MAX_PENDING)
async_head.user_ring_idx = 0;
}
}static unsigned int pending_requests;
/*
* Handle a request that has just been submitted (either it has
* already been executed, or we have to account it as pending):
*/
static void handle_submitted_request(struct request *req, long done)
{
unsigned int nr;
if (done) {
/*
* This is the cached case - free the request:
*/
pr("cache completed req %p (threadlet stack %08lx)\n",
req, req->threadlet_stack);
req->next_free = freelist;
freelist = req;
return;
}
/*
* 'cachemiss' case - the syslet is not finished
* yet. We will be notified about its completion
* via the completion ring:
*/
assert(pending_requests < MAX_PENDING-1);
pending_requests++;
pr("req %p is pending. %d reqs pending.\n", req, pending_requests);
/*
* Attempt to complete requests - this is a fast
* check if there's no completions:
*/
nr = complete();
pending_requests -= nr;
/*
* If the ring is full then wait a bit:
*/
while (pending_requests == MAX_PENDING-1) {
pr("sys_async_wait()");
/*
* Wait for 4 events - to batch things a bit:
*/
sys_async_wait(4, async_head.user_ring_idx, &async_head);
nr = complete();
pending_requests -= nr;
pr("after wait: completed %d requests - still pending: %d\n",
nr, pending_requests);
}
}static void webserver(int l_sock)
{
struct sockaddr addr;
struct request *req;
socklen_t addrlen;
long done;
int sock;
async_head_init();
/*
* We dont use syslets for the first request (or for oversized
* and/or erroneous requests):
*/
for (;;) {
req = alloc_req();
if (!req)
break;
sock = accept(l_sock, &addr, &addrlen);
if (sock < 0)
break;
req->sock = sock;
done = threadlet_exec(handle_request, req,
req->threadlet_stack, &async_head);
handle_submitted_request(req, done);
}
}static int tcp_listen_socket(int *l_socket)
{
int ret;
int one = 1;
struct sockaddr_in ad;
/*
* Setup for server socket
*/
memset(&ad, 0, sizeof(ad));
ad.sin_family = AF_INET;
ad.sin_addr.s_addr = htonl(INADDR_ANY);
ad.sin_port = htons(2222);
*l_socket = socket(PF_INET, SOCK_STREAM, IPPROTO_TCP);
if (*l_socket < 0) {
perror("socket (server)");
return *l_socket;
}
ret = setsockopt(*l_socket, SOL_SOCKET, SO_REUSEADDR, (char *)&one,
sizeof(int));
if (ret < 0) {
perror("setsockopt");
return ret;
}
ret = bind(*l_socket, (struct sockaddr *)&ad, sizeof(ad));
if (ret < 0) {
perror("bind");
return ret;
}
ret = listen(*l_socket, 1000);
if (ret < 0) {
perror("listen");
return ret;
}
return 0;
}int main(int argc, char *argv[])
{
int ret;
int l_sock;
struct sched_param p = { sched_priority: 0 };
sched_setscheduler(getpid(), 3 /* SCHED_BATCH */, &p);
ret = tcp_listen_socket(&l_sock);
if (ret)
exit(ret);
printf("listening on port 2222\n");
printf("using threadlets\n");
webserver(l_sock);
async_head_exit();
exit(0);
}Re: threadlets as 'naive pool of threads', epoll, some measurements
Quoted 9 lines · Ingo Molnar
yet another performance update - with the fixed 'heaps of stupid threads'
evserver_threadlet.ccode attached below i got:Quoted 2 lines
evserver_epoll: 9400 reqs/sec
evserver_epoll_threadlet: 9400 reqs/sec
evserver_threadlet: 9000 reqs/sec
Quoted 8 lines
so the overhead, instead of the 10x slowdown Evgeniy predicted/feared, is 4% for this particular, very event-centric workload.
why? because Evgeniy still overlooks what i've mentioned so many times:
that there is lots of inherent 'caching' possible even in this particular '8000 clients' workload, which even the most stupid threadlet queueing model is able to take advantage of. The maximum level of parallelism that i've measured during this test was 161 threads.
:)
I feared _ONLY_ situation when thousands of thereads are eating my brain
- so case when 161 threads are running simultanesoulsy is not that bad compared to what micro-design can do (of its best/worst) at all!
So, caching is good - threadlets do not spawn a new thread, kevent returns immediately, but in case of things are not that shine - threadlets spawnd a new thread, while kevent process next request or waits for all completed.
I'm a bit stuck right now with my benchmarks - intel core2 machine requires reinstallation (it is installed for amd64 arch of debian testing and admins at paid work setup my internet connection down to miserable bytes per second (who said that hacked/social engineered 1mb/sec can live forever?), so expect it tomorrow), via epia one is under stress testing right now.
Quoted 1 line
Ingo
Signature · 1 line
Evgeniy Polyakov
Re: threadlets as 'naive pool of threads', epoll, some measurements
Quoted 19 lines · Evgeniy Polyakov
Quoted 12 lines
yet another performance update - with the fixed 'heaps of stupid threads'
evserver_threadlet.ccode attached below i got:Quoted 2 lines
evserver_epoll: 9400 reqs/sec
evserver_epoll_threadlet: 9400 reqs/sec
evserver_threadlet: 9000 reqs/secso the overhead, instead of the 10x slowdown Evgeniy predicted/feared, is 4% for this particular, very event-centric workload.
Quoted 6 lines
why? because Evgeniy still overlooks what i've mentioned so many
times: that there is lots of inherent 'caching' possible even in this particular '8000 clients' workload, which even the most stupid threadlet queueing model is able to take advantage of. The maximum level of parallelism that i've measured during this test was 161 threads.
Quoted 6 lines
:)
I feared _ONLY_ situation when thousands of thereads are eating my brain - so case when 161 threads are running simultanesoulsy is not that bad compared to what micro-design can do (of its best/worst) at all!
even with ten thousand threads it is still pretty fast. Certainly not '10 times slower' as you claimed. And it takes only a single, trivial outer event loop to lift it up to the performance levels of a pure event based server.
conclusion: currently i dont see a compelling need for the kevents subsystem. epoll is a pretty nice API and it covers most of the event sources and nicely builds upon our existing poll() infrastructure.
furthermore, i very much contest your claim that a high-performance, highly scalable webserver needs a kevent+nonblock design. Even if i ignore all the obvious usability and maintainance-cost advantages of threadlets.
Quoted 4 lines
So, caching is good - threadlets do not spawn a new thread, kevent returns immediately, but in case of things are not that shine - threadlets spawnd a new thread, while kevent process next request or waits for all completed.
no. Please read the evserver_threadlet.c code. There's no kevent in there. There's no epoll() in there. All that you can see there is the natural behavior of pure threadlets. And it's not a workload /I/ picked for threadlets - it is a workload, filesize, parallelism level and request handling function /you/ picked for "event-servers".
Ingo
Re: threadlets as 'naive pool of threads', epoll, some measurements
Quoted 9 lines · Ingo Molnar
Quoted 4 lines
I feared _ONLY_ situation when thousands of thereads are eating my brain - so case when 161 threads are running simultanesoulsy is not that bad compared to what micro-design can do (of its best/worst) at all!
even with ten thousand threads it is still pretty fast. Certainly not '10 times slower' as you claimed. And it takes only a single, trivial outer event loop to lift it up to the performance levels of a pure event based server.
I did not claim that it will be 10 times slower, I said that it will be slower, my '10 times slower', which are actually '15% of hte total time' is a reply to your 'fast as sync' model, no need to repaing the picture :)
Quoted 8 lines
conclusion: currently i dont see a compelling need for the kevents subsystem. epoll is a pretty nice API and it covers most of the event sources and nicely builds upon our existing
poll()infrastructure.furthermore, i very much contest your claim that a high-performance, highly scalable webserver needs a kevent+nonblock design. Even if i ignore all the obvious usability and maintainance-cost advantages of threadlets.
Ok, I see your point, you insult something you did not ever try to understand, that is your right.
Quoted 10 lines
Quoted 4 lines
So, caching is good - threadlets do not spawn a new thread, kevent returns immediately, but in case of things are not that shine - threadlets spawnd a new thread, while kevent process next request or waits for all completed.
no. Please read the
evserver_threadlet.ccode. There's no kevent in there. There's noepoll()in there. All that you can see there is the natural behavior of pure threadlets. And it's not a workload /I/ picked for threadlets - it is a workload, filesize, parallelism level and request handling function /you/ picked for "event-servers".
I know that there is no kevents there, that would be really strange if you would test it in your environment after all that empty kevent releases.
Enough, you say micro-thread design is superior - ok, that is your point.
Quoted 1 line
Ingo
Signature · 1 line
Evgeniy Polyakov
Re: threadlets as 'naive pool of threads', epoll, some measurements
Quoted 10 lines · Evgeniy Polyakov
Quoted 6 lines
no. Please read the
evserver_threadlet.ccode. There's no kevent in there. There's noepoll()in there. All that you can see there is the natural behavior of pure threadlets. And it's not a workload /I/ picked for threadlets - it is a workload, filesize, parallelism level and request handling function /you/ picked for "event-servers".I know that there is no kevents there, that would be really strange if you would test it in your environment after all that empty kevent releases.
i havent got around figuring out the last v2.6.20 based kevent release, and your git tree is v2.6.21-rc1 based. Do you have some easy URL for me to fetch the last v2.6.20 kevent release?
Quoted 2 lines
Enough, you say micro-thread design is superior - ok, that is your point.
note that threadlets are not 'micro-threads'. A threadlet is more of an 'optional thread' (as i mentioned it earlier): whenever it does anything that makes it distinct from a plain function call, it's converted into a separate thread by the kernel. Otherwise it behaves like a plain function call and returns.
Ingo
Re: threadlets as 'naive pool of threads', epoll, some measurements
Quoted 10 lines · Ingo Molnar
Quoted 7 lines · Evgeniy Polyakov
Quoted 6 lines
no. Please read the
evserver_threadlet.ccode. There's no kevent in there. There's noepoll()in there. All that you can see there is the natural behavior of pure threadlets. And it's not a workload /I/ picked for threadlets - it is a workload, filesize, parallelism level and request handling function /you/ picked for "event-servers".
Quoted 7 lines
Quoted 3 lines
I know that there is no kevents there, that would be really strange if you would test it in your environment after all that empty kevent releases.
i havent got around figuring out the last v2.6.20 based kevent release, and your git tree is v2.6.21-rc1 based. Do you have some easy URL for me to fetch the last v2.6.20 kevent release?
I use kevent-36 release patches on top of 2.6.20 tree. There are some syscall numbers overlap with thrteadlet patches, but rejectsa re trivial.
Quoted 8 lines
Quoted 2 lines
Enough, you say micro-thread design is superior - ok, that is your point.
note that threadlets are not 'micro-threads'. A threadlet is more of an 'optional thread' (as i mentioned it earlier): whenever it does anything that makes it distinct from a plain function call, it's converted into a separate thread by the kernel. Otherwise it behaves like a plain function call and returns.
I know.
But it is rare case for the most situations, when things do not block, so I called it micro-thread, since it spawns a new thread (get from preallocated pool) for parallel processing.
Quoted 1 line
Ingo
Signature · 1 line
Evgeniy Polyakov
Re: threadlets as 'naive pool of threads', epoll, some measurements
Quoted 9 lines · Evgeniy Polyakov
Quoted 8 lines
Quoted 2 lines
Enough, you say micro-thread design is superior - ok, that is your point.
note that threadlets are not 'micro-threads'. A threadlet is more of an 'optional thread' (as i mentioned it earlier): whenever it does anything that makes it distinct from a plain function call, it's converted into a separate thread by the kernel. Otherwise it behaves like a plain function call and returns.
Quoted 4 lines
I know.
But it is rare case for the most situations, when things do not block, so I called it micro-thread, since it spawns a new thread (get from preallocated pool) for parallel processing.
ugh. Because 'it spawns a new thread from a preallocated pool' you are arbitrarily renaming threadlets to 'micro-threads'?? The kernel could be using a transparent thread pool for ordinary pthread recycling itself (and will possibly do so in the future) - that does not make them a micro-thread one iota. So please stop calling them micro-threads, threadlets are a distinctly separate concept ...
( And i guess you should know it perfectly well from my past mails in
this thread that i dont like micro-thread concepts at all, so are you perhaps calling threadlets 'micro-threads' intentionally, just to force a predictably negative reaction from me? Maybe i should start renaming your code too and refer to kevents as 'kpoll'? That too makes absolutely zero sense. This is getting really silly. )
Ingo
Re: threadlets as 'naive pool of threads', epoll, some measurements
Quoted 24 lines · Ingo Molnar
Quoted 9 lines · Evgeniy Polyakov
Quoted 8 lines
Quoted 2 lines
Enough, you say micro-thread design is superior - ok, that is your point.
note that threadlets are not 'micro-threads'. A threadlet is more of an 'optional thread' (as i mentioned it earlier): whenever it does anything that makes it distinct from a plain function call, it's converted into a separate thread by the kernel. Otherwise it behaves like a plain function call and returns.
Quoted 4 lines
I know.
But it is rare case for the most situations, when things do not block, so I called it micro-thread, since it spawns a new thread (get from preallocated pool) for parallel processing.ugh. Because 'it spawns a new thread from a preallocated pool' you are arbitrarily renaming threadlets to 'micro-threads'?? The kernel could be using a transparent thread pool for ordinary pthread recycling itself (and will possibly do so in the future) - that does not make them a micro-thread one iota. So please stop calling them micro-threads, threadlets are a distinctly separate concept ...
Quoted 6 lines
( And i guess you should know it perfectly well from my past mails in
this thread that i dont like micro-thread concepts at all, so are you perhaps calling threadlets 'micro-threads' intentionally, just to force a predictably negative reaction from me? Maybe i should start renaming your code too and refer to kevents as 'kpoll'? That too makes absolutely zero sense. This is getting really silly. )
I already think about renaming kevent aio, since it uses kaio name, which you frequently reference too, but you definitely did not think about kevent.
And out of curiosity, how masichistic I would look if I intentinally want to receive negative reaction from you :)
As far as you can recall, in all syslet related threads I was always for them, and definitely against micro-threads, but when we come to the land of IO processing using event driven model - here I can not agree with you.
So, ok, no micro-thread name.
Quoted 1 line
Ingo
Signature · 1 line
Evgeniy Polyakov
Re: threadlets as 'naive pool of threads', epoll, some measurements
Quoted 1 line · Evgeniy Polyakov
So, ok, no micro-thread name.
thanks!
Ingo
Re: threadlets as 'naive pool of threads', epoll, some measurements
Quoted 3 lines · Ingo Molnar
Quoted 1 line
So, ok, no micro-thread name.
thanks!
:) no problem!
Quoted 1 line
Ingo
Signature · 1 line
Evgeniy Polyakov
Re: threadlets as 'naive pool of threads', epoll, some measurements
In reply to Ingo MolnarQuoted 26 lines · Ingo Molnar
update:
Quoted 4 lines
i have tried the one Evgeniy provided in the URL:
http://tservice.net.ru/~s0mbre/archive/kevent/evserver_epoll.c
Quoted 8 lines
and 'ab -k -c8000 -n80000' almost always aborts with:
apr_socket_recv: Connection reset by peer (104)in the few cases it finishes, i got the following epoll result, over gigabit ethernet, on an UP Athlon64 box:
eserver_epoll: 7800 reqs/sec
eserver_epoll.chad a number of bugs. The most serious one was the apparently buggy use of "EPOLLET" (edge-triggered events). Removing that and moving epoll to level-triggered (which is slower but does not result in missed events) gives:
eserver_epoll: 9400 reqs/secQuoted 3 lines
the same with the most naive implementation of the same, using threadlets:
Quoted 6 lines
Quoted 1 line
eserver_threadlet: 5800 reqs/sec
eserver_epoll_threadlet: 9400 reqs/secas expected, the level of extra blocking triggered by this is low - even if the full request function runs without nonblock assumptions.
That looks pretty good. I started (spare time allowing) laying down the based for a more realistic test as far webserver-like benchmarking. I want to compare three solutions, that uses the same internal code (as far as request parsing and content delivery).
1) A fully thread classical web "server". This does the trivial accept+pthread_create and in there does open+fstat+sendhdrs+sendfile. This is already done:
http://www.xmailserver.org/thrhttp.c
Do a gcc -o thrhttp thrhttp.c -lpthread and you're done.
2) A coronet (coroutine+epoll library) handling for network events/dispatch, plus GUASI (Generic Userspace Asyncronous Syscall Interface) to handle generic IO.
libpcl: http://www.xmailserver.org/libpcl.html coronet: http://www.xmailserver.org/coronet-lib.html GUASI: http://www.xmailserver.org/guasi-lib.html
cghttpd: http://www.xmailserver.org/cghttpd-home.html
All these are configure+make+make_install installable.
The cghttpd server has the same parsing/content-delivery features of thrhttp, but it uses coroutine+epoll to handle network events, and GUASI (an old pthread-based userspace implementation of async execution) to give open/fstat/sendfile an async behaviour. It implements the epoll_wait() (conet_events_wait() acutally, but that maps directly to epoll_wait()) hosted inside an async GUASI request, and the GUASI dispatch loop handle the case with:
if (cookie == conet_events_wait_cookie)
handle_network_events();3) Finally, a very similar to cghttpd implementation, but this time using the *syslets* to handle async requests. That should be a prety easy change to cghttpd, by the mean of replacing the CGHTTPD_SYSCALL() macro with proper syslet code. The big advantage of the syslets here, is that they do have the cachehit optimization, while GUASI always have to trigger a queueing.
Of course, since the only machine I have with enough RAM to keep many thousands of session active is a dual Opteron, I'd need to have an x86-64 version of the patch. That shouldn't be a big problem though.
This hopefully may prove two points. First, a fully threaded solution does not scale well when dealing with thousands and thousands of sessions. Second, the cachehit syslet trick is The Man in the syslet code, and kicks userspace solution #2 ass.
In the meantime, I think Jens tests are more meaningfull, as far of usage field goes, than any network based test.
- Davide
Quoted 1 line · Davide Libenzi
Also, the
evtest_kevent_removecall is superfluous with epoll.
it's only used in the error path AFAICS.
but you are right about evserver_epoll/kevent.c incorrectly assuming that things wont block in evtest_callback_client(), which, after receiving the "there's stuff on the input socket" event does:
recvmsg(sock), fd = open(); sendfile(sock, fd) close(fd);
while evserver_threadlet.c, even this naive implementation, does not assume that we wont block in that function.
Quoted 2 lines
In any case, comparing epoll/kevent with 100K active sessions, against threadlets, is not exactly a fair/appropriate test for it.
fully agreed.
Ingo
Quoted 13 lines · Ingo Molnar
Quoted 1 line
Also, the
evtest_kevent_removecall is superfluous with epoll.it's only used in the error path AFAICS.
but you are right about
evserver_epoll/kevent.cincorrectly assuming that things wont block inevtest_callback_client(), which, after receiving the "there's stuff on the input socket" event does:recvmsg(sock), fd = open(); sendfile(sock, fd) close(fd);
Quoted 7 lines
while
evserver_threadlet.c, even this naive implementation, does not assume that we wont block in that function.Quoted 2 lines
In any case, comparing epoll/kevent with 100K active sessions, against threadlets, is not exactly a fair/appropriate test for it.
fully agreed.
Hi.
I will highlight several items in this mail:
1. evserver_epoll.c is broken in that regard, that it does not close a socket - I tried to make it possible to work with keepalive, but failed. So close of the socket is a must, like in evserver_kevent.c
2. keepalive is not supported - it is a hack server at the end.
3. this test does not assume that above snippet blocks or does not - it is _typical_ case of the web server with one working thread (per cpu) - every op can block, so there is no problem - threadlet will reschedule, event based will block (bad for them).
4. benchmark does not cover all possible cases - initial goal of that servers was to show how fast/slow _event_ generation/processing is in epoll/kevent case, but not to create real-life web server. lighttpd for example can not be used as a good bench too, since its arch does not support some kevent extensions (which do not exist in epoll), and, looking to the number of comments in kevent threads, I'm not motivated to change it at all.
So, drawing a line, evserver_* is a simple event driven server, it does have disadvantages, but the same approach only favours threadlet model. Having millions or thousands of connections works against threadlets, but we compare ultimate cases, it is a one of the possible tests.
So...
I'm cooking up a git tree with kevents and threadlets, which I will test in via epia (1ghz, 256 mb of ram, 100mbit lan) and Intel(R) Core(TM)2 CPU 6600 @ 2.40GHz (2gb of ram, 1gbit lan) later today with kevent/epoll/threadlet if wine will not end suddenly.
I will use Ingo's evserver_threadlet server as plong as evserver_epoll (with fixed closing) and evserver_kevent.c.
Eventually I can move all of them into one file.
Client is 'ab' on my desktop core duo 3.7 ghz. Machines are connected over gigabit dlink dgs-1216t switch (which freezes on slightly broken dhcp and tftp packets btw).
Stay tuned.
P.S. Linus, if you do not mind, I will postpone scholastic masturbation about event vs process context in IO. One sentence only - page fault and filename lookup both wait - they wait for new page is ready or inode is read from the disk, eventually they wake up, thing which wakes them up _is_ an event.
Quoted 1 line
Ingo
Signature · 1 line
Evgeniy Polyakov
Quoted 2 lines · Evgeniy Polyakov
I will use Ingo's
evserver_threadletserver as plong asevserver_epoll(with fixed closing) andevserver_kevent.c.
please also try evserver_epoll_threadlet.c that i've attached below - it uses epoll as the main event mechanism but does threadlets for request handling.
This is a one step more intelligent threadlet queueing model than 'thousands of threads' - although obviously epoll alone should do well too with this trivial workload.
Ingo
---------------------------->
#include <sys/types.h>
#include <sys/socket.h>
#include <sys/resource.h>
#include <sys/wait.h>
#include <sys/ioctl.h>
#include <sys/stat.h>
#include <sys/time.h>
#include <sys/poll.h>
#include <sys/sendfile.h>
#include <sys/epoll.h>
#include <netinet/in.h>
#include <netinet/tcp.h>
#include <arpa/inet.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <errno.h>
#include <string.h>
#include <fcntl.h>
#include <time.h>
#include <ctype.h>
#include <netdb.h>
#define DEBUG 0
#include "syslet.h"
#include "sys.h"
#include "threadlet.h"
struct request {
struct request *next_free;
/*
* The threadlet stack is part of the request structure
* and is thus reused as threadlets complete:
*/
unsigned long threadlet_stack;
/*
* These are all the request-specific parameters:
*/
long sock;
};/*
* Freelist to recycle requests:
*/
static struct request *freelist;
/*
* Allocate a request and set up its syslet atoms:
*/
static struct request *alloc_req(void)
{
struct request *req;
/*
* Occasionally we have to refill the new-thread stack
* entry:
*/
if (!async_head.new_thread_stack) {
async_head.new_thread_stack = thread_stack_alloc();
pr("allocated new thread stack: %08lx\n",
async_head.new_thread_stack);
}
if (freelist) {
req = freelist;
pr("reusing req %p, threadlet stack %08lx\n",
req, req->threadlet_stack);
freelist = freelist->next_free;
req->next_free = NULL;
return req;
}
req = calloc(1, sizeof(struct request));
pr("allocated req %p\n", req);
req->threadlet_stack = thread_stack_alloc();
pr("allocated thread stack %08lx\n", req->threadlet_stack);
return req;
}/*
* Check whether there are any completions queued for user-space
* to finish up:
*/
static unsigned long complete(void)
{
unsigned long completed = 0;
struct request *req;
for (;;) {
req = (void *)completion_ring[async_head.user_ring_idx];
if (!req)
return completed;
completed++;
pr("completed req %p (threadlet stack %08lx)\n",
req, req->threadlet_stack);
req->next_free = freelist;
freelist = req;
/*
* Clear the completion pointer. To make sure the
* kernel never stomps upon still unhandled completions
* in the ring the kernel only writes to a NULL entry,
* so user-space has to clear it explicitly:
*/
completion_ring[async_head.user_ring_idx] = NULL;
async_head.user_ring_idx++;
if (async_head.user_ring_idx == MAX_PENDING)
async_head.user_ring_idx = 0;
}
}static unsigned int pending_requests;
/*
* Handle a request that has just been submitted (either it has
* already been executed, or we have to account it as pending):
*/
static void handle_submitted_request(struct request *req, long done)
{
unsigned int nr;
if (done) {
/*
* This is the cached case - free the request:
*/
pr("cache completed req %p (threadlet stack %08lx)\n",
req, req->threadlet_stack);
req->next_free = freelist;
freelist = req;
return;
}
/*
* 'cachemiss' case - the syslet is not finished
* yet. We will be notified about its completion
* via the completion ring:
*/
assert(pending_requests < MAX_PENDING-1);
pending_requests++;
pr("req %p is pending. %d reqs pending.\n", req, pending_requests);
/*
* Attempt to complete requests - this is a fast
* check if there's no completions:
*/
nr = complete();
pending_requests -= nr;
/*
* If the ring is full then wait a bit:
*/
while (pending_requests == MAX_PENDING-1) {
pr("sys_async_wait()");
/*
* Wait for 4 events - to batch things a bit:
*/
sys_async_wait(4, async_head.user_ring_idx, &async_head);
nr = complete();
pending_requests -= nr;
pr("after wait: completed %d requests - still pending: %d\n",
nr, pending_requests);
}
}#include <linux/types.h>
//#define ulog(f, a...) fprintf(stderr, f, ##a)
#define ulog(f, a...)
#define ulog_err(f, a...) printf(f ": %s [%d].\n", ##a, strerror(errno), errno)
static int kevent_ctl_fd, main_server_s;
static void usage(char *p)
{
ulog("Usage: %s -a addr -p port -f kevent_path -t timeout -w wait_num\n", p);
}static int evtest_server_init(char *addr, unsigned short port)
{
struct hostent *h;
int s, on;
struct sockaddr_in sa;
if (!addr) {
ulog("%s: Bind address cannot be NULL.\n", __func__);
return -1;
}
h = gethostbyname(addr);
if (!h) {
ulog_err("%s: Failed to get address of %s.\n", __func__, addr);
return -1;
}
s = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);
if (s == -1) {
ulog_err("%s: Failed to create server socket", __func__);
return -1;
}
fcntl(s, F_SETFL, O_NONBLOCK);
memcpy(&(sa.sin_addr.s_addr), h->h_addr_list[0], 4);
sa.sin_port = htons(port);
sa.sin_family = AF_INET;
on = 1;
setsockopt(s, SOL_SOCKET, SO_REUSEADDR, &on, 4);
if (bind(s, (struct sockaddr *)&sa, sizeof(struct sockaddr_in)) == -1) {
ulog_err("%s: Failed to bind to %s", __func__, addr);
close(s);
return -1;
}
if (listen(s, 30000) == -1) {
ulog_err("%s: Failed to listen on %s", __func__, addr);
close(s);
return -1;
}
return s;
}static int evtest_kevent_remove(int fd)
{
int err;
struct epoll_event event;
event.events = EPOLLIN | EPOLLET;
event.data.fd = fd;
err = epoll_ctl(kevent_ctl_fd, EPOLL_CTL_DEL, fd, &event);
if (err < 0) {
ulog_err("Failed to perform control REMOVE operation");
return err;
}
return err;
}static int evtest_kevent_init(int fd)
{
int err;
struct timeval tm;
struct epoll_event event;
event.events = EPOLLIN | EPOLLET;
event.data.fd = fd;
err = epoll_ctl(kevent_ctl_fd, EPOLL_CTL_ADD, fd, &event);
gettimeofday(&tm, NULL);
ulog("%08lu:%06lu: fd=%3d, err=%1d.\n", tm.tv_sec, tm.tv_usec, fd, err);
if (err < 0) {
ulog_err("Failed to perform control ADD operation: fd=%d, events=%08x", fd, event.events);
return err;
}
return err;
}static long handle_request(void *__req)
{
struct request *req = __req;
int s = req->sock, err, fd;
off_t offset;
int count;
char path[] = "/tmp/index.html";
char buf[4096];
struct timeval tm;
count = 40960;
offset = 0;
err = recv(s, buf, sizeof(buf), 0);
if (err < 0) {
ulog_err("Failed to read data from s=%d", s);
goto err_out_remove;
}
if (err == 0) {
gettimeofday(&tm, NULL);
ulog("%08lu:%06lu: Client exited: fd=%d.\n", tm.tv_sec, tm.tv_usec, s);
goto err_out_remove;
}
fd = open(path, O_RDONLY);
if (fd == -1) {
ulog_err("Failed to open '%s'", path);
err = -1;
goto err_out_remove;
}
#if 0
do {
err = read(fd, buf, sizeof(buf));
if (err <= 0)
break;
err = send(s, buf, err, 0);
if (err <= 0)
break;
} while (1);
#endif
err = sendfile(s, fd, &offset, count);
{
int on = 0;
setsockopt(s, SOL_TCP, TCP_CORK, &on, sizeof(on));
}
close(fd);
if (err < 0) {
ulog_err("Failed send %d bytes: fd=%d.\n", count, s);
goto err_out_remove;
}
gettimeofday(&tm, NULL);
ulog("%08lu:%06lu: %d bytes has been sent to client fd=%d.\n", tm.tv_sec, tm.tv_usec, err, s);
close(s);
return complete_threadlet_fn(req, &async_head);err_out_remove:
evtest_kevent_remove(s); close(s); return complete_threadlet_fn(req, &async_head); }
static int evtest_callback_client(int sock)
{
struct request *req;
long done;
req = alloc_req();
if (!req) {
printf("no req\n");
evtest_kevent_remove(sock);
return -ENOMEM;
}
req->sock = sock;
done = threadlet_exec(handle_request, req,
req->threadlet_stack, &async_head);
handle_submitted_request(req, done);
return 0;
}static int evtest_callback_main(int s)
{
int cs, err;
struct sockaddr_in csa;
socklen_t addrlen = sizeof(struct sockaddr_in);
struct timeval tm;
memset(&csa, 0, sizeof(csa));
if ((cs = accept(s, (struct sockaddr *)&csa, &addrlen)) == -1) {
ulog_err("Failed to accept client");
return -1;
}
fcntl(cs, F_SETFL, O_NONBLOCK);
gettimeofday(&tm, NULL);
ulog("%08lu:%06lu: Accepted connect from %s:%d.\n",
tm.tv_sec, tm.tv_usec,
inet_ntoa(csa.sin_addr), ntohs(csa.sin_port));
err = evtest_kevent_init(cs);
if (err < 0) {
close(cs);
return -1;
}
return 0;
}static int evtest_kevent_wait(unsigned int timeout, unsigned int wait_num)
{
int num, err;
struct timeval tm;
struct epoll_event event[256];
int i;
err = epoll_wait(kevent_ctl_fd, event, 256, -1);
if (err < 0) {
ulog_err("Failed to perform control operation");
return num;
}
gettimeofday(&tm, NULL);
num = err;
ulog("%08lu.%06lu: Wait: num=%d.\n", tm.tv_sec, tm.tv_usec, num);
for (i=0; i<num; ++i) {
if (event[i].data.fd == main_server_s)
err = evtest_callback_main(event[i].data.fd);
else
err = evtest_callback_client(event[i].data.fd);
}
return err;
}int main(int argc, char *argv[])
{
int ch, err;
char *addr;
unsigned short port;
unsigned int timeout, wait_num;
addr = "0.0.0.0";
port = 8080;
timeout = 1000;
wait_num = 1;
async_head_init();
while ((ch = getopt(argc, argv, "f:n:t:a:p:h")) > 0) {
switch (ch) {
case 't':
timeout = atoi(optarg);
break;
case 'n':
wait_num = atoi(optarg);
break;
case 'a':
addr = optarg;
break;
case 'p':
port = atoi(optarg);
break;
case 'f':
break;
default:
usage(argv[0]);
return -1;
}
}
kevent_ctl_fd = epoll_create(10);
if (kevent_ctl_fd == -1) {
ulog_err("Failed to epoll descriptor");
return -1;
}
main_server_s = evtest_server_init(addr, port);
if (main_server_s < 0)
return main_server_s;
err = evtest_kevent_init(main_server_s);
if (err < 0)
goto err_out_exit;
while (1) {
err = evtest_kevent_wait(timeout, wait_num);
}err_out_exit:
close(kevent_ctl_fd); async_head_exit(); return 0; }
Quoted 10 lines · Ingo Molnar
Quoted 2 lines
I will use Ingo's
evserver_threadletserver as plong asevserver_epoll(with fixed closing) andevserver_kevent.c.please also try
evserver_epoll_threadlet.cthat i've attached below - it uses epoll as the main event mechanism but does threadlets for request handling.This is a one step more intelligent threadlet queueing model than 'thousands of threads' - although obviously epoll alone should do well too with this trivial workload.
No problem.
If I will complete setup today before I go climbing (I need to do some paid job too), I will post results here and in my blog (without political correctness).
Btw, 'evserver' in the name means 'event server', so you might think about changing the name :)
Stay tuned.
Quoted 1 line
Ingo
Signature · 1 line
Evgeniy Polyakov
Quoted 2 lines · Evgeniy Polyakov
Btw, 'evserver' in the name means 'event server', so you might think about changing the name :)
why should i change the name? The 'outer' loop, which feeds requests to threadlets, is an epoll based event loop. The inner loop, where all the application complexity resides, is a threadlet. This is the "more intelligent queueing" model i talked about in my reply to David 4 days ago:
http://lkml.org/lkml/2007/2/22/180
http://lkml.org/lkml/2007/2/22/191
Ingo
Quoted 6 lines · Ingo Molnar
Quoted 2 lines · Evgeniy Polyakov
Btw, 'evserver' in the name means 'event server', so you might think about changing the name :)
Quoted 8 lines
why should i change the name? The 'outer' loop, which feeds requests to threadlets, is an epoll based event loop. The inner loop, where all the application complexity resides, is a threadlet. This is the "more intelligent queueing" model i talked about in my reply to David 4 days ago:
http://lkml.org/lkml/2007/2/22/180
http://lkml.org/lkml/2007/2/22/191
:)
Ingo, of course it was a joke.
Even having main dispatcher as epoll/kevent loop, the _whole_ threadlet model is absolutely micro-thread in nature and not state machine/event. So it does not have events at all, especially with speculations about removing completion notifications - fire and forget model.
Quoted 1 line
Ingo
Signature · 1 line
Evgeniy Polyakov
Quoted 3 lines · Evgeniy Polyakov
Even having main dispatcher as epoll/kevent loop, the _whole_ threadlet model is absolutely micro-thread in nature and not state machine/event.
Evgeniy, i'm not sure how many different ways to tell this to you, but you are not listening, you are not learning and you are still not getting it at all.
The scheduler /IS/ a generic work/event queue. And it's pretty damn fast. No amount of badmouthing will change that basic fact. Not exactly as fast as a special-purpose queueing system (for all the reasons i outlined to you, and which you ignored), but it gets pretty damn close even for the web workload /you/ identified, and offers a user-space programming model that is about 1000 times more useful than state-machines.
Ingo
Quoted 11 lines · Ingo Molnar
Quoted 3 lines · Evgeniy Polyakov
Even having main dispatcher as epoll/kevent loop, the _whole_ threadlet model is absolutely micro-thread in nature and not state machine/event.
Evgeniy, i'm not sure how many different ways to tell this to you, but you are not listening, you are not learning and you are still not getting it at all.
Quoted 7 lines
The scheduler /IS/ a generic work/event queue. And it's pretty damn fast. No amount of badmouthing will change that basic fact. Not exactly as fast as a special-purpose queueing system (for all the reasons i outlined to you, and which you ignored), but it gets pretty damn close even for the web workload /you/ identified, and offers a user-space programming model that is about 1000 times more useful than state-machines.
Meanwhile on practiceal side:
via epia kevent/epoll/threadlet:
client: ab -c500 -n5000 $url
kevent: 849.72
epoll: 538.16
threadlet:
gcc ./evserver_epoll_threadlet.c -o ./evserver_epoll_threadlet In file included from ./evserver_epoll_threadlet.c:30:
./threadlet.h: In function ‘threadlet_exec’:
./threadlet.h:46: error: can't find a register in class ‘GENERAL_REGS’ while reloading ‘asm’
That particular asm optimization fails to compile.
Quoted 1 line
Ingo
Signature · 1 line
Evgeniy Polyakov
Quoted 34 lines · Evgeniy Polyakov
Quoted 11 lines · Ingo Molnar
Quoted 3 lines · Evgeniy Polyakov
Even having main dispatcher as epoll/kevent loop, the _whole_ threadlet model is absolutely micro-thread in nature and not state machine/event.
Evgeniy, i'm not sure how many different ways to tell this to you, but you are not listening, you are not learning and you are still not getting it at all.
Quoted 7 lines
The scheduler /IS/ a generic work/event queue. And it's pretty damn fast. No amount of badmouthing will change that basic fact. Not exactly as fast as a special-purpose queueing system (for all the reasons i outlined to you, and which you ignored), but it gets pretty damn close even for the web workload /you/ identified, and offers a user-space programming model that is about 1000 times more useful than state-machines.
Meanwhile on practiceal side:
via epia kevent/epoll/threadlet:client: ab -c500 -n5000 $url
kevent: 849.72
epoll: 538.16
threadlet:
gcc ./evserver_epoll_threadlet.c -o ./evserver_epoll_threadlet In file included from ./evserver_epoll_threadlet.c:30:
./threadlet.h: In function ‘threadlet_exec’:
./threadlet.h:46: error: can't find a register in class ‘GENERAL_REGS’ while reloading ‘asm’
Quoted 1 line
That particular asm optimization fails to compile.
it's not really an asm optimization but an API glue. I'm using:
gcc -O2 -g -Wall -o evserver_epoll_threadlet evserver_epoll_threadlet.c -fomit-frame-pointer
does that work for you?
Ingo
Quoted 37 lines · Ingo Molnar
Quoted 34 lines · Evgeniy Polyakov
Quoted 11 lines · Ingo Molnar
Quoted 3 lines · Evgeniy Polyakov
Even having main dispatcher as epoll/kevent loop, the _whole_ threadlet model is absolutely micro-thread in nature and not state machine/event.
Evgeniy, i'm not sure how many different ways to tell this to you, but you are not listening, you are not learning and you are still not getting it at all.
Quoted 7 lines
The scheduler /IS/ a generic work/event queue. And it's pretty damn fast. No amount of badmouthing will change that basic fact. Not exactly as fast as a special-purpose queueing system (for all the reasons i outlined to you, and which you ignored), but it gets pretty damn close even for the web workload /you/ identified, and offers a user-space programming model that is about 1000 times more useful than state-machines.
Meanwhile on practiceal side:
via epia kevent/epoll/threadlet:client: ab -c500 -n5000 $url
kevent: 849.72
epoll: 538.16
threadlet:
gcc ./evserver_epoll_threadlet.c -o ./evserver_epoll_threadlet In file included from ./evserver_epoll_threadlet.c:30:
./threadlet.h: In function ‘threadlet_exec’:
./threadlet.h:46: error: can't find a register in class ‘GENERAL_REGS’ while reloading ‘asm’
Quoted 7 lines
Quoted 1 line
That particular asm optimization fails to compile.
it's not really an asm optimization but an API glue. I'm using:
gcc -O2 -g -Wall -o
evserver_epoll_threadletevserver_epoll_threadlet.c-fomit-frame-pointerdoes that work for you?
Yes, -fomit-frame-point make the deal.
In average, threadlet runs as fast as epoll. Just because there are _no_ rescheduling in that case.
I added a printk into __async_schedule() and started ab -c7000 -n20000 http://192.168.4.80/
against slow via epia, and got only one of them. Client is latest ../async-test-v4/evserver_epoll_threadlet
Btw, I need to admit, that I have totally broken kevent tree there - it does not work on that machine on higher loads, so I'm investigating that problem now.
Quoted 1 line
Ingo
Signature · 1 line
Evgeniy Polyakov
Quoted 5 lines · Evgeniy Polyakov
Quoted 1 line
does that work for you?
Yes, -fomit-frame-point make the deal.
In average, threadlet runs as fast as epoll.
yeah.
Quoted 1 line
Just because there are _no_ rescheduling in that case.
in my test it was 'little', not 'no'. But yes, that's exactly my point:
we can remove the nonblock hackeries from event loops and just concentrate on making it schedule in less than 10-20% of the cases. Even a relatively high 10-20% rescheduling rate is hardly measurable with threadlets, while it gives a 10%-20% regression (and possibly bad latencies) for the pure epoll/kevent server.
and such a mixed technique is simply not possible with ordinary user-space threads, because there it's an all-or-nothing affair: either you go fully to threads (at which point we are again back to a fully threaded design, now also saddled with event loop overhead), or you try to do user-space threads, which Just Make Little Sense (tm).
so threadlets remove the biggest headache from event loops: they dont have to be '100% nonblocking' anymore. No O_NONBLOCK overhead, no complex state machines - just handle the most common event type via an outer event loop and keep the other 99% of server complexity in plain procedural programming. 1% of state-machine code is perfectly acceptable.
Ingo
Quoted 3 lines · Ingo Molnar
please also try
evserver_epoll_threadlet.cthat i've attached below - it uses epoll as the main event mechanism but does threadlets for request handling.
find updated code below - your evserver_epoll.c spuriously missed event edges - so i changed it back to level-triggered. While that is not as fast as edge-triggered, it does not result in spurious hangs and workflow 'hickups' during the test.
Could this be the reason why in your testing kevents outperformed epoll?
Also, i have removed the set-nonblocking calls because they are not needed under threadlets.
[ to build this code, copy it into the async-test/ directory and build
it there - or copy the *.h files from async-test/ directory into your build directory. ]
Ingo
-------{ evserver_epoll_threadlet.c }--------------> #include <sys/types.h>
#include <sys/socket.h>
#include <sys/resource.h>
#include <sys/wait.h>
#include <sys/ioctl.h>
#include <sys/stat.h>
#include <sys/time.h>
#include <sys/poll.h>
#include <sys/sendfile.h>
#include <sys/epoll.h>
#include <netinet/in.h>
#include <netinet/tcp.h>
#include <arpa/inet.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <errno.h>
#include <string.h>
#include <fcntl.h>
#include <time.h>
#include <ctype.h>
#include <netdb.h>
#define DEBUG 0
#include "syslet.h"
#include "sys.h"
#include "threadlet.h"
struct request {
struct request *next_free;
/*
* The threadlet stack is part of the request structure
* and is thus reused as threadlets complete:
*/
unsigned long threadlet_stack;
/*
* These are all the request-specific parameters:
*/
long sock;
};/*
* Freelist to recycle requests:
*/
static struct request *freelist;
/*
* Allocate a request and set up its syslet atoms:
*/
static struct request *alloc_req(void)
{
struct request *req;
/*
* Occasionally we have to refill the new-thread stack
* entry:
*/
if (!async_head.new_thread_stack) {
async_head.new_thread_stack = thread_stack_alloc();
pr("allocated new thread stack: %08lx\n",
async_head.new_thread_stack);
}
if (freelist) {
req = freelist;
pr("reusing req %p, threadlet stack %08lx\n",
req, req->threadlet_stack);
freelist = freelist->next_free;
req->next_free = NULL;
return req;
}
req = calloc(1, sizeof(struct request));
pr("allocated req %p\n", req);
req->threadlet_stack = thread_stack_alloc();
pr("allocated thread stack %08lx\n", req->threadlet_stack);
return req;
}/*
* Check whether there are any completions queued for user-space
* to finish up:
*/
static unsigned long complete(void)
{
unsigned long completed = 0;
struct request *req;
for (;;) {
req = (void *)completion_ring[async_head.user_ring_idx];
if (!req)
return completed;
completed++;
pr("completed req %p (threadlet stack %08lx)\n",
req, req->threadlet_stack);
req->next_free = freelist;
freelist = req;
/*
* Clear the completion pointer. To make sure the
* kernel never stomps upon still unhandled completions
* in the ring the kernel only writes to a NULL entry,
* so user-space has to clear it explicitly:
*/
completion_ring[async_head.user_ring_idx] = NULL;
async_head.user_ring_idx++;
if (async_head.user_ring_idx == MAX_PENDING)
async_head.user_ring_idx = 0;
}
}static unsigned int pending_requests;
/*
* Handle a request that has just been submitted (either it has
* already been executed, or we have to account it as pending):
*/
static void handle_submitted_request(struct request *req, long done)
{
unsigned int nr;
if (done) {
/*
* This is the cached case - free the request:
*/
pr("cache completed req %p (threadlet stack %08lx)\n",
req, req->threadlet_stack);
req->next_free = freelist;
freelist = req;
return;
}
/*
* 'cachemiss' case - the syslet is not finished
* yet. We will be notified about its completion
* via the completion ring:
*/
assert(pending_requests < MAX_PENDING-1);
pending_requests++;
pr("req %p is pending. %d reqs pending.\n", req, pending_requests);
/*
* Attempt to complete requests - this is a fast
* check if there's no completions:
*/
nr = complete();
pending_requests -= nr;
/*
* If the ring is full then wait a bit:
*/
while (pending_requests == MAX_PENDING-1) {
pr("sys_async_wait()");
/*
* Wait for 4 events - to batch things a bit:
*/
sys_async_wait(4, async_head.user_ring_idx, &async_head);
nr = complete();
pending_requests -= nr;
pr("after wait: completed %d requests - still pending: %d\n",
nr, pending_requests);
}
}#include <linux/types.h>
//#define ulog(f, a...) fprintf(stderr, f, ##a)
#define ulog(f, a...)
#define ulog_err(f, a...) printf(f ": %s [%d].\n", ##a, strerror(errno), errno)
static int kevent_ctl_fd, main_server_s;
static void usage(char *p)
{
ulog("Usage: %s -a addr -p port -f kevent_path -t timeout -w wait_num\n", p);
}static int evtest_server_init(char *addr, unsigned short port)
{
struct hostent *h;
int s, on;
struct sockaddr_in sa;
if (!addr) {
ulog("%s: Bind address cannot be NULL.\n", __func__);
return -1;
}
h = gethostbyname(addr);
if (!h) {
ulog_err("%s: Failed to get address of %s.\n", __func__, addr);
return -1;
}
s = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);
if (s == -1) {
ulog_err("%s: Failed to create server socket", __func__);
return -1;
}// fcntl(s, F_SETFL, O_NONBLOCK);
memcpy(&(sa.sin_addr.s_addr), h->h_addr_list[0], 4);
sa.sin_port = htons(port);
sa.sin_family = AF_INET;
on = 1;
setsockopt(s, SOL_SOCKET, SO_REUSEADDR, &on, 4);
if (bind(s, (struct sockaddr *)&sa, sizeof(struct sockaddr_in)) == -1) {
ulog_err("%s: Failed to bind to %s", __func__, addr);
close(s);
return -1;
}
if (listen(s, 30000) == -1) {
ulog_err("%s: Failed to listen on %s", __func__, addr);
close(s);
return -1;
}
return s;
}#define EPOLL_EVENT_MASK (EPOLLIN | EPOLLERR | EPOLLPRI)
static int evtest_kevent_remove(int fd)
{
int err;
struct epoll_event event;
event.events = EPOLL_EVENT_MASK;
event.data.fd = fd;
err = epoll_ctl(kevent_ctl_fd, EPOLL_CTL_DEL, fd, &event);
if (err < 0) {
ulog_err("Failed to perform control REMOVE operation");
return err;
}
return err;
}static int evtest_kevent_init(int fd)
{
int err;
struct timeval tm;
struct epoll_event event;
event.events = EPOLL_EVENT_MASK;
event.data.fd = fd;
err = epoll_ctl(kevent_ctl_fd, EPOLL_CTL_ADD, fd, &event);
gettimeofday(&tm, NULL);
ulog("%08lu:%06lu: fd=%3d, err=%1d.\n", tm.tv_sec, tm.tv_usec, fd, err);
if (err < 0) {
ulog_err("Failed to perform control ADD operation: fd=%d, events=%08x", fd, event.events);
return err;
}
return err;
}#define MAX_FILES 1000000
/*
* Debug check:
*/
static struct request *fd_to_req[MAX_FILES];
static long handle_request(void *__req)
{
struct request *req = __req;
int s = req->sock, err, fd;
off_t offset;
int count;
char path[] = "/tmp/index.html";
char buf[4096];
struct timeval tm;
if (!fd_to_req[s])
ulog_err("Bad: no request to fd?");
count = 40960;
offset = 0;
err = recv(s, buf, sizeof(buf), 0);
if (err < 0) {
ulog_err("Failed to read data from s=%d", s);
goto err_out_remove;
}
if (err == 0) {
gettimeofday(&tm, NULL);
ulog("%08lu:%06lu: Client exited: fd=%d.\n", tm.tv_sec, tm.tv_usec, s);
goto err_out_remove;
}
fd = open(path, O_RDONLY);
if (fd == -1) {
ulog_err("Failed to open '%s'", path);
err = -1;
goto err_out_remove;
}
#if 0
do {
err = read(fd, buf, sizeof(buf));
if (err <= 0)
break;
err = send(s, buf, err, 0);
if (err <= 0)
break;
} while (1);
#endif
err = sendfile(s, fd, &offset, count);
{
int on = 0;
setsockopt(s, SOL_TCP, TCP_CORK, &on, sizeof(on));
}
close(fd);
if (err < 0) {
ulog_err("Failed send %d bytes: fd=%d.\n", count, s);
goto err_out_remove;
}
gettimeofday(&tm, NULL);
ulog("%08lu:%06lu: %d bytes has been sent to client fd=%d.\n", tm.tv_sec, tm.tv_usec, err, s);
close(s);
fd_to_req[s] = NULL;
return complete_threadlet_fn(req, &async_head);err_out_remove:
evtest_kevent_remove(s); close(s); fd_to_req[s] = NULL; return complete_threadlet_fn(req, &async_head); }
static int evtest_callback_client(int sock)
{
struct request *req;
long done;
if (fd_to_req[sock]) {
ulog_err("Bad: request overlap?");
return 0;
}
req = alloc_req();
if (!req) {
ulog_err("Bad: no req\n");
evtest_kevent_remove(sock);
return -ENOMEM;
}
req->sock = sock;
fd_to_req[sock] = req;
done = threadlet_exec(handle_request, req,
req->threadlet_stack, &async_head);
handle_submitted_request(req, done);
return 0;
}static int evtest_callback_main(int s)
{
int cs, err;
struct sockaddr_in csa;
socklen_t addrlen = sizeof(struct sockaddr_in);
struct timeval tm;
memset(&csa, 0, sizeof(csa));
if ((cs = accept(s, (struct sockaddr *)&csa, &addrlen)) == -1) {
ulog_err("Failed to accept client");
return -1;
}// fcntl(cs, F_SETFL, O_NONBLOCK);
gettimeofday(&tm, NULL);
ulog("%08lu:%06lu: Accepted connect from %s:%d.\n",
tm.tv_sec, tm.tv_usec,
inet_ntoa(csa.sin_addr), ntohs(csa.sin_port));
err = evtest_kevent_init(cs);
if (err < 0) {
close(cs);
return -1;
}
return 0;
}static int evtest_kevent_wait(unsigned int timeout, unsigned int wait_num)
{
int num, err;
struct timeval tm;
struct epoll_event event[256];
int i;
err = epoll_wait(kevent_ctl_fd, event, 256, -1);
if (err < 0) {
ulog_err("Failed to perform control operation");
return num;
}
gettimeofday(&tm, NULL);
num = err;
ulog("%08lu.%06lu: Wait: num=%d.\n", tm.tv_sec, tm.tv_usec, num);
for (i=0; i<num; ++i) {
if (event[i].data.fd == main_server_s)
err = evtest_callback_main(event[i].data.fd);
else
err = evtest_callback_client(event[i].data.fd);
}
return err;
}int main(int argc, char *argv[])
{
int ch, err;
char *addr;
unsigned short port;
unsigned int timeout, wait_num;
addr = "0.0.0.0";
port = 8080;
timeout = 1000;
wait_num = 1;
async_head_init();
while ((ch = getopt(argc, argv, "f:n:t:a:p:h")) > 0) {
switch (ch) {
case 't':
timeout = atoi(optarg);
break;
case 'n':
wait_num = atoi(optarg);
break;
case 'a':
addr = optarg;
break;
case 'p':
port = atoi(optarg);
break;
case 'f':
break;
default:
usage(argv[0]);
return -1;
}
}
kevent_ctl_fd = epoll_create(10);
if (kevent_ctl_fd == -1) {
ulog_err("Failed to epoll descriptor");
return -1;
}
main_server_s = evtest_server_init(addr, port);
if (main_server_s < 0)
return main_server_s;
err = evtest_kevent_init(main_server_s);
if (err < 0)
goto err_out_exit;
while (1) {
err = evtest_kevent_wait(timeout, wait_num);
}err_out_exit:
close(kevent_ctl_fd); async_head_exit(); return 0; }
Quoted 11 lines · Ingo Molnar
Quoted 3 lines · Ingo Molnar
please also try
evserver_epoll_threadlet.cthat i've attached below - it uses epoll as the main event mechanism but does threadlets for request handling.find updated code below - your
evserver_epoll.cspuriously missed event edges - so i changed it back to level-triggered. While that is not as fast as edge-triggered, it does not result in spurious hangs and workflow 'hickups' during the test.
Hmm, exact the same evserver_epoll.c you downloaded works ok for me, although yes, it is buggy in that regard that it does not contain socket close when data is transferred.
Quoted 1 line
Could this be the reason why in your testing kevents outperformed epoll?
I will try to check. In theory without _ET it should perfoem much worse, but in practice its performance is essentially the same (the same applies to kevent without KEVENT_REQ_ET flag - since the same socket almost never is used several times, it is purely zero overhead to have or not have that flag set).
Quoted 6 lines
Also, i have removed the set-nonblocking calls because they are not needed under threadlets.
[ to build this code, copy it into the async-test/ directory and build
it there - or copy the *.h files from async-test/ directory into your build directory. ]
Ok, right now I'm compiling kevent/threadlet tree on my test machines.
Quoted 1 line
Ingo
Signature · 1 line
Evgeniy Polyakov
Quoted 7 lines · Ingo Molnar
Quoted 3 lines · Ingo Molnar
please also try
evserver_epoll_threadlet.cthat i've attached below - it uses epoll as the main event mechanism but does threadlets for request handling.
Quoted 6 lines
find updated code below - your
evserver_epoll.cspuriously missed event edges - so i changed it back to level-triggered. While that is not as fast as edge-triggered, it does not result in spurious hangs and workflow 'hickups' during the test.Could this be the reason why in your testing kevents outperformed epoll?
This is how I handle a read (write/accept/connect, same thing) inside coronet (coroutine+epoll async library - http://www.xmailserver.org/coronet-lib.html ).
static int conet_read_ll(struct sk_conn *conn, char *buf, int nbyte) {
int n;
while ((n = read(conn->sfd, buf, nbyte)) < 0) {
if (errno == EINTR)
continue;
if (errno != EAGAIN && errno != EWOULDBLOCK)
return -1;
if (!(conn->events & EPOLLIN)) {
conn->events = EPOLLIN;
if (conet_mod_conn(conn, conn->events) < 0)
return -1;
}
if (conet_yield(conn) < 0)
return -1;
}
return n;
}I use EPOLLET and, you don't change the interest set until you actually get an EAGAIN. *Many* read/write mode changes in the usage will simply happen w/out an epoll_ctl() needed. The conet_mod_conn() function does the actual epoll_ctl() and add EPOLLET to the specified event set. The conet_yield() function end up calling the libpcl's co_resume(), that is basically a switch-to-next-coroutine-until-fd-becomes-ready (maps directly to a swapcontext).
That cuts 50+% of the epoll_ctl(EPOLL_CTL_MOD).
- Davide
Quoted 8 lines · Evgeniy Polyakov
Quoted 3 lines
Do you have any link where i could check the type of HTTP parsing and send transport you are (or will be) using? What type of http client are you using to measure, with precisely what options?
For example this ones (essentially the same, except that epoll and kevent are used):
http://tservice.net.ru/~s0mbre/archive/kevent/evserver_kevent.c
http://tservice.net.ru/~s0mbre/archive/kevent/evserver_epoll.c
Client is 'ab' with high number of connections and concurrency. For example for athlon64 3500 I used concurrency of 8000 connections and 80k total.
Signature · 1 line
Evgeniy Polyakov
Quoted 4 lines · Evgeniy Polyakov
Quoted 4 lines
Quoted 3 lines
Do you have any link where i could check the type of HTTP parsing and send transport you are (or will be) using? What type of http client are you using to measure, with precisely what options?
Quoted 8 lines
Quoted 4 lines
For example this ones (essentially the same, except that epoll and kevent are used):
http://tservice.net.ru/~s0mbre/archive/kevent/evserver_kevent.c
http://tservice.net.ru/~s0mbre/archive/kevent/evserver_epoll.cClient is 'ab' with high number of connections and concurrency. For example for athlon64 3500 I used concurrency of 8000 connections and 80k total.
ok. So it's:
ab -c 8000 -n 80000 http://yourserver/tmp/index.html
right? How large is index.html typically - the default 40960 bytes, and with a constructed HTTP reply header already included in the file?
Ingo