ABISS open issues, food for thought, etc.
We recently had some discussions on where ABISS could use some help. It turned out that we actually have a number of open construction sites ;-) This is also a bit of a status update on what's planned from Philips.
I've already posted an earlier version of this list a few weeks ago only to abiss-general. I'm cross-posting it again, to keep the list in the loop. This is also slightly updated.
1) support for guarantees for writing. At the moment, writing is always best-effort. I think a good part of the writing issues will be easy to solve once we better control memory management, see below.
2) elimination of stray delays. Various people are already working on limiting long non-preemptible execution paths in the kernel, either by searching for them and explicitly making them preemptible, or by using the SMP support infrastructure to automatically add many preemption points. So I think delays resulting from lack of preemption will disappear even if we don't do anything specifically for ABISS.
The other source of delays are things we have to wait for. Some of these things are expected, predictable, and we try to take care of them, e.g. disk IO. Other things pop up at random and are hard to avoid. I suspect that the single most significant source of the latter kind of delays is memory management.
We plan to be working on the MM type of delays. I have that idea that perhaps introducing relatively "hard" memory partitions could help to improve things, e.g. by making a process get all allocations explicitly and implicitly done for it (with exceptions, like networking, where allocations cannot always be easily associated with specific processes), from the same "partition", and doing all housekeeping work in the background such that it doesn't interfere with that process.
At the moment, the most promising approach seems to be to use virtual NUMA nodes for this. The idea is that an application would reserve its memory by assigning it to such a node and then bind itself to the node. This needs experimentation.
3) The API. At the moment, what we have is a mixture of what we think would be useful for applications, and things that theoretical reasoning tells us are required (in the context of the single-rate model (*) we're using). It would be good to get some feedback from the application side of whether this API is any good.
(*) I think it's fair to say that every additional parameter
increases the incomprehensibility of a QoS model by about one order of magnitude. Right now, we have two parameters (rate and buffer size) the user must supply. So the question is less of the "which additional parameters would be useful for your application" but more of the "how can we help you use the existing parameters" type.
The API only exposes parameters (hopefully :-) understood by the application, i.e. the data rate it wants and its own buffering needs. These parameters depend in what the application does and how it does it, so they can't be guessed from the outside. (One can of course always add another layer of abstraction that constrains application semantics, e.g. by assuming a specific file content and structure and/or providing access mechanisms, and which then can then also set the parameters.)
Additional parameters are then added (usually they just increase the buffering) by the kernel and the ABISS daemon (see below).
4) Resource management. This needs a significant amount of work. The current architecture is very flexible in that the user-space ABISS daemon has full control over what happens with requests, and can implement its own admission scheme.
Real-world admission control has to consist of at least resource control (centered around a suitable resource model) and administrative policy. The latter may or may not take resource information into account. (E.g. "only allow ABISS to take up to 50% of the disk bandwidth" would be an example for taking resource information into account, "only let user X use priorities" would be an example for not needing resource information.)
The resource model depends on various parameters that we currently don't consider, such as drive behaviour. Also, contrary to what abissd currently implements, resources aren't always additive. (E.g. if you read a disk at 1 MB/sec, the maximum bandwidth remaining for other users of the same disk is usually decreased by a lot more than 1 MB/sec, because of the seeks and rotational delays we're introducing.)
We currently don't have any plans to work on the admission control side, so help there is desperately needed.
There's also the issue of modifying parameters after initial admission control. E.g. the buffer size used in the kernel is a function of what the application requested and a number of other parameters. If some of the latter change, we may have to modify in-kernel parameters. We're experimenting with that right now.
5) Large contiguous allocations should also help to make it easier to obtain tight access latency bounds. However, it's not quite clear which part of the system should try to make such allocations happen. At the moment, ABISS has fairly complex code that lets it take over allocation policy (currently only for FAT), so it can also make large allocations, but I'd rather see this handled by the file system itself, or perhaps a separate subsystem shared by the file systems (similar to how journaling is implemented).
Various file systems (ext3, XFS, affs) already implement some from of reservation or preallocation.
- Werner
Signature · 3 lines
_________________________________________________________________________ / Werner Almesberger, Buenos Aires, Argentina [email protected] / /_http://www.almesberger.net/____________________________________________/