Partitioning the kernel
I'm wondering whether we have or need a formalisation of how work might be divided in future kernels.
The question I'm interested in is how the work gets split up among various components at different levels within a single box (not SMP with many at the same level, or various multi-box techniques), in particular how you separate computation and I/O given some intelligence in devices other than the main CPU (or SMP set).
There are a bunch of examples to look at:
IBM mainframe
"channel processors" do all the I/O
main CPU sets up a control block, does an EXCP instruction there's an interrupt when operation completes or fails
VAX 782: basically two 780s with a big cable between busses
one has disk controllers, most of the (VMS) kernel other has serial I/O, runs all user processes
various smart network or disk controllers and really smart ones that do RAID or Crypto
I2O stuff on newer PCs
Larry McVoy's suggestion that the right way to run, say, a 32-CPU
box is with something like 8 separate kernels, each using 4 CPUs
If one of those runs the file system for everyone, this somewhat
overlaps the techniques listed above.
All of these demonstrably work, but each partitions the work between processors in a somewhat different way.
What I'm wondering is whether, given that many drivers have a top-half vs. bottom-half split as a fairly basic part of their design, it would make sense to make it a design goal to have a clean partition at that boundary.
On well-endowed systems, you then have the main CPUs running the top half of everything, while I2O processors handle all the bottom halves and the I/O interrupts. On lesser boxes, the CPU does both halves.
It seems to me this might give a cleaner design than one where the work is partitioned between devices at some other boundary.
If the locks you need between top and bottom halves of the driver are also controlling most or all CPU-to-I2O communication, it might go some way toward avoiding the "locking cliff" McVoy talks of.
Quoted 42 lines · Sandy Harris
I'm wondering whether we have or need a formalisation of how work might be divided in future kernels.
The question I'm interested in is how the work gets split up among various components at different levels within a single box (not SMP with many at the same level, or various multi-box techniques), in particular how you separate computation and I/O given some intelligence in devices other than the main CPU (or SMP set).
There are a bunch of examples to look at:
IBM mainframe
"channel processors" do all the I/O
main CPU sets up a control block, does an EXCP instruction there's an interrupt when operation completes or failsVAX 782: basically two 780s with a big cable between busses
one has disk controllers, most of the (VMS) kernel other has serial I/O, runs all user processesvarious smart network or disk controllers and really smart ones that do RAID or Crypto
I2O stuff on newer PCs
Larry McVoy's suggestion that the right way to run, say, a 32-CPU
box is with something like 8 separate kernels, each using 4 CPUs
If one of those runs the file system for everyone, this somewhat
overlaps the techniques listed above.All of these demonstrably work, but each partitions the work between processors in a somewhat different way.
What I'm wondering is whether, given that many drivers have a top-half vs. bottom-half split as a fairly basic part of their design, it would make sense to make it a design goal to have a clean partition at that boundary.
On well-endowed systems, you then have the main CPUs running the top half of everything, while I2O processors handle all the bottom halves and the I/O interrupts. On lesser boxes, the CPU does both halves.
Quoted 6 lines
It seems to me this might give a cleaner design than one where the work is partitioned between devices at some other boundary.
If the locks you need between top and bottom halves of the driver are also controlling most or all CPU-to-I2O communication, it might go some way toward avoiding the "locking cliff" McVoy talks of.
A small cheap processor to do this with would be the ETRAX 100LX (LX = Linux) Put an ETRAX100LX (integrated IDE, ethernet, and ...) on an IDE controller. Telnet / SSH to your PCI boards :-)
Cheapest possible system might be one without a main CPU...
It would be possible to rebalance where to create the interface over time.
/RogerL
Signature · 3 lines
Roger Larsson Skellefteå Sweden