TOE brain dump
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 5 lines · Werner Almesberger
- instead of putting a different stack on the TOE, a general-purpose processor (probably with some enhancements, and certainly with optimized data paths) is added to the NIC
Modern NPUs generally do this.
You need to have something like this to handle e.g. routing of GE
traffic.
Check for example:
http://www.vitesse.com/products/categories.cfm?family_id=5&category_id=16
Quoted 1 line · Ihar 'Philips' Filipau
Modern NPUs generally do this.
Unfortunately, they don't - they run *some* code, but that is rarely a Linux kernel, or a substantial part of it.
- Werner
Signature · 3 lines
_________________________________________________________________________ / Werner Almesberger, Buenos Aires, Argentina [email protected] / /_http://www.almesberger.net/____________________________________________/
Quoted 8 lines · Werner Almesberger
Quoted 1 line · Ihar 'Philips' Filipau
Modern NPUs generally do this.
Unfortunately, they don't - they run *some* code, but that is rarely a Linux kernel, or a substantial part of it.
Embedded CPU we are using is based MIPS, and has a lot of specialized
instructions.
It makes not that much sense to run kernel (especially Linux) on CPU
which is optimized for handling of network packets. (And has actually several co-processors to help in this task).
How much sense it makes to run general purpose OS (optimized for PCs
and servers) on devices which can make only couple of functions? (and no MMU btw)
It is a whole idea behind this kind of CPUs - to do a few of
functions - but to do them good.
If you will start stretching CPUs like this to fit Linux kernel - it
will generally just increase price. Probably there are some markets which can afford this.
Remeber - "Small is beatiful" (c) - and linux kernel far from it.
Our routing code which handles two GE interfaces (actually not pure
GE, but up to 2.5GB) fits into 3k. 3k of code - and that's it. not 650kb of bzip compressed bloat. And it handles two interfaces, handles fast data path from siblign interfaces, handles up to 1E6 routes. 3k of code. not 650k of bzip.
Quoted 22 lines · Ihar 'Philips' Filipau
Quoted 5 lines · Werner Almesberger
Quoted 1 line · Ihar 'Philips' Filipau
Modern NPUs generally do this.
Unfortunately, they don't - they run *some* code, but that is rarely a Linux kernel, or a substantial part of it.
Embedded CPU we are using is based MIPS, and has a lot of specialized
instructions.
It makes not that much sense to run kernel (especially Linux) on CPU
which is optimized for handling of network packets. (And has actually several co-processors to help in this task).
How much sense it makes to run general purpose OS (optimized for PCs
and servers) on devices which can make only couple of functions? (and no MMU btw)
It is a whole idea behind this kind of CPUs - to do a few of
functions - but to do them good.If you will start stretching CPUs like this to fit Linux kernel - it
will generally just increase price. Probably there are some markets which can afford this.
Quoted 6 lines
Remeber - "Small is beatiful" (c) - and linux kernel far from it. Our routing code which handles two GE interfaces (actually not pure
GE, but up to 2.5GB) fits into 3k. 3k of code - and that's it. not 650kb of bzip compressed bloat. And it handles two interfaces, handles fast data path from siblign interfaces, handles up to 1E6 routes. 3k of code. not 650k of bzip.
And it handles ipfilter?
and LSM security hooks?
how about IPSec?
and IPv6?
I don't think so.
Quoted 3 lines · Ihar 'Philips' Filipau
It makes not that much sense to run kernel (especially Linux) on CPU
which is optimized for handling of network packets. (And has actually several co-processors to help in this task).
All you need to do is to make the CPU capable of running the kernel (well, some of it), but it doesn't have to be particularly good at running anything but the TCP/IP code. And you can still benefit from most of the features of NPUs, such as a specialized memory architecture, parallel data paths, accelerated operations, etc.
- Werner
Signature · 3 lines
_________________________________________________________________________ / Werner Almesberger, Buenos Aires, Argentina [email protected] / /_http://www.almesberger.net/____________________________________________/
Quoted 10 lines · Werner Almesberger
Quoted 3 lines · Ihar 'Philips' Filipau
It makes not that much sense to run kernel (especially Linux) on CPU
which is optimized for handling of network packets. (And has actually several co-processors to help in this task).All you need to do is to make the CPU capable of running the kernel (well, some of it), but it doesn't have to be particularly good at running anything but the TCP/IP code. And you can still benefit from most of the features of NPUs, such as a specialized memory architecture, parallel data paths, accelerated operations, etc.
also how many of the standard kernel features could you turn off? do you really need filesystems for example?
could userspace be eliminated? (if you have some way to give the config commands to the kernel on the NIC and get the log messages back to the main kernel what else do you need?)
a lot of the other IO buffer stuff can be trimmed back (as per config_embedded)
what else could be done to use the kernel features taht are wanted without bringin extra baggage along?
David Lang
Quoted 1 line · David Lang
also how many of the standard kernel features could you turn off?
You don't turn them off - you just don't run them. What I'm suggesting is not a separate system that runs a stripped-down Linux kernel, but rather a device that looks like another node in a NUMA system.
There might be a point in completely excluding subsystems that will never be used on that NIC anyway, but that's already an optimization.
- Werner
Signature · 3 lines
_________________________________________________________________________ / Werner Almesberger, Buenos Aires, Argentina [email protected] / /_http://www.almesberger.net/____________________________________________/
Quoted 11 lines · Werner Almesberger
Quoted 1 line · David Lang
also how many of the standard kernel features could you turn off?
You don't turn them off - you just don't run them. What I'm suggesting is not a separate system that runs a stripped-down Linux kernel, but rather a device that looks like another node in a NUMA system.
There might be a point in completely excluding subsystems that will never be used on that NIC anyway, but that's already an optimization.
I would think that it's much more difficult to run NUMA across different types of CPU's then it would be to run a seperate kernel on the NIC.
I'm thinking clustering instead of single-system-image.
David Lang
Quoted 2 lines · David Lang
I would think that it's much more difficult to run NUMA across different types of CPU's
I'd view this as a new and interesting challenge :-) Besides, if one would use Alan's idea, and just use an amd64, or such, the CPUs wouldn't be all that different in the end.
One added benefit of using similar CPUs would be that also bits of user space (e.g. a copy loop) could migrate to the NIC.
Quoted 1 line
then it would be to run a seperate kernel on the NIC.
Yes, but that separate kernel would need new administrative interfaces, and things like route changes would be difficult to handle. (That is, if you still want this to appear as a single system to user space.) It would certainly be better that running a completely proprietary solution, but you still get a few nasty problems.
- Werner
Signature · 3 lines
_________________________________________________________________________ / Werner Almesberger, Buenos Aires, Argentina [email protected] / /_http://www.almesberger.net/____________________________________________/
Quoted 7 lines · Werner Almesberger
Quoted 2 lines · David Lang
I would think that it's much more difficult to run NUMA across different types of CPU's
I'd view this as a new and interesting challenge :-) Besides, if one would use Alan's idea, and just use an amd64, or such, the CPUs wouldn't be all that different in the end.
you missed Alan's point, he was saying you don't do TOE on the NIC, you just add another CPU to your main system and use non-TOE NIC's the way you do today.
Quoted 8 lines
Quoted 1 line
then it would be to run a seperate kernel on the NIC.
Yes, but that separate kernel would need new administrative interfaces, and things like route changes would be difficult to handle. (That is, if you still want this to appear as a single system to user space.) It would certainly be better that running a completely proprietary solution, but you still get a few nasty problems.
Any time you create a cluster of machines you want to create som nice administrative interfaces for them to maintain your own sanity (you don't think sysadmins login to every machine on a 1000 node beowolf cluster do you :-)
in trying to run a single kernel across different types of CPU's you run into some really nasty problems (different machine code, and even if it's the same family of processor it could require very different optimizations, imagine the two processor types useing different word lengths or endian order)
Larry McVoy has the right general idea when he says buy another box to do the job, he is just missing the idea that there are some advantages of coupling the cluster more tightly then you can do with a seperate box.
David Lang
Quoted 1 line · David Lang
you missed Alan's point, he was saying you don't do TOE on the NIC,
Only as far as "traditional TOE" is concerned. My idea is precisely to avoid treating TOE as a special case.
Quoted 2 lines
just add another CPU to your main system and use non-TOE NIC's the way you do today.
For a start, that may be good enough, even though you miss a lot of nice hardware optimizations.
Quoted 2 lines
Any time you create a cluster of machines you want to create som nice administrative interfaces for them to maintain your own sanity
You've got a point there. The question is whether these interface really cover everything we need, and - more importantly - whether they still have the same semantics.
Quoted 3 lines
Larry McVoy has the right general idea when he says buy another box to do the job, he is just missing the idea that there are some advantages of coupling the cluster more tightly then you can do with a seperate box.
Clusters are nice, but they don't help if your bottleneck is per-packet processing overhead with a single NIC, or if you can't properly distribute the applications.
I'm not saying that TOE, even if done in a maintainable way, is always the right approach. E.g. if all you need is a fast path to main memory, Dave's flow cache would be a much cheaper solution. If you can distribute the workload, and the extra hardware doesn't bother you, your clusters become attractive.
- Werner
Signature · 3 lines
_________________________________________________________________________ / Werner Almesberger, Buenos Aires, Argentina [email protected] / /_http://www.almesberger.net/____________________________________________/
Quoted 11 lines · Werner Almesberger
Quoted 1 line · David Lang
you missed Alan's point, he was saying you don't do TOE on the NIC,
Only as far as "traditional TOE" is concerned. My idea is precisely to avoid treating TOE as a special case.
Quoted 2 lines
just add another CPU to your main system and use non-TOE NIC's the way you do today.
For a start, that may be good enough, even though you miss a lot of nice hardware optimizations.
exactly, Alan is saying that the hardware optimizations aren't nessasary. putting an Opteron on a NIC card just to match the other processors in your system seems like a huge amount of overkill. you aren't going to have nearly the same access to memory so that processor will be crippled, but stil cost full price (and then some, remember you have to supply the thing with power and cool it)
Quoted 6 lines
Quoted 2 lines
Any time you create a cluster of machines you want to create som nice administrative interfaces for them to maintain your own sanity
You've got a point there. The question is whether these interface really cover everything we need, and - more importantly - whether they still have the same semantics.
as long as tools are written that have the same command line semantics the rest of the complexity can be hidden. and even this isn't strictly nessasary, these are special purpose cards and a special procedure for configuring them isn't unreasonable.
Quoted 8 lines
Quoted 3 lines
Larry McVoy has the right general idea when he says buy another box to do the job, he is just missing the idea that there are some advantages of coupling the cluster more tightly then you can do with a seperate box.
Clusters are nice, but they don't help if your bottleneck is per-packet processing overhead with a single NIC, or if you can't properly distribute the applications.
Quoted 6 lines
I'm not saying that TOE, even if done in a maintainable way, is always the right approach. E.g. if all you need is a fast path to main memory, Dave's flow cache would be a much cheaper solution. If you can distribute the workload, and the extra hardware doesn't bother you, your clusters become attractive.
I'm saying treat the one machine with 10 of these specialty NIC's in it as a 11 machine cluster, one machne running your server software and 10 others running your networking.
David Lang
I'd suggest that all of you look at the fact that all of these offload card companies have ended up dieing. I don't know of a single one that made it to profitability. Doesn't that tell you something? What has changed that makes this a good idea?
Signature · 2 lines
--- Larry McVoy lm at bitmover.com http://www.bitmover.com/lm
Quoted 4 lines · Larry McVoy
I'd suggest that all of you look at the fact that all of these offload card companies have ended up dieing. I don't know of a single one that made it to profitability. Doesn't that tell you something? What has changed that makes this a good idea?
1) So far, most of the battle has been about data transfers. Now, per-packet overhead is becoming an issue.
2) AFAIK, they all went for designs that isolated their code from the main stack. That's one thing that, IMHO, has to change.
Is this enough to make TOE succeed ? I don't know.
- Werner
Signature · 3 lines
_________________________________________________________________________ / Werner Almesberger, Buenos Aires, Argentina [email protected] / /_http://www.almesberger.net/____________________________________________/
Quoted 6 lines · Werner Almesberger
Quoted 4 lines · Larry McVoy
I'd suggest that all of you look at the fact that all of these offload card companies have ended up dieing. I don't know of a single one that made it to profitability. Doesn't that tell you something? What has changed that makes this a good idea?
Quoted 8 lines
1) So far, most of the battle has been about data transfers. Now, per-packet overhead is becoming an issue.
2) AFAIK, they all went for designs that isolated their code from the main stack. That's one thing that, IMHO, has to change.
Is this enough to make TOE succeed ? I don't know.
Jeff Mogul recently wrote an interesting paper called "TCP Offload is a Dumb Idea Whose Time Has Come":
http://www.usenix.org/events/hotos03/tech/full_papers/mogul/mogul_html/
(It's 6 pages long and in HTML - easy to read.)
After explaining why TCP offload is a dumb idea, he goes on to argue that *if* storage area networks are replaced with switched ethernet, and RDMA becomes popular, TCP offload might make sense for sending data to your disks.
This is a good, short paper to read if you are interested in TOE for any reason.
-VAL
Quoted 1 line · David Lang
exactly, Alan is saying that the hardware optimizations aren't nessasary.
Eventually, you'll want them, and if it's only to lower the chip or pin count.
Quoted 4 lines
putting an Opteron on a NIC card just to match the other processors in your system seems like a huge amount of overkill. you aren't going to have nearly the same access to memory so that processor will be crippled, but stil cost full price
You might be able to get them for free ;-) Just pick the rejects where the FPU or such doesn't quite work. Call it amd64sx :-)
But even if you get regular CPUs, they're not *that* expensive. Particularly not for a first generation design.
Quoted 2 lines
(and then some, remember you have to supply the thing with power and cool it)
Yes, this, chip count, and chip surface are what make me feel queasy when thinking of somebody using something as powerful as an amd64.
Quoted 2 lines
as long as tools are written that have the same command line semantics the rest of the complexity can be hidden.
You want to be API and probably even ABI-compatible, so that user-space demons (routing, management, etc.) work, too.
Quoted 3 lines
and even this isn't strictly
nessasary, these are special purpose cards and a special procedure for configuring them isn't unreasonable.
I'd think thrice before buying a card that requires me to change my entire network management system - and change it again, if I ever decide to switch brands, or if the next generation of that special NIC gets a little more special.
Quoted 3 lines
I'm saying treat the one machine with 10 of these specialty NIC's in it as a 11 machine cluster, one machne running your server software and 10 others running your networking.
You can probably afford rather fancy TOE hardware for the price of ten cluster nodes, a high-speed LAN to connect your cluster, and a switch that connects the high-speed link to the ten not-quite-so-high-speed links.
Likewise for power, cooling, and space.
And that's still assuming you can actually distribute all this.
- Werner
Signature · 3 lines
_________________________________________________________________________ / Werner Almesberger, Buenos Aires, Argentina [email protected] / /_http://www.almesberger.net/____________________________________________/
One thing that you could do *if* you cared to go to a SYSVr4 streams-like approach is just to push *some* of the TCP/IP stack onto the card, as one or more streams modules.
Peter C