packet re-ordering on SMP machines.
It appears from my initial tests on 2.4.20-pre4, that packets sent on one port, and received on another port on the same machine (via a cross-over cable), can be re-ordered. I see about 2000 reordered packets per 5,000,000 packets sent (sending about 70,000 packets-per-second on a dual-port e1000 NIC.)
By re-ordered, I mean that a method called from process_backlog in dev.c is being handed packets in a different order than they are being poked into the driver with hard_start_xmit on the other interface. If each CPU can be running the process_backlog, then I can see how this could be happening.
1) Is this expected behaviour?
2) Is there any standard (ie configurable) way to enforce strict ordering on an SMP system?
3) If answer to 2 is no, would you all be interested in a patch that did allow strict ordering (if indeed I can figure out how to write one)?
Thanks,
Ben
Signature · 3 lines
Ben Greear <[email protected]> <Ben_Greear AT excite.com> President of Candela Technologies Inc http://www.candelatech.com ScryMUD: http://scry.wanfear.com http://scry.wanfear.com/~greear
Use IRQ affinity settings if you want strict packet receive ordering.
Quoted 7 lines · Ben Greear
By re-ordered, I mean that a method called from
process_backlogindev.cis being handed packets in a different order than they are being poked into the driver withhard_start_xmiton the other interface. If each CPU can be running theprocess_backlog, then I can see how this could be happening.1) Is this expected behaviour?
Yes
Quoted 2 lines
2) Is there any standard (ie configurable) way to enforce strict ordering on an SMP system?
No
Quoted 2 lines
3) If answer to 2 is no, would you all be interested in a patch that did allow strict ordering (if indeed I can figure out how to write one)?
You should never need it. Ethernet, hubs, switches, routers, internet backbones etc will all cause packet re-ordering. You should also expect the percentage of re-ordered frames on the net to rise and rise.
Quoted 19 lines · Alan Cox
Quoted 7 lines · Ben Greear
By re-ordered, I mean that a method called from
process_backlogindev.cis being handed packets in a different order than they are being poked into the driver withhard_start_xmiton the other interface. If each CPU can be running theprocess_backlog, then I can see how this could be happening.1) Is this expected behaviour?
Yes
Quoted 2 lines
2) Is there any standard (ie configurable) way to enforce strict ordering on an SMP system?
No
Quoted 7 lines
Quoted 2 lines
3) If answer to 2 is no, would you all be interested in a patch that did allow strict ordering (if indeed I can figure out how to write one)?
You should never need it. Ethernet, hubs, switches, routers, internet backbones etc will all cause packet re-ordering. You should also expect the percentage of re-ordered frames on the net to rise and rise.
I would like to detect the number of pkts that such backbone hardware does re-order, so if my end machine is also re-ordering, I cannot get valid numbers.
Thanks,
Ben
Quoted 8 lines
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Signature · 3 lines
Ben Greear <[email protected]> <Ben_Greear AT excite.com> President of Candela Technologies Inc http://www.candelatech.com ScryMUD: http://scry.wanfear.com http://scry.wanfear.com/~greear
Quoted 3 lines · Ben Greear
I would like to detect the number of pkts that such backbone hardware does re-order, so if my end machine is also re-ordering, I cannot get valid numbers.
Good point
From: Alan Cox <[email protected]>
Date: 25 Aug 2002 15:41:13 +0100
Quoted 2 lines
3) If answer to 2 is no, would you all be interested in a patch that did allow strict ordering (if indeed I can figure out how to write one)?
You should never need it. Ethernet, hubs, switches, routers, internet backbones etc will all cause packet re-ordering. You should also expect the percentage of re-ordered frames on the net to rise and rise.
But to be honest, NAPI was meant to dramatically reduce the "SMP specific" causes of packet reordering. When a driver does use NAPI, the reordering due to SMP goes way down.