New net features for added performance
Disclaimer: This is 2.5, repeat, 2.5 material.
I've talked about the following items with a couple people on this list in private. I wanted to bring these up again, to see if anyone has comments on the following suggested netdevice changes for the upcoming 2.5 development series of kernels.
1) Rx Skb recycling. It would be nice to have skbs returned to the driver after the net core is done with them, rather than have netif_rx free the skb. Many drivers pre-allocate a number of maximum-sized skbs into which the net card DMA's data. If netif_rx returned the SKB instead of freeing it, the driver could simply flip the DescriptorOwned bit for that buffer, giving it immediately back to the net card.
Advantages: A de-allocation immediately followed by a reallocation is eliminated, less L1 cache pollution during interrupt handling. Potentially less DMA traffic between card and host.
Disadvantages?
2) Tx packet grouping. If the net core has knowledge that more packets will be following the current one being sent to dev->hard_start_xmit(), it should pass that knowledge on to dev->hard_start_xmit(), either as an estimated number yet-to-be-sent, or just as a flag that "more is coming."
Advantages: This lets the net driver make smarter decisions about Tx interrupt mitigation, Tx buffer queueing, etc.
Disadvantages? Can this sort of knowledge be obtained by a netdevice right now, without any kernel modifications?
3) Slabbier packet allocation. Even though skb allocation is decently fast, you are still looking at an skb buffer head grab and a kmalloc, for each [dev_]alloc_skb call. I was wondering if it would be possible to create a helper function for drivers which would improve the hot-path considerably:
static struct skbuff *ether_alloc_skb (int size)
{
if (size >= preallocated_skb_list->skb->size) {
dequeue_skb_from_list()
if (preallocate_size < low_water_limit)
schedule_tasklet(refill_skb_list);
return skb;
}
return dev_alloc_skb(size);
}The skbs from this list would be allocated by a tasklet in the background to the maximum size requested by the ethernet driver. If you wanted to waste even more memory, you could allocate from per-CPU lists..
Disadvantages? Doing this might increase cache pollution due to increased code and data size, but I think the hot path is much improved (dequeue a properly sized, initialized, skb-reserved'd skb off a list) and would help mitigate the impact of sudden bursts of traffic.
Signature · 3 lines
Jeff Garzik | "You see, in this world there's two kinds of Building 1024 | people, my friend: Those with loaded guns MandrakeSoft | and those who dig. You dig." --Blondie
Quoted 5 lines · Jeff Garzik
Advantages: A de-allocation immediately followed by a reallocation is eliminated, less L1 cache pollution during interrupt handling. Potentially less DMA traffic between card and host.
Disadvantages?
You need a new mechanism to cope with low memory situations because the drivers can tie up quite a bit of memory (in fact you gave up unified memory management).
Quoted 5 lines
3) Slabbier packet allocation. Even though skb allocation is decently fast, you are still looking at an skb buffer head grab and a kmalloc, for each [dev_]alloc_skb call. I was wondering if it would be possible to create a helper function for drivers which would improve the hot-path considerably:
[...]
If you need such a horror it just means there is something wrong with slab. Better fix slab.
4) Better support for aligned RX by only copying the header, no the whole packet, to end up with an aligned IP header. Unless the driver knows about all protocol lengths this means the stack needs to support "parse header in this buffer, then switch to other buffer with computed offset for data"
-Andi
Quoted 7 lines · Andi Kleen
Quoted 4 lines · Jeff Garzik
Advantages: A de-allocation immediately followed by a reallocation is eliminated, less L1 cache pollution during interrupt handling. Potentially less DMA traffic between card and host.
Quoted 5 lines
Quoted 1 line
Disadvantages?
You need a new mechanism to cope with low memory situations because the drivers can tie up quite a bit of memory (in fact you gave up unified memory management).
I think you misunderstand.. netif_rx frees the skb. In this example:
netif_rx(skb); /* free skb of size PKT_BUF_SZ */ skb = dev_alloc_skb(PKT_BUF_SZ)
an alloc of a PKT_BUF_SZ'd skb immediately follows a free of a same-sized skb. 100% of the time.
It seems an obvious shortcut to me, to have __netif_rx or similar -clear- the skb head not free it. No changes to memory management or additional low memory situations created by this, AFAICS.
Quoted 4 lines
4) Better support for aligned RX by only copying the header, no the whole packet, to end up with an aligned IP header. Unless the driver knows about all protocol lengths this means the stack needs to support "parse header in this buffer, then switch to other buffer with computed offset for data"
This requires scatter-gather hardware support, right? If so, would this support only exist for checksumming hardware -- like the current zerocopy -- or would non-checksumming SG hardware like tulip be supported too?
Jeff
Signature · 3 lines
Jeff Garzik | "You see, in this world there's two kinds of Building 1024 | people, my friend: Those with loaded guns MandrakeSoft | and those who dig. You dig." --Blondie
Quoted 14 lines · Jeff Garzik
Quoted 7 lines · Andi Kleen
Quoted 4 lines · Jeff Garzik
Advantages: A de-allocation immediately followed by a reallocation is eliminated, less L1 cache pollution during interrupt handling. Potentially less DMA traffic between card and host.
Quoted 5 lines
Quoted 1 line
Disadvantages?
You need a new mechanism to cope with low memory situations because the drivers can tie up quite a bit of memory (in fact you gave up unified memory management).
Quoted 7 lines
I think you misunderstand..
netif_rxfrees the skb. In this example:
netif_rx(skb); /* free skb of sizePKT_BUF_SZ*/ skb =dev_alloc_skb(PKT_BUF_SZ)an alloc of a PKT_BUF_SZ'd skb immediately follows a free of a same-sized skb. 100% of the time.
Free/Alloc gives the mm the chance to throttle it by failing, and also to recover from fragmentation by packing the slabs. If you don't do it you need to add a hook somewhere that gets triggered on low memory situations and frees the buffers.
Quoted 9 lines
Quoted 4 lines
4) Better support for aligned RX by only copying the header, no the whole packet, to end up with an aligned IP header. Unless the driver knows about all protocol lengths this means the stack needs to support "parse header in this buffer, then switch to other buffer with computed offset for data"
This requires scatter-gather hardware support, right? If so, would this support only exist for checksumming hardware -- like the current zerocopy -- or would non-checksumming SG hardware like tulip be supported too?
It doesn't need any hardware support. In fact it is especially helpful for the tulip. The idea is that instead of copying the whole packet to get an aligned header (e.g. on the alpha or other boxes where unaligned accesses are very expensive) you just copy the first 128 byte that probably contain the header. For the data it doesn't matter much if it's unaligned; copy_to_user and csum_copy_to_user can deal with that fine.
-Andi
Quoted 20 lines · Jeff Garzik
Quoted 7 lines · Andi Kleen
Quoted 4 lines · Jeff Garzik
Advantages: A de-allocation immediately followed by a reallocation is eliminated, less L1 cache pollution during interrupt handling. Potentially less DMA traffic between card and host.
Quoted 5 lines
Quoted 1 line
Disadvantages?
You need a new mechanism to cope with low memory situations because the drivers can tie up quite a bit of memory (in fact you gave up unified memory management).
I think you misunderstand..
netif_rxfrees the skb. In this example:
netif_rx(skb); /* free skb of sizePKT_BUF_SZ*/ skb =dev_alloc_skb(PKT_BUF_SZ)
Quoted 6 lines
an alloc of a PKT_BUF_SZ'd skb immediately follows a free of a same-sized skb. 100% of the time.
It seems an obvious shortcut to me, to have
__netif_rxor similar -clear- the skb head not free it. No changes to memory management or additional low memory situations created by this, AFAICS.
Sorry... I should also point out that I was thinking of tulip architecture and similar architectures, where you have a fixed number of Skbs allocated at all times, and that number doesn't change for the lifetime of the driver.
Clearly not all cases would benefit from skb recycling, but there are a number of rx-ring-based systems where this would be useful, and (AFAICS) reduce the work needed to be done by the system, and reduce the amount of overall DMA traffic by a bit.
Jeff
Signature · 3 lines
Jeff Garzik | "You see, in this world there's two kinds of Building 1024 | people, my friend: Those with loaded guns MandrakeSoft | and those who dig. You dig." --Blondie
Quoted 9 lines · Jeff Garzik
Sorry... I should also point out that I was thinking of tulip architecture and similar architectures, where you have a fixed number of Skbs allocated at all times, and that number doesn't change for the lifetime of the driver.
Clearly not all cases would benefit from skb recycling, but there are a number of rx-ring-based systems where this would be useful, and (AFAICS) reduce the work needed to be done by the system, and reduce the amount of overall DMA traffic by a bit.
A simple way to do it currently is just to compare the new skb with the old one. If it is the same, do a shortcut. That should usually work out when the system has enough memory.
-Andi
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
"Jeff" == Jeff Garzik <[email protected]> writes:
Jeff> 1) Rx Skb recycling. It would be nice to have skbs returned to the Jeff> driver after the net core is done with them, rather than have netif_rx Jeff> free the skb. Many drivers pre-allocate a number of maximum-sized skbs Jeff> into which the net card DMA's data. If netif_rx returned the SKB Jeff> instead of freeing it, the driver could simply flip the DescriptorOwned Jeff> bit for that buffer, giving it immediately back to the net card.
Jeff> Disadvantages?
netif_rx() would have to copy the buffer.
Right now, it just puts it on the queue towards the BH. For it to return
the skb would require that all processing occur inside of netif_rx() (a la BSD), or that it copy the buffer.
Jeff> 3) Slabbier packet allocation. Even though skb allocation is decently Jeff> fast, you are still looking at an skb buffer head grab and a
I think that if you had this, and you also returned skb's to this list on
a per device basis (change skb->free, I think) instead of to the general pool, you probably eliminate your request #1.
] Train travel features AC outlets with no take-off restrictions|gigabit is no[ ] Michael Richardson, Solidum Systems Oh where, oh where has|problem with[ ] [email protected] www.solidum.com the little fishy gone?|PAX.port 1100[
] panic("Just another NetBSD/notebook using, kernel hacking, security guy"); [
Jeff Garzik wrote:
(about optimizing kernel network code for busmastering NIC's)
Quoted 1 line
Disclaimer: This is 2.5, repeat, 2.5 material.
Related question: are there any 100Mbit NICs with cpu's onboard? Something mainstream/affordable?(i.e. not 1G ethernet)
Just recently someone posted asking some technical question about ARMlinux for and intel card with 2 1G ports, 8 100M ports,
an onboard ARM cpu and 4 other uControllers... seems to me
that ultimately the networking code should go in that direction:
immagine having the *NIC* do most of this... no cache pollution problems...
Quoted 1 line · Jeff Garzik
Disclaimer: This is 2.5, repeat, 2.5 material.
[snip]
Quoted 10 lines
1) Rx Skb recycling. It would be nice to have skbs returned to the driver after the net core is done with them, rather than have
netif_rxfree the skb. Many drivers pre-allocate a number of maximum-sized skbs into which the net card DMA's data. Ifnetif_rxreturned the SKB instead of freeing it, the driver could simply flip the DescriptorOwned bit for that buffer, giving it immediately back to the net card.Advantages: A de-allocation immediately followed by a reallocation is eliminated, less L1 cache pollution during interrupt handling. Potentially less DMA traffic between card and host.
This could be quite useful for the network driver I maintain (it's made it to the -ac patch set for 2.4, but not yet into the main kernel tarball). At the momement, it allocates 127 "buckets" (skb's under linux) at start of day and posts them to the card. After that, it maintains a minimum of 80 data buffers available to the card at any one time. There's a noticable performance hit when the driver has to reallocate new skbs to keep above the threshold. I try to recycle as much as possible w/in the driver (i.e. really small incoming packets get a new skb allocated for them and the original buffer is put back on the queue), but it would be nice to be able to recycle even more of the skbs.
Quoted 1 line
Disadvantages?
As has been pointed out, there's a certain loss of control over allocation of memory (could check for low memory conditions before sending the skb back to the driver, but . . .). I do see a failure to allocate all 127 skbs, occasionally, when the driver is first loaded (only way to get around this is to reboot the system).
Quoted 11 lines
2) Tx packet grouping. If the net core has knowledge that more packets will be following the current one being sent to
dev->hard_start_xmit(), it should pass that knowledge on todev->hard_start_xmit(), either as an estimated number yet-to-be-sent, or just as a flag that "more is coming."Advantages: This lets the net driver make smarter decisions about Tx interrupt mitigation, Tx buffer queueing, etc.
Disadvantages? Can this sort of knowledge be obtained by a netdevice right now, without any kernel modifications?
In my experience, Tx interrupt mitigation is of little benefit. I actually saw a performance increase of ~20% when I turned off Tx interrupt mitigation in my driver (could have been poor implementation on my part).
--
Noah Romer |"Calm down, it's only ones and zeros." - this message [email protected] |brought to you by The Network PGP key available |"Time will have its say, it always does." - Celltrex by finger or email |from Flying to Valhalla by Charles Pellegrino
On Sat, Feb 24, 2001 at 09:32:59PM -0500, Jeremy Jackson wrote:
Related question: are there any 100Mbit NICs with cpu's onboard?
Yes, but the only ones I've seen to date are magic and do special things (like VPN or hardware crypto). I'm not sure without 'magic' requirements there is much point for 100M on modern hardware.
Not affordable and whilst moving some of the IP stack onto the card (I think this is what are alluding to) would be extremely non-trivial especially if you want all the components (host OS, multiple networks cards) to talk to each other asynchronously and you would all have to deal with buggy hardware that doesn't like doing PCI-PCI transfers and such like.
That said, it would be an extemely neat thing to do from a technical perspective, but I don't know if you would ever get really good performance from it.
--cw
Quoted 7 lines · Jeff Garzik
...
1) Rx Skb recycling.
...
2) Tx packet grouping.
...
3) Slabbier packet allocation.
Let's see what the profiler says. 10 seconds of TCP xmit followed by 10 seconds of TCP receive. 100 mbits/sec. Kernel 2.4.2+ZC.
c0119470 do_softirq 97 0.7132 c020e718 ip_output 99 0.3694 c020a2c8 ip_route_input 103 0.2893 c01fdc4c skb_release_data 113 1.0089 c021312c tcp_sendmsg 113 0.0252 c0129c64 kmalloc 117 0.3953 c0112efc __wake_up_sync 128 0.6667 c01fdd24 __kfree_skb 153 0.6071 c020e824 ip_queue_xmit 154 0.1149 c011be80 del_timer 163 2.2639 c0222fac tcp_v4_rcv 173 0.1022 c010a778 handle_IRQ_event 178 1.4833 c01127fc schedule 200 0.1259 c01d39f8 boomerang_rx 332 0.2823 c024284c csum_partial_copy_generic 564 2.2742 c01d2c84 boomerang_start_xmit 654 0.9033
c0242b3c __generic_copy_from_user 733 12.2167c01d329c boomerang_interrupt 910 0.8818c01071f4 poll_idle 41813 1306.6562
00000000 total 48901 0.0367
7088 non-idle ticks.
153+117+113 ticks in skb/memory type functions.
So, naively, the most which can be saved here by optimising the skb and memory usage is 5% of networking load. (1% of system load @100 mbps)
Total device driver cost is 27% of the networking load.
All the meat is in the interrupt load. The 3com driver transfers about three packets per interrupt. Here's the system load (dual CPU):
Doing 100mbps TCP send with netperf: 14.9%
Doing 100mbps TCP receive with netperf: 23.3%
When tx interrupt mitigation is disabled we get 1.5 packets per interrupt doing transmit:
Doing 100mbps TCP send with netperf: 16.1%
Doing 100mbps TCP receive with netperf: 24.0%
So a 2x reduction in interrupt frequency on TCP transmit has saved 1.2% of system load. That's 8% of networking load, and, presumably, 30% of the driver load. That all seems to make sense.
The moral?
- Tuning skb allocation isn't likely to make much difference.
- At the device-driver level the most effective thing is to reduce the number of interrupts.
- If we can reduce the driver cost to *zero*, we improve TCP efficiency by 27%.
- At the system level the most important thing is to rewrite applications to use sendfile(). (But Rx is more expensive than Tx, so even this ain't the main game).
I agree that batching skbs into hard_start_xmit() may allow some driver optimisations. Pass it a vector of skbs rather than one, and let it return an indication of how many were actually consumed. But we'd need to go through an exercise like the above beforehand - it may not be worth the protocol-level trauma.
I suspect that a thorough analysis of the best way to use Linux networking, and then a rewrite of important applications so they use the result of that analysis would pay dividends.
-
Quoted 1 line · Jeff Garzik
1) Rx Skb recycling.
Sounds like a potentially useful idea. To solve the most immediate memory pressure problems, maybe VM could provide some function that does a kfree in cases of memory shortage, and that does nothing otherwise, so the driver could offer to free the skb after netif_rx. You still need to go over the list in idle periods, though.
Quoted 1 line
2) Tx packet grouping.
Hmm, I think we need an estimate of how long a packet train you'd usually get. A flag looks reasonably inexpensive. Estimated numbers sound like over-engineering.
Quoted 2 lines
Disadvantages? Can this sort of knowledge be obtained by a netdevice right now, without any kernel modifications?
Question is what the hardware really needs. If you can change the interrupt point easily, it's probably cheapest to do all the work in hard_start_xmit.
Quoted 1 line
3) Slabbier packet allocation.
Hmm, this may actually be worse during bursts: if you burst exceeds the preallocated size, you have to perform more expensive/slower operations (e.g. running a tasklet) to refill your cache.
- Werner
Signature · 3 lines
_________________________________________________________________________ / Werner Almesberger, ICA, EPFL, CH [email protected] / /_IN_N_032__Tel_+41_21_693_6621__Fax_+41_21_693_6610_____________________/
Quoted 3 lines · Chris Wedgwood
That said, it would be an extemely neat thing to do from a technical perspective, but I don't know if you would ever get really good performance from it.
Well, you'd have to re-design the networking code to support NUMA architectures, with a fairly fine granularity. I'm not sure you'd gain anything except possibly for the forwarding fast path.
A cheaper, and probably more useful possibility is hardware assistance for specific operations. E.g. hardware-accelerated packet classification looks interesting. I'd also like to see hardware-assistance for shaping on other media than ATM.
- Werner
Signature · 3 lines
_________________________________________________________________________ / Werner Almesberger, ICA, EPFL, CH [email protected] / /_IN_N_032__Tel_+41_21_693_6621__Fax_+41_21_693_6610_____________________/
Quoted 6 lines · Jeremy Jackson
Jeff Garzik wrote:
(about optimizing kernel network code for busmastering NIC's)
Quoted 1 line
Disclaimer: This is 2.5, repeat, 2.5 material.
Quoted 7 lines
Related question: are there any 100Mbit NICs with cpu's onboard? Something mainstream/affordable?(i.e. not 1G ethernet)
Just recently someone posted asking some technical question about ARMlinux for and intel card with 2 1G ports, 8 100M ports,
an onboard ARM cpu and 4 other uControllers... seems to me
that ultimately the networking code should go in that direction:
immagine having the *NIC* do most of this... no cache pollution problems...
Dunno, but the latest Motorola ColdFire microcontroller has Ethernet built in. I think it's even 100baseTX, but I could be mistaken.
--------------------------------------------------------------
from: Jonathan "Chromatix" Morton mail: [email protected] (not for attachments) big-mail: [email protected] uni-mail: [email protected]
The key to knowledge is not to rely on people to teach you it.
Get VNC Server for Macintosh from http://www.chromatix.uklinux.net/vnc/
-----BEGIN GEEK CODE BLOCK-----
Version 3.12
GCS$/E/S dpu(!) s:- a20 C+++ UL++ P L+++ E W+ N- o? K? w--- O-- M++$ V? PS PE- Y+ PGP++ t- 5- X- R !tv b++ DI+++ D G e+ h+ r- y+
-----END GEEK CODE BLOCK-----
On Sun, Feb 25, 2001 at 01:41:56PM +0100, Werner Almesberger wrote:
Well, you'd have to re-design the networking code to support NUMA architectures, with a fairly fine granularity. I'm not sure you'd gain anything except possibly for the forwarding fast path.
I'm not convince for a general purpose OS you would gain anything at all; but an an intellectual exercise it's a fascinating idea.
I'd make a good PhD thesis.
--cw
Andrew Morton wrote:
(kernel profile of TCP tx/rx)So, naively, the most which can be saved here by optimising
Quoted 3 lines
the skb and memory usage is 5% of networking load. (1% of system load @100 mbps)
For a local tx/rx. (open question) What happens with a router box with netfilter and queueing? Perhaps this type of optimisation will help more in that case?
think about a box with 4 1G NICs being able to route AND do QoS per conntrack connection (ala RSVP and such)
Really what I'm looking for is something like SGI's STP (Scheduled Transfer Protocol). mmap your tcp recieve buffer, and have a card smart enough to figure out header alignment, (i.e. know header size based on protocol number) transfer only that, let the kernel process it, then tell the card to DMA the data from the buffer right into process memory. (or other NIC)
Make it possible to have the performance of a Juniper network processor + flexiblity of Linux.
In message <[email protected]> you write:
Quoted 11 lines
Quoted 5 lines · Jeff Garzik
Advantages: A de-allocation immediately followed by a reallocation is eliminated, less L1 cache pollution during interrupt handling. Potentially less DMA traffic between card and host.
Disadvantages?
You need a new mechanism to cope with low memory situations because the drivers can tie up quite a bit of memory (in fact you gave up unified memory management).
Also, you still need to "clean" the skb (it can hold device and nfct references).
Rusty.
--
Premature optmztion is rt of all evl. --DK
Quoted 1 line · Jeff Garzik
1) Rx Skb recycling.
...
Quoted 3 lines
Advantages: A de-allocation immediately followed by a reallocation is eliminated, less L1 cache pollution during interrupt handling. Potentially less DMA traffic between card and host.
...
Quoted 1 line
Disadvantages?
It simply cannot work, as Alexey stated, in normal circumstances netif_rx() queues until the user reads the data. This is the whole basis of our receive packet processing model within softint/user context.
Secondly, I can argue that skb recycling can give _worse_ cache performance. If the next use and access by the card to the skb data is deferred, this gives the cpu a chance to displace those lines in it's cache naturally via displacement instead of being forced quickly to do so when the device touches that data.
If the device forces the cache displacement, those cache lines become empty until filled with something later (smaller utilization of total cache contents) whereas natural displacement puts useful data into the cache at the time of the displacement (larger utilization of total cache contents).
It is an NT/windows driver API rubbish idea, and it is full crap.
Quoted 1 line
2) Tx packet grouping.
...
Quoted 1 line
Disadvantages?
See Torvalds vs. world discussion on this list about API entry points which pass multiple pages at a time versus simpler ones which pass only a single page at a time. :-)
Quoted 1 line
3) Slabbier packet allocation.
...
Quoted 4 lines
Disadvantages? Doing this might increase cache pollution due to increased code and data size, but I think the hot path is much improved (dequeue a properly sized, initialized, skb-reserved'd skb off a list) and would help mitigate the impact of sudden bursts of traffic.
I don't know what I think about this one, but my hunch is that it will lead to worse data packing via such an allocator.
Later,
David S. Miller
[email protected]
Quoted 1 line · Andi Kleen
4) Better support for aligned RX by only copying the header
Andi you can make this now:
1) Add new "post-header data pointer" field in SKB.
2) Change drivers to copy into aligned headroom as you mention, and they set this new post-header pointer as appropriate. For normal drivers without alignment problem, generic code sets the pointer up just like it does the rest of the SKB header pointers now.
3) Enforce correct usage of it in all the networking :-)
I would definitely accept such a patch for the 2.5.x series. It seems to be a nice idea and I currently see no holes in it.
Later,
David S. Miller
[email protected]
Quoted 7 lines · David S. Miller
Quoted 1 line · Andi Kleen
4) Better support for aligned RX by only copying the header
Andi you can make this now:
1) Add new "post-header data pointer" field in SKB.
That would imply to let the drivers parse all headers to figure out the length. I think it's better to have a "header base" and "data base" pointer. The driver would just copy some standard size that likely contains all of the header
When you're finished with the header use skb->database+(skb->hdrptr-skb->hdrbase) to get the start of data.
Or did I misunderstand you?
Quoted 1 line
3) Enforce correct usage of it in all the networking :-)
,) -- the tricky part.
-Andi
Quoted 8 lines · David S. Miller
Quoted 1 line · Jeff Garzik
2) Tx packet grouping.
...
Quoted 1 line
Disadvantages?
See Torvalds vs. world discussion on this list about API entry points which pass multiple pages at a time versus simpler ones which pass only a single page at a time. :-)
I only want to know if more are coming, not actually pass multiples..
Jeff
Signature · 3 lines
Jeff Garzik | "You see, in this world there's two kinds of Building 1024 | people, my friend: Those with loaded guns MandrakeSoft | and those who dig. You dig." --Blondie
Quoted 1 line · Andi Kleen
Or did I misunderstand you?
What is wrong with making methods, keyed off of the ethernet protocol ID, that can do the "I know where/how-long headers are" stuff for that protocol? Only cards with the problem call into this function vector or however we arrange it, and then for those that don't have these problems at all we can make NULL a special value for this "post-header" pointer.
You can pick some arbitrary number, sure, that is another way to do it. Such a size would need to be chosen very carefully though.
Later,
David S. Miller
[email protected]
Quoted 1 line · Jeff Garzik
I only want to know if more are coming, not actually pass multiples..
Ok, then my only concern is that the path from "I know more is coming" down to hard_start_xmit invocation is long. It would mean passing a new piece of state a long distance inside the stack from SKB origin to device.
Later,
David S. Miller
[email protected]
Quoted 10 lines · David S. Miller
Quoted 1 line · Andi Kleen
Or did I misunderstand you?
What is wrong with making methods, keyed off of the ethernet protocol ID, that can do the "I know where/how-long headers are" stuff for that protocol? Only cards with the problem call into this function vector or however we arrange it, and then for those that don't have these problems at all we can make NULL a special value for this "post-header" pointer.
I had a dream about a NIC that would do exactly the above by itsself. The dumb cards would use the above code, and the smart ones' drivers would overload the functions and allow the NIC to do it.
"Tell me of the waters of your homeworld, Usul" :)
Except the driver interacts differently than netif_rx... knowing the protocol it DMA's the header only(it knows the length then too)
(SMC's epic100's descriptors can do this, but the card can't do the de-mux on proto id, forcing the network core to run in the ISR so the card can finish DMA and not exhaust it's tiny memory.) The network code can
then do all the routing/netfilter/QoS stuff, and tell the card to DMA the payload into the TX queue of another NIC (or queue the header with a pointer to the payload in the PCI address space of the incoming NIC heh heh) OR into the process' mmap'ed TCP receive buffer ala SGI's STP.
Hello!
Quoted 3 lines
Quoted 1 line
3) Enforce correct usage of it in all the networking :-)
,) -- the tricky part.
No tricks, IP[v6] is already enforced to be clever; all the rest are free to do this, if they desire. And btw, driver need not to parse anything, but its internal stuff and even aligning eth II header can be made in eth_type_trans().
Actually, it is possible now not changing anything but driver. Fortunately, I removed stupid tulip from alpha, so that I have no impetus to try this myself. 8)
Alexey
Hi!
Quoted 7 lines
Quoted 2 lines
an alloc of a PKT_BUF_SZ'd skb immediately follows a free of a same-sized skb. 100% of the time.
Free/Alloc gives the mm the chance to throttle it by failing, and also to recover from fragmentation by packing the slabs. If you don't do it you need to add a hook somewhere that gets triggered on low memory situations and frees the buffers.
And what? It makes allocation longer lived. Our MM should survive that just fine.
Signature · 2 lines
Philips Velo 1: 1"x4"x8", 300gram, 60, 12MB, 40bogomips, linux, mutt, details at http://atrey.karlin.mff.cuni.cz/~pavel/velo/index.html.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Jeff" == Jeff Garzik <[email protected]> writes:
Jeff> 1) Rx Skb recycling. It would be nice to have skbs returned to Jeff> the driver after the net core is done with them, rather than Jeff> have netif_rx free the skb. Many drivers pre-allocate a number Jeff> of maximum-sized skbs into which the net card DMA's data. If Jeff> netif_rx returned the SKB instead of freeing it, the driver Jeff> could simply flip the DescriptorOwned bit for that buffer, Jeff> giving it immediately back to the net card.
Jeff> Advantages: A de-allocation immediately followed by a Jeff> reallocation is eliminated, less L1 cache pollution during Jeff> interrupt handling. Potentially less DMA traffic between card Jeff> and host.
Jeff> Disadvantages?
I already tried this with the AceNIC GigE driver some time ago, and after Ingo came up with a per-CPU slab patch the gain was gone. I am not sure the complexity is worth it.
Jes
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Noah" == Noah Romer <[email protected]> writes:
Noah> In my experience, Tx interrupt mitigation is of little Noah> benefit. I actually saw a performance increase of ~20% when I Noah> turned off Tx interrupt mitigation in my driver (could have been Noah> poor implementation on my part).
You need to define performance increase here. TX interrupt coalescing can still be a win in the systems load department.
Jes
LILO error with 2.4.3-pre1...
In reply to Jeff GarzikHmm, needed 2.4.3-pre1 and went to install with LILO using 'lilo -v' and got this:
LILO version 21.4-4, Copyright (C) 1992-1998 Werner Almesberger 'lba32' extensions Copyright (C) 1999,2000 John Coffman
Reading boot sector from /dev/hda
Merging with /boot/boot.b
Boot image: /boot/vmlinuz-2.4.2
Added linux *
Boot image: /boot/vmlinuz-2.4.3-pre1
Fatal: geo_comp_addr: Cylinder number is too big (1274 > 1023)
Neato. I don't have time to dig through LILO source code right now, so here are my system specs:
Linux Distribution: RedHat 6.2 with all latest updates
Hard Disk: Maxtor 52049H3 (20GB) IDE
CPU: Dual PII-266MHz
RAM: 256MB PC100
Result of 'fdisk /dev/hda -l':
Disk /dev/hda: 255 heads, 63 sectors, 2491 cylinders
Units = cylinders of 16065 * 512 bytes
Device Boot Start End Blocks Id System
/dev/hda1 * 1 1513 12153141 83 Linux
/dev/hda2 1514 1530 136552+ 82 Linux swap
/dev/hda3 1531 2491 7719232+ 83 LinuxI have no idea why the 1023 limit is coming up considering 2.4.2 and LILO were working just fine together and I have a newer BIOS that has not problems detecting the driver properly. Go ahead, call me idiot :).
-Steve
Signature · 4 lines
Steven J. Hill - Embedded SW Engineer Public Key: 'http://www.cotw.com/pubkey.txt' FPR1: E124 6E1C AF8E 7802 A815 FPR2: 7D72 829C 3386 4C4A E17D
Re: LILO error with 2.4.3-pre1...
Quoted 3 lines · Steven J. Hill
I have no idea why the 1023 limit is coming up considering 2.4.2 and LILO were working just fine together and I have a newer BIOS that has not problems detecting the driver properly. Go ahead, call me idiot :).
OK, you're an idiot :). It only worked before because all the files that lilo used just happened to be below cylinder 1024. Your partition goes past cyl 1024 and your new kernel is using space above 1024. Find a version of lilo that can cope with cyl >= 1024 (is there one?) or move the kernel below cyl 1024. You might need to repartition your disk to get / all below 1024.
Re: LILO error with 2.4.3-pre1...
----- Original Message -----
From: "Keith Owens" <[email protected]>
To: <[email protected]>
Cc: <[email protected]>
Sent: Saturday, March 03, 2001 8:39 PM
Subject: Re: LILO error with 2.4.3-pre1...
Quoted 9 lines
Quoted 3 lines · Steven J. Hill
I have no idea why the 1023 limit is coming up considering 2.4.2 and LILO were working just fine together and I have a newer BIOS that has not problems detecting the driver properly. Go ahead, call me idiot :).
OK, you're an idiot :). It only worked before because all the files that lilo used just happened to be below cylinder 1024. Your partition goes past cyl 1024 and your new kernel is using space above 1024.
I would agree with this explanation.
Quoted 1 line
Find a version of lilo that can cope with cyl >= 1024 (is there one?)
Uh, the version he has can cope with this, see the following:
Quoted 2 lines
LILO version 21.4-4, Copyright (C) 1992-1998 Werner Almesberger 'lba32' extensions Copyright (C) 1999,2000 John Coffman
The lba32 extensions should take care of this, of course you have to add 'lba32' as a line in your lilo.conf before lilo actually uses them (and, I assume, the BIOS must support the LBA extensions, but it seems most modern ones do).
Give that a try. Works for me.
Later,
Tom
RE: LILO error with 2.4.3-pre1...
In reply to Steven J. HillQuoted 1 line
'lba32' extensions Copyright (C) 1999,2000 John Coffman
^^^^^^
Add lba32 as the top line in lilo.conf. Re-run lilo.
Quoted 1 line
Fatal:
geo_comp_addr: Cylinder number is too big (1274 > 1023)
Before 2.4.3pre1, your kernel just happened to toss itself below cylinder 1024.
Quoted 1 line
Go ahead, call me idiot :).
Idiot. :-)
--
Regards,
Andre Tomt
Re: LILO error with 2.4.3-pre1...
Quoted 6 lines · Andre Tomt
Quoted 2 lines
'lba32' extensions Copyright (C) 1999,2000 John Coffman
^^^^^^Add lba32 as the top line in lilo.conf. Re-run lilo.
Quoted 9 lines
Quoted 1 line
Fatal:
geo_comp_addr: Cylinder number is too big (1274 > 1023)Before 2.4.3pre1, your kernel just happened to toss itself below cylinder 1024.
Quoted 1 line
Go ahead, call me idiot :).
Idiot. :-)
And since Andre was the last person to email me and call me an idiot, I will reply to his response :). Yup, that was the case and I added 'lba32' to my '/etc/lilo.conf' and things work great. I knew it was something simple, but I just don't pay attention to LILO much anymore. Thanks everyone.
-Steve
Signature · 4 lines
Steven J. Hill - Embedded SW Engineer Public Key: 'http://www.cotw.com/pubkey.txt' FPR1: E124 6E1C AF8E 7802 A815 FPR2: 7D72 829C 3386 4C4A E17D
Re: LILO error with 2.4.3-pre1...
In reply to Steven J. HillQuoted 9 lines
LILO version 21.4-4, Copyright (C) 1992-1998 Werner Almesberger 'lba32' extensions Copyright (C) 1999,2000 John Coffman
Boot image:
/boot/vmlinuz-2.4.3-pre1
Fatal:geo_comp_addr: Cylinder number is too big (1274 > 1023)I have no idea why the 1023 limit is coming up considering 2.4.2 and LILO were working just fine together and I have a newer BIOS that has not problems detecting the driver properly. Go ahead, call me idiot :).
You need to specify the lba32 option in your config
Re: LILO error with 2.4.3-pre1...
In reply to Steven J. HillQuoted 6 lines · Keith Owens
Quoted 3 lines · Steven J. Hill
I have no idea why the 1023 limit is coming up considering 2.4.2 and LILO were working just fine together and I have a newer BIOS that has not problems detecting the driver properly. Go ahead, call me idiot :).
Quoted 6 lines
OK, you're an idiot :). It only worked before because all the files that lilo used just happened to be below cylinder 1024. Your partition goes past cyl 1024 and your new kernel is using space above 1024. Find a version of lilo that can cope with cyl >= 1024 (is there one?) or move the kernel below cyl 1024. You might need to repartition your disk to get / all below 1024.
Call me idiot too but please explain what is wrong here:
# cat /etc/lilo.conf boot = /dev/hda timeout = 150 vga = 4 ramdisk = 0 lba32 append = "hdc=scsi" prompt
image = /boot/vmlinuz-2.4.2
root = /dev/hda2
read-only
label = Linux
other = /dev/hda3
label = win
table = /dev/hda
# fdisk -l /dev/hda
Disk /dev/hda: 255 heads, 63 sectors, 1650 cylinders Units = cylinders of 16065 * 512 bytes
Device Boot Start End Blocks Id System /dev/hda1 1 17 136521 82 Linux swap /dev/hda2 18 1165 9221310 83 Linux /dev/hda3 * 1166 1650 3895762+ c Win95 FAT32 (LBA) root@taz:~# lilo -v LILO version 21.7, Copyright (C) 1992-1998 Werner Almesberger Linux Real Mode Interface library Copyright (C) 1998 Josh Vanderhoof Development beyond version 21 Copyright (C) 1999-2001 John Coffman Released 24-Feb-2001 and compiled at 18:31:02 on Mar 3 2001.
Reading boot sector from /dev/hda
Merging with /boot/boot.b
Boot image: /boot/vmlinuz-2.4.2
Added Linux *
Boot other: /dev/hda3, on /dev/hda, loader /boot/chain.b Device 0x0300: Invalid partition table, 3rd entry
3D address: 63/254/141 (2281229)
Linear address: 1/0/1165 (18715725)
Mar 2 20:26:29 taz kernel: hda: IBM-DJNA-371350, ATA DISK drive Mar 2 20:26:29 taz kernel: hda: 26520480 sectors (13578 MB) w/1966KiB Cache, CHS=1650/255/63
Is anybody able to explain the error?
That partition contains a valid VFAT partition with win98se installed on it (and it works fine, ofc if I remove lilo from MBR).
Signature · 3 lines
Mircea Damian E-mails: [email protected], [email protected] WebPage: http://taz.mania.k.ro/~dmircea/
Re: LILO error with 2.4.3-pre1...
In reply to Steven J. HillQuoted 1 line · Mircea Damian
Call me idiot too but please explain what is wrong here:
What is wrong is that this is the kernel list, not the LILO list.
Quoted 5 lines
root@taz:~# lilo -v LILO version 21.7, Copyright (C) 1992-1998 Werner Almesberger Device 0x0300: Invalid partition table, 3rd entry 3D address: 63/254/141 (2281229) Linear address: 1/0/1165 (18715725)
Read the README in the LILO distribution.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Werner" == Werner Almesberger <[email protected]> writes:
Quoted 1 line · Werner> Jeff Garzik wrote:
Quoted 1 line
3) Slabbier packet allocation.
Werner> Hmm, this may actually be worse during bursts: if you burst Werner> exceeds the preallocated size, you have to perform more Werner> expensive/slower operations (e.g. running a tasklet) to refill Werner> your cache.
You may want to look at how I did this in the acenic driver. If the water mark goes below a certain level I schedule the tasklet, if it gets below an urgent watermark I do the allocation in the interrupt handler itself.
This is of course mainly useful for cards which give you deep queues.
Jes