NAPI Race?
It looks to me like net_rx_action() might suffer from a race, which in turn might explain some weirdness in my driver test results.
Here's the essence of the function from net/core/dev.c:
net_rx_action()
{
local_irq_disable();
while (!list_empty(&queue->poll_list)) {
local_irq_enable();
/* do stuff */
local_irq_disable();
}
local_irq_enable();
}Say I receive a packet. net_rx_action() processes it in the while loop and reenables interrupts. But just before net_rx_action() returns, I receive another packet, and __netif_rx_schedule() gets called from the driver. Then the soft irq is raised from within itself. If I'm not interrupted for some other reason, the packet will get processed only at the next jiffie when the soft irq is invoked again.
Am I mistaken?
As an aside, it looks also as though the design might technically allow the network driver to starve the CPU (the very situation NAPI was designed to protect against). If I receive a new packet always right after returning from net_rx_action(), the interrupt will cause the soft irq to be executed immediately. It's true that this scenario would require a very accurately calibrated packet stream, but in my business that just might take place.
Marko
Signature · 1 line
Marko Rauhamaa mailto:[email protected] http://pacujo.net/marko/
Quoted 16 lines · Marko Rauhamaa
It looks to me like
net_rx_action()might suffer from a race, which in turn might explain some weirdness in my driver test results.Here's the essence of the function from
net/core/dev.c:
net_rx_action(){ local_irq_disable(); while (!list_empty(&queue->poll_list)) { local_irq_enable(); /* do stuff */ local_irq_disable(); } local_irq_enable(); }
Quoted 8 lines
Say I receive a packet.
net_rx_action()processes it in the while loop and reenables interrupts. But just beforenet_rx_action()returns, I receive another packet, and__netif_rx_schedule()gets called from the driver. Then the soft irq is raised from within itself. If I'm not interrupted for some other reason, the packet will get processed only at the next jiffie when the soft irq is invoked again.Am I mistaken?
Probably not, as I tested the reception timing accuracy against an independent hardware "packet timestamper", and out of 2 million packets, 3 were delayed by up to 5ms on the linux box (e100 NAPI). There were about 10 packets delayed between 1ms and 5ms.
Pádraig.
Hello!
Quoted 4 lines
interrupted for some other reason, the packet will get processed only at the next jiffie when the soft irq is invoked again.
Am I mistaken?
Yes, you are wrong. It is processed as soon as possible.
Quoted 3 lines
As an aside, it looks also as though the design might technically allow the network driver to starve the CPU (the very situation NAPI was designed to protect against).
Nope. NAPI is not expected to cure starvation caused by softirqs.
Alexey
Quoted 6 lines
Quoted 4 lines
interrupted for some other reason, the packet will get processed only at the next jiffie when the soft irq is invoked again.
Am I mistaken?
Yes, you are wrong. It is processed as soon as possible.
If I receive a packet at the tail end of net_rx_action(), we will schedule the softirq again. But do_softirq() explicitly refuses to run the same softirq right away. The softirq will be invoked at the next interrupt, timer tick, system call (?) or when ksoftirqd is scheduled. It may happen that none of these events occur for milliseconds.
Quoted 5 lines
Quoted 3 lines
As an aside, it looks also as though the design might technically allow the network driver to starve the CPU (the very situation NAPI was designed to protect against).
Nope. NAPI is not expected to cure starvation caused by softirqs.
Well, it almost does. You can blast a NAPI driver with a packet flood, and the system is happy and responsive -- no interrupts are generated, and packets are polled by ksoftirqd. However, you can find a packet rate that will cause the CPU to spend virtually all of its time in NAPI.
Marko
Signature · 1 line
Marko Rauhamaa mailto:[email protected] http://pacujo.net/marko/
Quoted 2 lines · On 08 Oct 2003 14:17:31 -0700 Marko Rauhamaa <[email protected]> wrote:
But
do_softirq()explicitly refuses to run
the same softirq right away.
Check current 2.6.x sources, it does loop a certain number of times even for the same softirq type.
Quoted 4 lines
Well, it almost does. You can blast a NAPI driver with a packet flood, and the system is happy and responsive -- no interrupts are generated, and packets are polled by ksoftirqd. However, you can find a packet rate that will cause the CPU to spend virtually all of its time in NAPI.
This situation can be created with non-NAPI drivers too.
Alexey is trying to explain to you what the true cause of the problem is, and it's not NAPI, it's softirq starvation.