Skip to contentSkip to conversation

dpaa2-eth: use napi_schedule to be compatible with PREEMPT_RT

PATCH Applied Vladimir Oltean 3 participants 8 messages Aug 3, 2020 – Aug 24, 2020

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

PATCH 1/2 dpaa2-eth: use napi_schedule to be compatible with PREEMPT_RT

From: Jiafei Pan <[email protected]>

The driver calls napi_schedule_irqoff() from a context where, in RT, hardirqs are not disabled, since the IRQ handler is force-threaded.

In the call path of this function, __raise_softirq_irqoff() is modifying its per-CPU mask of pending softirqs that must be processed, using or_softirq_pending(). The or_softirq_pending() function is not atomic, but since interrupts are supposed to be disabled, nobody should be preempting it, and the operation should be safe.

Nonetheless, when running with hardirqs on, as in the PREEMPT_RT case, it isn't safe, and the pending softirqs mask can get corrupted, resulting in softirqs being lost and never processed.

To have common code that works with PREEMPT_RT and with mainline Linux, we can use plain napi_schedule() instead. The difference is that napi_schedule() (via __napi_schedule) also calls local_irq_save, which disables hardirqs if they aren't already. But, since they already are disabled in non-RT, this means that in practice we don't see any measurable difference in throughput or latency with this patch.

---
 drivers/net/ethernet/freescale/dpaa2/dpaa2-eth.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/net/ethernet/freescale/dpaa2/dpaa2-eth.c +1 -1
index 9b4028c0e34c..50f52fe2012f 100644
@@ -2330,7 +2330,7 @@ static void cdan_cb(struct dpaa2_io_notification_ctx *ctx)
 	/* Update NAPI statistics */
 	ch->stats.cdan++;
 
-	napi_schedule_irqoff(&ch->napi);
+	napi_schedule(&ch->napi);
 }
 
 /* Allocate and configure a DPCON object */
Signature · 1 line
2.25.1
#1 lore raw

PATCH 2/2 enetc: use napi_schedule to be compatible with PREEMPT_RT

From: Jiafei Pan <[email protected]>

The driver calls napi_schedule_irqoff() from a context where, in RT, hardirqs are not disabled, since the IRQ handler is force-threaded.

In the call path of this function, __raise_softirq_irqoff() is modifying its per-CPU mask of pending softirqs that must be processed, using or_softirq_pending(). The or_softirq_pending() function is not atomic, but since interrupts are supposed to be disabled, nobody should be preempting it, and the operation should be safe.

Nonetheless, when running with hardirqs on, as in the PREEMPT_RT case, it isn't safe, and the pending softirqs mask can get corrupted, resulting in softirqs being lost and never processed.

To have common code that works with PREEMPT_RT and with mainline Linux, we can use plain napi_schedule() instead. The difference is that napi_schedule() (via __napi_schedule) also calls local_irq_save, which disables hardirqs if they aren't already. But, since they already are disabled in non-RT, this means that in practice we don't see any measurable difference in throughput or latency with this patch.

---
 drivers/net/ethernet/freescale/enetc/enetc.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/net/ethernet/freescale/enetc/enetc.c +1 -1
index f50353cbb4db..f78ca7b343d2 100644
@@ -270,7 +270,7 @@ static irqreturn_t enetc_msix(int irq, void *data)
 	for_each_set_bit(i, &v->tx_rings_map, ENETC_MAX_NUM_TXQS)
 		enetc_wr_reg(v->tbier_base + ENETC_BDR_OFF(i), 0);
 
-	napi_schedule_irqoff(&v->napi);
+	napi_schedule(&v->napi);
 
 	return IRQ_HANDLED;
 }
Signature · 1 line
2.25.1
maintainer Applied
#2 lore raw

Re: [PATCH net-next 2/2] enetc: use napi_schedule to be compatible with PREEMPT_RT

From: Vladimir Oltean <[email protected]>
Date: Mon, 3 Aug 2020 23:10:09 +0300

Quoted 22 lines

From: Jiafei Pan <[email protected]>

The driver calls napi_schedule_irqoff() from a context where, in RT, hardirqs are not disabled, since the IRQ handler is force-threaded.

In the call path of this function, __raise_softirq_irqoff() is modifying its per-CPU mask of pending softirqs that must be processed, using or_softirq_pending(). The or_softirq_pending() function is not atomic, but since interrupts are supposed to be disabled, nobody should be preempting it, and the operation should be safe.

Nonetheless, when running with hardirqs on, as in the PREEMPT_RT case, it isn't safe, and the pending softirqs mask can get corrupted, resulting in softirqs being lost and never processed.

To have common code that works with PREEMPT_RT and with mainline Linux, we can use plain napi_schedule() instead. The difference is that napi_schedule() (via __napi_schedule) also calls local_irq_save, which disables hardirqs if they aren't already. But, since they already are disabled in non-RT, this means that in practice we don't see any measurable difference in throughput or latency with this patch.

Quoted 2 lines

Applied.

#3 lore raw

Re: [PATCH net-next 2/2] enetc: use napi_schedule to be compatible with PREEMPT_RT

Quoted 3 lines · David Miller

From: Vladimir Oltean <[email protected]>

Quoted 2 lines

The driver calls napi_schedule_irqoff() from a context where, in RT, hardirqs are not disabled, since the IRQ handler is force-threaded.

…

Quoted 5 lines
Quoted 3 lines

Applied.

Could these two patches be forwarded -stable, please? The changelog describes this as a problem on PREEMPT_RT but this also happens on !RT with the `threadirqs' commandline switch.

Sebastian

#4 lore raw

Re: [PATCH net-next 2/2] enetc: use napi_schedule to be compatible with PREEMPT_RT

Quoted 9 lines · Sebastian Andrzej Siewior
Quoted 3 lines · David Miller

From: Vladimir Oltean <[email protected]>

Quoted 2 lines

The driver calls napi_schedule_irqoff() from a context where, in RT, hardirqs are not disabled, since the IRQ handler is force-threaded.

…

Quoted 4 lines
Quoted 3 lines
Quoted 7 lines
Quoted 1 line

Applied.

Could these two patches be forwarded -stable, please? The changelog describes this as a problem on PREEMPT_RT but this also happens on !RT with the `threadirqs' commandline switch.

Sebastian

I expect the driver maintainers to have something to say about this. I didn't test on stable kernels, and at least for dpaa2-eth, the change would need to go pretty deep down the stable line.

Also, not really sure who is using the threadirqs option except for testing purposes.

Thanks,
-Vladimir

#5 lore raw

Re: [PATCH net-next 2/2] enetc: use napi_schedule to be compatible with PREEMPT_RT

Quoted 3 lines · Vladimir Oltean

I expect the driver maintainers to have something to say about this. I didn't test on stable kernels, and at least for dpaa2-eth, the change would need to go pretty deep down the stable line.

Yes, each affected and maintained stable kernel. This would also ensure that it is part stable-RT trees.

Quoted 2 lines

Also, not really sure who is using the threadirqs option except for testing purposes.

Oh.

Quoted 2 lines

Thanks,
-Vladimir

Sebastian

#6 lore raw

Re: [PATCH net-next 2/2] enetc: use napi_schedule to be compatible with PREEMPT_RT

In reply to Vladimir Oltean

Hi Sasha,

Quoted 22 lines · Vladimir Oltean
Quoted 9 lines · Sebastian Andrzej Siewior
Quoted 3 lines · David Miller

From: Vladimir Oltean <[email protected]>

Quoted 2 lines

The driver calls napi_schedule_irqoff() from a context where, in RT, hardirqs are not disabled, since the IRQ handler is force-threaded.

…

Quoted 4 lines
Quoted 3 lines
Quoted 7 lines
Quoted 1 line

Applied.

Could these two patches be forwarded -stable, please? The changelog describes this as a problem on PREEMPT_RT but this also happens on !RT with the `threadirqs' commandline switch.

Sebastian

I expect the driver maintainers to have something to say about this. I didn't test on stable kernels, and at least for dpaa2-eth, the change would need to go pretty deep down the stable line.

Quoted 5 lines

Also, not really sure who is using the threadirqs option except for testing purposes.

Thanks,
-Vladimir

Do you think that this type of request is something that AUTOSEL can handle?

Thanks,
-Vladimir

maintainer Applied
#7 lore raw
In reply to Vladimir Oltean

From: Vladimir Oltean <[email protected]>
Date: Mon, 3 Aug 2020 23:10:08 +0300

Quoted 22 lines

From: Jiafei Pan <[email protected]>

The driver calls napi_schedule_irqoff() from a context where, in RT, hardirqs are not disabled, since the IRQ handler is force-threaded.

In the call path of this function, __raise_softirq_irqoff() is modifying its per-CPU mask of pending softirqs that must be processed, using or_softirq_pending(). The or_softirq_pending() function is not atomic, but since interrupts are supposed to be disabled, nobody should be preempting it, and the operation should be safe.

Nonetheless, when running with hardirqs on, as in the PREEMPT_RT case, it isn't safe, and the pending softirqs mask can get corrupted, resulting in softirqs being lost and never processed.

To have common code that works with PREEMPT_RT and with mainline Linux, we can use plain napi_schedule() instead. The difference is that napi_schedule() (via __napi_schedule) also calls local_irq_save, which disables hardirqs if they aren't already. But, since they already are disabled in non-RT, this means that in practice we don't see any measurable difference in throughput or latency with this patch.

Quoted 2 lines

Applied.