net: apm: xgene: force XGene enet driver to re-balance IRQ usage
PATCH 1/1 net: apm: xgene: force XGene enet driver to re-balance IRQ usage
When using the user-space command 'tuned-adm profile network-latency', the XGene enet driver will cause a hang trying to enable IRQs while the system is trying to tune itself to be more responsive to network traffic; dmesg will even tell us that the enables/disables are not in balance. With this fix, we force the driver to only enable_irq() when there has been a previous disable_irq() -- i.e., force the number of calls to each to be balanced. This allows the kernel to continue operating.
In an ideal world, the XGene enet driver would be restructured to avoid this problem (it seems to be an artifact of when additional packets arrive and differences of opinion in how the NIC responds under those circumstances, some of which is controlled by firmware). In the XGene2 driver, this is not an issue.
However, the XGene (aka Mustang) where this NIC is used is most likely at the end of its useful life (APM which originally created the XGene has completely morphed into a new company). It is unlikely the driver
restructuring that is needed will ever be done. There are, however,
a bunch of these machines out in the real world, and there are many of us still using them daily (me, for example). So, while this patch is not the ideal way to repair the NIC driver, it does work and allows us to continue using these boxes for a while longer.
--- drivers/net/ethernet/apm/xgene/xgene_enet_main.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
drivers/net/ethernet/apm/xgene/xgene_enet_main.c +4 -1
@@ -866,8 +866,11 @@ static int xgene_enet_napi(struct napi_struct *napi, const int budget) processed =xgene_enet_process_ring (ring, budget);if (processed != budget) { +struct irq_desc *desc =irq_to_desc (ring->irq); +napi_complete_done (napi, processed); -enable_irq (ring->irq); +if (desc && desc->depth >0 ) +enable_irq (ring->irq); }return processed;
Signature · 1 line
2.17.1
David Miller
maintainerFrom: Al Stone <[email protected]>
Date: Mon, 17 Sep 2018 17:35:33 -0600
drivers/net/ethernet/apm/xgene/xgene_enet_main.c static int xgene_enet_napi(struct napi_struct *napi, const int budget) · line 873
Quoted 10 lines
drivers/net/ethernet/apm/xgene/xgene_enet_main.c +4
-1@@ -866,8 +866,11 @@ static int xgene_enet_napi(struct napi_struct *napi, const int budget) processed =xgene_enet_process_ring (ring, budget);if (processed != budget) { +struct irq_desc *desc =irq_to_desc (ring->irq); +napi_complete_done (napi, processed); -enable_irq (ring->irq); +if (desc && desc->depth >0 ) +enable_irq (ring->irq);
I really can't accept a patch that grovels into IRQ layer internals to work around a driver's IRQ enable/disable usage problem.
Sorry.
Quoted 17 lines · David Miller
From: Al Stone <[email protected]>
Date: Mon, 17 Sep 2018 17:35:33 -0600drivers/net/ethernet/apm/xgene/xgene_enet_main.c static int xgene_enet_napi(struct napi_struct *napi, const int budget) · line 873
Quoted 10 lines
drivers/net/ethernet/apm/xgene/xgene_enet_main.c +4
-1@@ -866,8 +866,11 @@ static int xgene_enet_napi(struct napi_struct *napi, const int budget) processed =xgene_enet_process_ring (ring, budget);if (processed != budget) { +struct irq_desc *desc =irq_to_desc (ring->irq); +napi_complete_done (napi, processed); -enable_irq (ring->irq); +if (desc && desc->depth >0 ) +enable_irq (ring->irq);I really can't accept a patch that grovels into IRQ layer internals to work around a driver's IRQ enable/disable usage problem.
Quoted 1 line
Sorry.
No worries. I hesitated even sending it, actually. The rewrite of the driver that is needed is just something no one seems to have any interest in, or any time for, but it really needs it to fix this properly (a colleague has found at least one other issue in the structure of this driver, for example). The cost of doing that rewrite/restructure, though, far outweighs the benefit for this device -- especially when the workaround is "don't run tuned".
I'll put a proper fix back on the list of Interesting Things to Work On When Time Allows, for now.
Thanks for the quick -- and kind :) -- response. I really appreciate it.
Signature · 8 lines
ciao, al ----------------------------------- Al Stone Software Engineer Red Hat, Inc. [email protected] -----------------------------------
Quoted 10 lines · David Miller
From: Al Stone <[email protected]>
Date: Mon, 17 Sep 2018 17:35:33 -0600drivers/net/ethernet/apm/xgene/xgene_enet_main.c static int xgene_enet_napi(struct napi_struct *napi, const int budget) · line 871
Quoted 7 lines
drivers/net/ethernet/apm/xgene/xgene_enet_main.c +2
-0@@ -866,8 +866,11 @@ static int xgene_enet_napi(struct napi_struct *napi, const int budget) processed =xgene_enet_process_ring (ring, budget);if (processed != budget) { +struct irq_desc *desc =irq_to_desc (ring->irq); +napi_complete_done (napi, processed);
The problem could be that the driver isn't checking the napi_complete_done() return code. It was changed to return a bool and the check should be more like:
if ((processed != budget) && napi_complete_done(napi, processed)) {If it returns false, then the driver will get called for polling again after having issued enable_irq() and it well then issue the enable_irq() a second (or more) time without having the matching diable_irq().
Thanks,
Tom
Quoted 9 lines
drivers/net/ethernet/apm/xgene/xgene_enet_main.c static int xgene_enet_napi(struct napi_struct *napi, const int budget) · line 873
Quoted 3 lines
drivers/net/ethernet/apm/xgene/xgene_enet_main.c +2
-1-enable_irq (ring->irq); +if (desc && desc->depth >0 ) +enable_irq (ring->irq);I really can't accept a patch that grovels into IRQ layer internals to work around a driver's IRQ enable/disable usage problem.
Sorry.
Quoted 18 lines · Thomas
Quoted 10 lines · David Miller
From: Al Stone <[email protected]>
Date: Mon, 17 Sep 2018 17:35:33 -0600drivers/net/ethernet/apm/xgene/xgene_enet_main.c static int xgene_enet_napi(struct napi_struct *napi, const int budget) · line 871
Quoted 7 lines
drivers/net/ethernet/apm/xgene/xgene_enet_main.c +2
-0@@ -866,8 +866,11 @@ static int xgene_enet_napi(struct napi_struct *napi, const int budget) processed =xgene_enet_process_ring (ring, budget);if (processed != budget) { +struct irq_desc *desc =irq_to_desc (ring->irq); +napi_complete_done (napi, processed);The problem could be that the driver isn't checking the
napi_complete_done()return code. It was changed to return a bool and the check should be more like:
Quoted 8 lines
if ((processed != budget) && napi_complete_done(napi, processed)) {If it returns false, then the driver will get called for polling again after having issued
enable_irq()and it well then issue theenable_irq()a second (or more) time without having the matchingdiable_irq().Thanks,
Tom
Aha, that might be. My apologies -- I play in ACPI but seldom in the network drivers, so was not fully aware of that change. I can give that a try.
Thanks for the pointer.
Signature · 8 lines
ciao, al ----------------------------------- Al Stone Software Engineer Red Hat, Inc. [email protected] -----------------------------------
Quoted 25 lines · Al Stone
Quoted 24 lines · Thomas
Quoted 10 lines · David Miller
From: Al Stone <[email protected]>
Date: Mon, 17 Sep 2018 17:35:33 -0600drivers/net/ethernet/apm/xgene/xgene_enet_main.c static int xgene_enet_napi(struct napi_struct *napi, const int budget) · line 871
Quoted 7 lines
drivers/net/ethernet/apm/xgene/xgene_enet_main.c +2
-0@@ -866,8 +866,11 @@ static int xgene_enet_napi(struct napi_struct *napi, const int budget) processed =xgene_enet_process_ring (ring, budget);if (processed != budget) { +struct irq_desc *desc =irq_to_desc (ring->irq); +napi_complete_done (napi, processed);The problem could be that the driver isn't checking the
napi_complete_done()return code. It was changed to return a bool and the check should be more like:if ((processed != budget) && napi_complete_done(napi, processed)) {If it returns false, then the driver will get called for polling again after having issued
enable_irq()and it well then issue theenable_irq()a second (or more) time without having the matchingdiable_irq().
Quoted 7 lines
Quoted 2 lines
Thanks,
TomAha, that might be. My apologies -- I play in ACPI but seldom in the network drivers, so was not fully aware of that change. I can give that a try.
Thanks for the pointer.
FWIW this is being discussed in this thread as well:
https://www.spinics.net/lists/netdev/msg523760.html
there should be an update to drivers that have a ndo_poll_controller() and don't check napi_complete_done(), though I am not clear who is doing that yet.
Signature · 1 line
Florian
Quoted 34 lines · Florian Fainelli
Quoted 25 lines · Al Stone
Quoted 24 lines · Thomas
Quoted 10 lines · David Miller
From: Al Stone <[email protected]>
Date: Mon, 17 Sep 2018 17:35:33 -0600drivers/net/ethernet/apm/xgene/xgene_enet_main.c static int xgene_enet_napi(struct napi_struct *napi, const int budget) · line 871
Quoted 7 lines
drivers/net/ethernet/apm/xgene/xgene_enet_main.c +2
-0@@ -866,8 +866,11 @@ static int xgene_enet_napi(struct napi_struct *napi, const int budget) processed =xgene_enet_process_ring (ring, budget);if (processed != budget) { +struct irq_desc *desc =irq_to_desc (ring->irq); +napi_complete_done (napi, processed);The problem could be that the driver isn't checking the
napi_complete_done()return code. It was changed to return a bool and the check should be more like:if ((processed != budget) && napi_complete_done(napi, processed)) {If it returns false, then the driver will get called for polling again after having issued
enable_irq()and it well then issue theenable_irq()a second (or more) time without having the matchingdiable_irq().Quoted 7 lines
Quoted 2 lines
Thanks,
TomAha, that might be. My apologies -- I play in ACPI but seldom in the network drivers, so was not fully aware of that change. I can give that a try.
Thanks for the pointer.
Quoted 7 lines
FWIW this is being discussed in this thread as well:
https://www.spinics.net/lists/netdev/msg523760.html
there should be an update to drivers that have a
ndo_poll_controller()and don't checknapi_complete_done(), though I am not clear who is doing that yet.
That might be tricky.
I remember one of the napi_complete_done() change had to be reverted, for some obscure reason.
So clearly, doing a mass update (without being able to test the driver on real hardware) might be risky.
Eric Dumazet
Tested-by Ville SyrjäläQuoted 4 lines · Eric Dumazet
I remember one of the
napi_complete_done()change had to be reverted, for some obscure reason.
That was not exactly a revert, :
commit 129c6cda2de2a8ac44fab096152469999b727faf
Author: Eric Dumazet <[email protected]> Date: Mon Sep 18 13:03:43 2017 -0700
8139too: revisit napi_complete_done() usage
It seems we have to be more careful in napi_complete_done() use. This patch is not a revert, as it seems we can avoid bug that Ville reported by moving the napi_complete_done() test in the spinlock section.
Many thanks to Ville for detective work and all tests.
Quoted 11 lines · Eric Dumazet
Quoted 4 lines · Eric Dumazet
I remember one of the
napi_complete_done()change had to be reverted, for some obscure reason.That was not exactly a revert, :
This is what I have so far for the drivers that both use napi_complete_done() without checking the return value and implement a ndo_poll_controller() callback:
https://github.com/ffainelli/linux/commits/napi-check
Quoted 25 lines
commit
129c6cda2de2a8ac44fab096152469999b727fafAuthor: Eric Dumazet <[email protected]> Date: Mon Sep 18 13:03:43 2017 -07008139too: revisit
napi_complete_done()usageIt seems we have to be more careful in
napi_complete_done()use. This patch is not a revert, as it seems we can avoid bug that Ville reported by moving thenapi_complete_done()test in the spinlock section.Many thanks to Ville for detective work and all tests.
Fixes 617f01211baf ("8139too: use napi_complete_done()")Reported-by Ville Syrjälä <[email protected]>Tested-by Ville Syrjälä <[email protected]>Signed-off-by David S. Miller <[email protected]>
Signature · 1 line
Florian
Quoted 11 lines · Florian Fainelli
Quoted 10 lines · Eric Dumazet
Quoted 4 lines · Eric Dumazet
I remember one of the
napi_complete_done()change had to be reverted, for some obscure reason.
Quoted 7 lines
Quoted 1 line
That was not exactly a revert, :
This is what I have so far for the drivers that both use
napi_complete_done()without checking the return value and implement ando_poll_controller()callback:
In fact, there is still to explain what the bug is.
napi_complete_done() return value can be ignored, unless drivers have to disable IRQ in their interrupt handler, using the following construct :
if (napi_schedule_prep(napi)) {
.... disable interrupt ....
__napi_schedule_irqoff(napi);
}It _can_ be used by other drivers to not rearm interrupts needlessly.
The bug discussed in this thread (re-balance IRQ usage) is of the same kind than
the one fixed in commit d7aba644ffdebf756e51e26a2229055211838e89 ("amd-xgbe: Enable IRQs only if napi_complete_done() is true")
Quoted 25 lines · Al Stone
Quoted 24 lines · Thomas
Quoted 10 lines · David Miller
From: Al Stone <[email protected]>
Date: Mon, 17 Sep 2018 17:35:33 -0600drivers/net/ethernet/apm/xgene/xgene_enet_main.c static int xgene_enet_napi(struct napi_struct *napi, const int budget) · line 871
Quoted 7 lines
drivers/net/ethernet/apm/xgene/xgene_enet_main.c +2
-0@@ -866,8 +866,11 @@ static int xgene_enet_napi(struct napi_struct *napi, const int budget) processed =xgene_enet_process_ring (ring, budget);if (processed != budget) { +struct irq_desc *desc =irq_to_desc (ring->irq); +napi_complete_done (napi, processed);The problem could be that the driver isn't checking the
napi_complete_done()return code. It was changed to return a bool and the check should be more like:if ((processed != budget) && napi_complete_done(napi, processed)) {If it returns false, then the driver will get called for polling again after having issued
enable_irq()and it well then issue theenable_irq()a second (or more) time without having the matchingdiable_irq().
Quoted 8 lines
Quoted 2 lines
Thanks,
TomAha, that might be. My apologies -- I play in ACPI but seldom in the network drivers, so was not fully aware of that change. I can give that a try.
Thanks for the pointer.
Yes, please look at commit d7aba644ffdebf756e51e26a2229055211838e89 ("amd-xgbe: Enable IRQs only if napi_complete_done() is true")