r8169: set IRQF_NO_THREAD if MSI(X) is enabled
Heiner Kallweit
maintainerPATCH 1/1 r8169: set IRQF_NO_THREAD if MSI(X) is enabled
We had to remove flag IRQF_NO_THREAD because it conflicts with shared interrupts in case legacy interrupts are used. Following up on the linked discussion set IRQF_NO_THREAD if MSI or MSI-X is used, because both guarantee that interrupt won't be shared.
--- drivers/net/ethernet/realtek/r8169_main.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
drivers/net/ethernet/realtek/r8169_main.c +3 -1
@@ -4690,6 +4690,7 @@ static int rtl_open(struct net_device *dev) {struct rtl8169_private *tp =netdev_priv (dev);struct pci_dev *pdev = tp->pci_dev; +unsigned long irqflags;int retval = -ENOMEM ;pm_runtime_get_sync (&pdev->dev);
@@ -4714,8 +4715,9 @@ static int rtl_open(struct net_device *dev)rtl_request_firmware (tp); + irqflags =pci_dev_msi_enabled (pdev) ?IRQF_NO_THREAD :IRQF_SHARED ; retval =request_irq (pci_irq_vector (pdev,0 ), rtl8169_interrupt, -IRQF_SHARED , dev->name, tp); + irqflags, dev->name, tp);if (retval <0 ) goto err_release_fw_2;
Signature · 1 line
2.29.2
Quoted 7 lines · Heiner Kallweit
We had to remove flag
IRQF_NO_THREADbecause it conflicts with shared interrupts in case legacy interrupts are used. Following up on the linked discussion setIRQF_NO_THREADif MSI or MSI-X is used, because both guarantee that interrupt won't be shared.
I am not sure if this utilization of the Link: tag is valid. I think it has a well-defined meaning and maintainers use it to provide a link to the email where the patch was picked from:
https://lkml.org/lkml/2011/4/6/421
Quoted 30 lines
--- drivers/net/ethernet/realtek/r8169_main.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)drivers/net/ethernet/realtek/r8169_main.c +3
-1@@ -4690,6 +4690,7 @@ static int rtl_open(struct net_device *dev) {struct rtl8169_private *tp =netdev_priv (dev);struct pci_dev *pdev = tp->pci_dev; +unsigned long irqflags;int retval = -ENOMEM ;pm_runtime_get_sync (&pdev->dev);@@ -4714,8 +4715,9 @@ static int rtl_open(struct net_device *dev)rtl_request_firmware (tp); + irqflags =pci_dev_msi_enabled (pdev) ?IRQF_NO_THREAD :IRQF_SHARED ; retval =request_irq (pci_irq_vector (pdev,0 ), rtl8169_interrupt, -IRQF_SHARED , dev->name, tp); + irqflags, dev->name, tp);if (retval <0 ) goto err_release_fw_2;Signature · 1 line
2.29.2
So all things considered, what do you want to achieve with this change? Is there other benefit with disabling force threading of the rtl8169_interrupt, or are you still looking to add back the napi_schedule_irqoff call?
Heiner Kallweit
maintainerQuoted 6 lines · Vladimir Oltean
Quoted 5 lines · Heiner Kallweit
We had to remove flag
IRQF_NO_THREADbecause it conflicts with shared interrupts in case legacy interrupts are used. Following up on the linked discussion setIRQF_NO_THREADif MSI or MSI-X is used, because both guarantee that interrupt won't be shared.
Quoted 8 lines
Quoted 2 lines
I am not sure if this utilization of the Link: tag is valid. I think it has a well-defined meaning and maintainers use it to provide a link to the email where the patch was picked from:
https://lkml.org/lkml/2011/4/6/421
Thanks for the link. There have been discussions whether to have the change log of patches as part of the commit message or not, and as part of this discussion how the Link tag can help. IIRC outcome was:
- Link tag can be used to point to a discussion elaborating on the evolution of a patch series
- Link tag can be used to point to a discussion explaining the motivation for a change
Having said that it's my understanding that this tag isn't to be used by the maintainers only. However maintainers may see this differently.
Quoted 27 lines
Quoted 27 lines
--- drivers/net/ethernet/realtek/r8169_main.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)drivers/net/ethernet/realtek/r8169_main.c +3
-1@@ -4690,6 +4690,7 @@ static int rtl_open(struct net_device *dev) {struct rtl8169_private *tp =netdev_priv (dev);struct pci_dev *pdev = tp->pci_dev; +unsigned long irqflags;int retval = -ENOMEM ;pm_runtime_get_sync (&pdev->dev);@@ -4714,8 +4715,9 @@ static int rtl_open(struct net_device *dev)rtl_request_firmware (tp); + irqflags =pci_dev_msi_enabled (pdev) ?IRQF_NO_THREAD :IRQF_SHARED ; retval =request_irq (pci_irq_vector (pdev,0 ), rtl8169_interrupt, -IRQF_SHARED , dev->name, tp); + irqflags, dev->name, tp);if (retval <0 ) goto err_release_fw_2;
Quoted 9 lines
Quoted 3 lines
Signature · 1 line
2.29.2So all things considered, what do you want to achieve with this change? Is there other benefit with disabling force threading of the
rtl8169_interrupt, or are you still looking to add back thenapi_schedule_irqoffcall?
As mentioned by Eric it doesn't make sense to make the minimal hard irq handlers used with NAPI a thread. This more contributes to the problem than to the solution. The change here reflects this. The actual discussion would be how to make the NAPI processing a thread (instead softirq).
For using napi_schedule_irqoff we most likely need something like if (pci_dev_msi_enabled(pdev))
napi_schedule_irqoff(napi); else napi_schedule(napi);
and I doubt that's worth it.
Quoted 3 lines · Heiner Kallweit
As mentioned by Eric it doesn't make sense to make the minimal hard irq handlers used with NAPI a thread. This more contributes to the problem than to the solution. The change here reflects this.
When you say that "it doesn't make sense", is there something that is actually measurably worse when the hardirq handler gets force-threaded? Rephrased, is it something that doesn't make sense in principle, or in practice?
My understanding is that this is not where the bulk of the NAPI poll processing is done anyway, so it should not have a severe negative impact on performance in any case.
On the other hand, moving as much code as possible outside interrupt context (be it hardirq or softirq) is beneficial to some use cases, because the scheduler is not in control of that code's runtime unless it is in a thread.
Quoted 2 lines
The actual discussion would be how to make the NAPI processing a thread (instead softirq).
I don't get it, so you prefer the hardirq handler to consume CPU time which is not accounted for by the scheduler, but for the NAPI poll, you do want the scheduler to account for it? So why one but not the other?
Quoted 6 lines
For using
napi_schedule_irqoffwe most likely need something like if (pci_dev_msi_enabled(pdev))napi_schedule_irqoff(napi); else napi_schedule(napi);and I doubt that's worth it.
Yes, probably not, hence my question.
Heiner Kallweit
maintainerQuoted 10 lines · Vladimir Oltean
Quoted 3 lines · Heiner Kallweit
As mentioned by Eric it doesn't make sense to make the minimal hard irq handlers used with NAPI a thread. This more contributes to the problem than to the solution. The change here reflects this.
When you say that "it doesn't make sense", is there something that is actually measurably worse when the hardirq handler gets force-threaded? Rephrased, is it something that doesn't make sense in principle, or in practice?
Quoted 9 lines
My understanding is that this is not where the bulk of the NAPI poll processing is done anyway, so it should not have a severe negative impact on performance in any case.
On the other hand, moving as much code as possible outside interrupt context (be it hardirq or softirq) is beneficial to some use cases, because the scheduler is not in control of that code's runtime unless it is in a thread.
According to my understanding the point is that executing the simple hard irq handler for NAPI drivers doesn't cost significantly more than executing the default hard irq handler (irq_default_primary_handler). Therefore threadifying it means more or less just overhead.
forced threading:
1. irq_default_primary_handler, wakes irq thread
2. threadified driver hard irq handler (basically just calling napi_schedule)
3. NAPI processing
IRQF_NO_THREAD:
1. driver hard irq handler, scheduling NAPI
2. NAPI processing
Quoted 7 lines
Quoted 2 lines
The actual discussion would be how to make the NAPI processing a thread (instead softirq).
I don't get it, so you prefer the hardirq handler to consume CPU time which is not accounted for by the scheduler, but for the NAPI poll, you do want the scheduler to account for it? So why one but not the other?
The CPU time for scheduling NAPI is neglectable, but doing all the rx and tx work in NAPI processing is significant effort.
Quoted 9 lines
Quoted 6 lines
For using
napi_schedule_irqoffwe most likely need something like if (pci_dev_msi_enabled(pdev))napi_schedule_irqoff(napi); else napi_schedule(napi);and I doubt that's worth it.
Yes, probably not, hence my question.
Quoted 4 lines · Heiner Kallweit
According to my understanding the point is that executing the simple hard irq handler for NAPI drivers doesn't cost significantly more than executing the default hard irq handler (
irq_default_primary_handler). Therefore threadifying it means more or less just overhead.
If that is really true, then sure. You could probably run a cyclictest under a ping flood just to make sure though.
Quoted 7 lines · On Sun, 1 Nov 2020 23:30:44 +0100 Heiner Kallweit wrote:
We had to remove flag
IRQF_NO_THREADbecause it conflicts with shared interrupts in case legacy interrupts are used. Following up on the linked discussion setIRQF_NO_THREADif MSI or MSI-X is used, because both guarantee that interrupt won't be shared.
Applied, based on my understanding on RT. But I do agree with Vladimir that it'd be good to see actual performance numbers for these sort of optimizations.