Skip to contentSkip to conversation

genirq/msi: fix crash when handling Multi-MSI

PATCH In review Tong Zhang 5 participants 10 messages Jan 17, 2022 – Jan 21, 2022

Subsystems (3)
lore
Share & tools
lore

Tong Zhang

author
#0 lore raw

PATCH 1/1 genirq/msi: fix crash when handling Multi-MSI

pci_msi_domain_check_cap() could return 1 when domain does not support multi MSI and user request multi MSI. This positive value will be used by __pci_enable_msi_range(). In previous refactor, this positive value is handled as error case which will cause kernel crash.

[    1.197953] BUG: KASAN: use-after-free in __pci_enable_msi_range+0x234/0x320
[    1.198327] Freed by task 1:
[    1.198327]  kfree+0x8f/0x2b0
[    1.198327]  msi_free_msi_descs_range+0xf5/0x130
[    1.198327]  msi_domain_alloc_irqs_descs_locked+0x8d/0xa0
[    1.198327]  __pci_enable_msi_range+0x1a4/0x320
[    1.198327]  pci_alloc_irq_vectors_affinity+0x135/0x1a0
[    1.198327]  pcie_port_device_register+0x4a1/0x5c0
[    1.198327]  pcie_portdrv_probe+0x50/0x100
Fixes 0f62d941acf9 ("genirq/msi: Provide msi_domain_alloc/free_irqs_descs_locked()")
---
 kernel/irq/msi.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/irq/msi.c +1 -1
index 2bdfce5edafd..57b1447a3bf1 100644
@@ -935,7 +935,7 @@ int msi_domain_alloc_irqs_descs_locked(struct irq_domain *domain, struct device
 		return ret;
 
 	ret = ops->domain_alloc_irqs(domain, dev, nvec);
-	if (ret)
+	if (ret < 0)
 		msi_domain_free_irqs_descs_locked(domain, dev);
 	return ret;
 }
Signature · 1 line
2.25.1
#1 lore raw
Quoted 6 lines · Tong Zhang

pci_msi_domain_check_cap() could return 1 when domain does not support multi MSI and user request multi MSI. This positive value will be used by __pci_enable_msi_range(). In previous refactor, this positive value is handled as error case which will cause kernel crash.

Quoted 9 lines
[    1.197953] BUG: KASAN: use-after-free in __pci_enable_msi_range+0x234/0x320
[    1.198327] Freed by task 1:
[    1.198327]  kfree+0x8f/0x2b0
[    1.198327]  msi_free_msi_descs_range+0xf5/0x130
[    1.198327]  msi_domain_alloc_irqs_descs_locked+0x8d/0xa0
[    1.198327]  __pci_enable_msi_range+0x1a4/0x320
[    1.198327]  pci_alloc_irq_vectors_affinity+0x135/0x1a0
[    1.198327]  pcie_port_device_register+0x4a1/0x5c0
[    1.198327]  pcie_portdrv_probe+0x50/0x100

I'm sorry, but you'll have to be a bit clearer in your commit message, because I cannot relate what you describe with the patch.

The real issue seems to be that a domain_alloc_irqs callback can return a positive, non-zero value, and I don't think this is expected.

How about this instead? If I am barking up the wrong tree, please provide a more accurate description of the problem you are seeing.

Thanks,

	M.
kernel/irq/msi.c +4 -2
index 2bdfce5edafd..da8bb6135627 100644
@@ -878,8 +878,10 @@ int __msi_domain_alloc_irqs(struct irq_domain *domain, struct device *dev,
 		virq = __irq_domain_alloc_irqs(domain, -1, desc->nvec_used,
 					       dev_to_node(dev), &arg, false,
 					       desc->affinity);
-		if (virq < 0)
-			return msi_handle_pci_fail(domain, desc, allocated);
+		if (virq < 0) {
+			ret = msi_handle_pci_fail(domain, desc, allocated);
+			return ret < 0 ? ret : 0;
+		}
 
 		for (i = 0; i < desc->nvec_used; i++) {
 			irq_set_msi_desc_off(virq, i, desc);
Signature · 1 line
Without deviation from the norm, progress is not possible.
author
#2 lore raw

Hello,

ops->msi_check could point to pci_msi_domain_check_cap that is the function in question

hence we can have following call stack

pci_msi_domain_check_cap (used by ops->msi_check(domain, info, dev)) msi_domain_prepare_irqs
__msi_domain_alloc_irqs msi_domain_alloc_irqs_descs_locked

What I am suggesting is commit 0f62d941acf9 changed how this return value is being handled and created a UAF

- Tong

Quoted 52 lines · Marc Zyngier
Quoted 6 lines · Tong Zhang

pci_msi_domain_check_cap() could return 1 when domain does not support multi MSI and user request multi MSI. This positive value will be used by __pci_enable_msi_range(). In previous refactor, this positive value is handled as error case which will cause kernel crash.

Quoted 9 lines
[    1.197953] BUG: KASAN: use-after-free in __pci_enable_msi_range+0x234/0x320
[    1.198327] Freed by task 1:
[    1.198327]  kfree+0x8f/0x2b0
[    1.198327]  msi_free_msi_descs_range+0xf5/0x130
[    1.198327]  msi_domain_alloc_irqs_descs_locked+0x8d/0xa0
[    1.198327]  __pci_enable_msi_range+0x1a4/0x320
[    1.198327]  pci_alloc_irq_vectors_affinity+0x135/0x1a0
[    1.198327]  pcie_port_device_register+0x4a1/0x5c0
[    1.198327]  pcie_portdrv_probe+0x50/0x100

I'm sorry, but you'll have to be a bit clearer in your commit message, because I cannot relate what you describe with the patch.

The real issue seems to be that a domain_alloc_irqs callback can return a positive, non-zero value, and I don't think this is expected.

How about this instead? If I am barking up the wrong tree, please provide a more accurate description of the problem you are seeing.

Thanks,

        M.
kernel/irq/msi.c +4 -3
index 2bdfce5edafd..da8bb6135627 100644
@@ -878,8 +878,10 @@ int __msi_domain_alloc_irqs(struct irq_domain *domain, struct device *dev,
                virq = __irq_domain_alloc_irqs(domain, -1, desc->nvec_used,
                                               dev_to_node(dev), &arg, false,
                                               desc->affinity);
-               if (virq < 0)
-                       return msi_handle_pci_fail(domain, desc, allocated);
+               if (virq < 0) {
+                       ret = msi_handle_pci_fail(domain, desc, allocated);
+                       return ret < 0 ? ret : 0;
+               }
 
                for (i = 0; i < desc->nvec_used; i++) {
                        irq_set_msi_desc_off(virq, i, desc);
 
--

Without deviation from the norm, progress is not possible.

#3 lore raw

Please avoid top-posting.

Quoted 8 lines · Tong Zhang

Hello,

ops->msi_check could point to pci_msi_domain_check_cap that is the function in question

hence we can have following call stack

Quoted 7 lines

pci_msi_domain_check_cap (used by ops->msi_check(domain, info, dev)) msi_domain_prepare_irqs
__msi_domain_alloc_irqs msi_domain_alloc_irqs_descs_locked

What I am suggesting is commit 0f62d941acf9 changed how this return value is being handled and created a UAF

OK, this makes more sense.

But msi_domain_prepare_irqs() shouldn't fail in this case, and we should proceed with the allocation of at least one vector, which isn't happening here.

Also, if __msi_domain_alloc_irqs() is supposed to return the number of irqs allocated, it isn't doing it consistently.

Thomas, can you shed some light on what is the intended behaviour here?

Thanks,

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
#4 lore raw
Quoted 16 lines · Marc Zyngier
Quoted 7 lines · Tong Zhang

pci_msi_domain_check_cap (used by ops->msi_check(domain, info, dev)) msi_domain_prepare_irqs
__msi_domain_alloc_irqs msi_domain_alloc_irqs_descs_locked

What I am suggesting is commit 0f62d941acf9 changed how this return value is being handled and created a UAF

OK, this makes more sense.

But msi_domain_prepare_irqs() shouldn't fail in this case, and we should proceed with the allocation of at least one vector, which isn't happening here.

Quoted 5 lines

Also, if __msi_domain_alloc_irqs() is supposed to return the number of irqs allocated, it isn't doing it consistently.

Thomas, can you shed some light on what is the intended behaviour here?

Let me stare at it.

#5 lore raw
Quoted 17 lines · Thomas Gleixner
Quoted 16 lines · Marc Zyngier
Quoted 7 lines · Tong Zhang

pci_msi_domain_check_cap (used by ops->msi_check(domain, info, dev)) msi_domain_prepare_irqs
__msi_domain_alloc_irqs msi_domain_alloc_irqs_descs_locked

What I am suggesting is commit 0f62d941acf9 changed how this return value is being handled and created a UAF

OK, this makes more sense.

But msi_domain_prepare_irqs() shouldn't fail in this case, and we should proceed with the allocation of at least one vector, which isn't happening here.

Quoted 7 lines
Quoted 5 lines

Also, if __msi_domain_alloc_irqs() is supposed to return the number of irqs allocated, it isn't doing it consistently.

Thomas, can you shed some light on what is the intended behaviour here?

Let me stare at it.

It's a subtle issue I overlooked. The UAF is due to

err:

	pci_msi_unmask(entry, msi_multi_mask(entry));

in msi_capability_init() because the core has torn down and freed the entry already.

The proposed patch "fixes" the issue for the PCI/MSI case, but could cause a memory leak for other callers.

Not sure yet what the proper fix is, but that has to wait until tomorrow when brain becomes awake again.

Thanks,

tglx

#6 lore raw

PATCH 1/1 PCI/MSI: Prevent UAF in error path

When the core MSI allocation fails, then the PCI/MSI code uses an already freed MSI descriptor to unmask the MSI mask register in order to bring it back into reset state.

Remove MSI_FLAG_FREE_MSI_DESCS from the PCI/MSI irqdomain flags and let the PCI/MSI code free the MSI descriptors after usage.

Fixes 0f62d941acf9 ("genirq/msi: Provide msi_domain_alloc/free_irqs_descs_locked()")
---
 drivers/pci/msi/irqdomain.c |    4 ++--
 drivers/pci/msi/legacy.c    |    1 -
 2 files changed, 2 insertions(+), 3 deletions(-)
drivers/pci/msi/irqdomain.c +2 -2
@@ -28,6 +28,7 @@ void pci_msi_teardown_msi_irqs(struct pc
 		msi_domain_free_irqs_descs_locked(domain, &dev->dev);
 	else
 		pci_msi_legacy_teardown_msi_irqs(dev);
+	msi_free_msi_descs(&dev->dev);
 }
 
 /**
@@ -171,8 +172,7 @@ struct irq_domain *pci_msi_create_irq_do
 	if (info->flags & MSI_FLAG_USE_DEF_CHIP_OPS)
 		pci_msi_domain_update_chip_ops(info);
 
-	info->flags |= MSI_FLAG_ACTIVATE_EARLY | MSI_FLAG_DEV_SYSFS |
-		       MSI_FLAG_FREE_MSI_DESCS;
+	info->flags |= MSI_FLAG_ACTIVATE_EARLY | MSI_FLAG_DEV_SYSFS;
 	if (IS_ENABLED(CONFIG_GENERIC_IRQ_RESERVATION_MODE))
 		info->flags |= MSI_FLAG_MUST_REACTIVATE;
 
drivers/pci/msi/legacy.c +0 -1
@@ -77,5 +77,4 @@ void pci_msi_legacy_teardown_msi_irqs(st
 {
 	msi_device_destroy_sysfs(&dev->dev);
 	arch_teardown_msi_irqs(dev);
-	msi_free_msi_descs(&dev->dev);
 }
Acked-by on v1 1/1
#7 lore raw

Re: [PATCH] PCI/MSI: Prevent UAF in error path

Quoted 10 lines · Thomas Gleixner

When the core MSI allocation fails, then the PCI/MSI code uses an already freed MSI descriptor to unmask the MSI mask register in order to bring it back into reset state.

Remove MSI_FLAG_FREE_MSI_DESCS from the PCI/MSI irqdomain flags and let the PCI/MSI code free the MSI descriptors after usage.

Fixes 0f62d941acf9 ("genirq/msi: Provide msi_domain_alloc/free_irqs_descs_locked()")

What does "UAF" stand for? Ah, "use after free" I guess?

Let me know if I should take this. Otherwise I assume it'll go whereever 0f62d941acf9 went.

Quoted 33 lines
---
 drivers/pci/msi/irqdomain.c |    4 ++--
 drivers/pci/msi/legacy.c    |    1 -
 2 files changed, 2 insertions(+), 3 deletions(-)
drivers/pci/msi/irqdomain.c +2 -2
@@ -28,6 +28,7 @@ void pci_msi_teardown_msi_irqs(struct pc
 		msi_domain_free_irqs_descs_locked(domain, &dev->dev);
 	else
 		pci_msi_legacy_teardown_msi_irqs(dev);
+	msi_free_msi_descs(&dev->dev);
 }
 
 /**
@@ -171,8 +172,7 @@ struct irq_domain *pci_msi_create_irq_do
 	if (info->flags & MSI_FLAG_USE_DEF_CHIP_OPS)
 		pci_msi_domain_update_chip_ops(info);
 
-	info->flags |= MSI_FLAG_ACTIVATE_EARLY | MSI_FLAG_DEV_SYSFS |
-		       MSI_FLAG_FREE_MSI_DESCS;
+	info->flags |= MSI_FLAG_ACTIVATE_EARLY | MSI_FLAG_DEV_SYSFS;
 	if (IS_ENABLED(CONFIG_GENERIC_IRQ_RESERVATION_MODE))
 		info->flags |= MSI_FLAG_MUST_REACTIVATE;
 
drivers/pci/msi/legacy.c +0 -1
@@ -77,5 +77,4 @@ void pci_msi_legacy_teardown_msi_irqs(st
 {
 	msi_device_destroy_sysfs(&dev->dev);
 	arch_teardown_msi_irqs(dev);
-	msi_free_msi_descs(&dev->dev);
 }
author Tested-by on v1 1/1
#8 lore raw

Re: [PATCH] PCI/MSI: Prevent UAF in error path

In reply to Thomas Gleixner
Quoted 39 lines · Thomas Gleixner

When the core MSI allocation fails, then the PCI/MSI code uses an already freed MSI descriptor to unmask the MSI mask register in order to bring it back into reset state.

Remove MSI_FLAG_FREE_MSI_DESCS from the PCI/MSI irqdomain flags and let the PCI/MSI code free the MSI descriptors after usage.

Fixes 0f62d941acf9 ("genirq/msi: Provide msi_domain_alloc/free_irqs_descs_locked()")
---
 drivers/pci/msi/irqdomain.c |    4 ++--
 drivers/pci/msi/legacy.c    |    1 -
 2 files changed, 2 insertions(+), 3 deletions(-)
drivers/pci/msi/irqdomain.c +2 -2
@@ -28,6 +28,7 @@ void pci_msi_teardown_msi_irqs(struct pc
                msi_domain_free_irqs_descs_locked(domain, &dev->dev);
        else
                pci_msi_legacy_teardown_msi_irqs(dev);
+       msi_free_msi_descs(&dev->dev);
 }
 
 /**
@@ -171,8 +172,7 @@ struct irq_domain *pci_msi_create_irq_do
        if (info->flags & MSI_FLAG_USE_DEF_CHIP_OPS)
                pci_msi_domain_update_chip_ops(info);
 
-       info->flags |= MSI_FLAG_ACTIVATE_EARLY | MSI_FLAG_DEV_SYSFS |
-                      MSI_FLAG_FREE_MSI_DESCS;
+       info->flags |= MSI_FLAG_ACTIVATE_EARLY | MSI_FLAG_DEV_SYSFS;
        if (IS_ENABLED(CONFIG_GENERIC_IRQ_RESERVATION_MODE))
                info->flags |= MSI_FLAG_MUST_REACTIVATE;
 

drivers/pci/msi/legacy.c void pci_msi_legacy_teardown_msi_irqs(st · line 80

Quoted 5 lines
drivers/pci/msi/legacy.c +0 -1
@@ -77,5 +77,4 @@ void pci_msi_legacy_teardown_msi_irqs(st
 {
        msi_device_destroy_sysfs(&dev->dev);
        arch_teardown_msi_irqs(dev);
-       msi_free_msi_descs(&dev->dev);
 }

Tested on my setup, it no longer crashes.
Thanks!
- Tong

#9 lore raw

[tip: irq/urgent] PCI/MSI: Prevent UAF in error path

In reply to Thomas Gleixner
The following commit has been merged into the irq/urgent branch of tip:

The following commit has been merged into the irq/urgent branch of tip:

Commit-ID:     a0af3d1104f752b6d0dba71788e3fddd67c857a7
Gitweb:        https://git.kernel.org/tip/a0af3d1104f752b6d0dba71788e3fddd67c857a7
Author:        Thomas Gleixner <[email protected]>
AuthorDate:    Wed, 19 Jan 2022 18:54:52 +01:00
Committer:     Thomas Gleixner <[email protected]>
CommitterDate: Fri, 21 Jan 2022 02:14:46 +01:00

PCI/MSI: Prevent UAF in error path

When the core MSI allocation fails, then the PCI/MSI code uses an already freed MSI descriptor to unmask the MSI mask register in order to bring it back into reset state.

Remove MSI_FLAG_FREE_MSI_DESCS from the PCI/MSI irqdomain flags and let the PCI/MSI code free the MSI descriptors after usage.

Fixes 0f62d941acf9 ("genirq/msi: Provide msi_domain_alloc/free_irqs_descs_locked()")
---
 drivers/pci/msi/irqdomain.c | 4 ++--
 drivers/pci/msi/legacy.c    | 1 -
 2 files changed, 2 insertions(+), 3 deletions(-)
drivers/pci/msi/irqdomain.c +2 -2
index 0d63541..e9cf318 100644
@@ -28,6 +28,7 @@ void pci_msi_teardown_msi_irqs(struct pci_dev *dev)
 		msi_domain_free_irqs_descs_locked(domain, &dev->dev);
 	else
 		pci_msi_legacy_teardown_msi_irqs(dev);
+	msi_free_msi_descs(&dev->dev);
 }
 
 /**
@@ -171,8 +172,7 @@ struct irq_domain *pci_msi_create_irq_domain(struct fwnode_handle *fwnode,
 	if (info->flags & MSI_FLAG_USE_DEF_CHIP_OPS)
 		pci_msi_domain_update_chip_ops(info);
 
-	info->flags |= MSI_FLAG_ACTIVATE_EARLY | MSI_FLAG_DEV_SYSFS |
-		       MSI_FLAG_FREE_MSI_DESCS;
+	info->flags |= MSI_FLAG_ACTIVATE_EARLY | MSI_FLAG_DEV_SYSFS;
 	if (IS_ENABLED(CONFIG_GENERIC_IRQ_RESERVATION_MODE))
 		info->flags |= MSI_FLAG_MUST_REACTIVATE;
 
drivers/pci/msi/legacy.c +0 -1
index cdbb468..db761ad 100644
@@ -77,5 +77,4 @@ void pci_msi_legacy_teardown_msi_irqs(struct pci_dev *dev)
 {
 	msi_device_destroy_sysfs(&dev->dev);
 	arch_teardown_msi_irqs(dev);
-	msi_free_msi_descs(&dev->dev);
 }