Skip to contentSkip to conversation

genirq, PCI/MSI: Support for per device MSI and PCI/IMS - Part 1 cleanups

PATCH Applied Thomas Gleixner 11 participants 163 messages Nov 11, 2022 – Mar 2, 2023 Page 2 of 2

Subsystems (25)
lore
Share & tools
lore
#144 lore raw

[tip: irq/core] PCI/MSI: Reject MSI-X early

In reply to Thomas Gleixner
The following commit has been merged into the irq/core branch of tip:

The following commit has been merged into the irq/core branch of tip:

Commit-ID:     99f3d279765725920aa5924fa445537a20129a6f
Gitweb:        https://git.kernel.org/tip/99f3d279765725920aa5924fa445537a20129a6f
Author:        Thomas Gleixner <[email protected]>
AuthorDate:    Fri, 11 Nov 2022 14:55:11 +01:00
Committer:     Thomas Gleixner <[email protected]>
CommitterDate: Thu, 17 Nov 2022 15:15:22 +01:00

PCI/MSI: Reject MSI-X early

Similar to PCI multi-MSI reject MSI-X enablement when a irq domain is attached to the device which does not support MSI-X.

---
 drivers/pci/msi/msi.c | 4 ++++
 1 file changed, 4 insertions(+)
drivers/pci/msi/msi.c +4 -0
index bc84647..0740acd 100644
@@ -762,6 +762,10 @@ int __pci_enable_msix_range(struct pci_dev *dev, struct msix_entry *entries, int
 	if (WARN_ON_ONCE(dev->msix_enabled))
 		return -EINVAL;
 
+	/* Check MSI-X early on irq domain enabled architectures */
+	if (!pci_msi_domain_supports(dev, MSI_FLAG_PCI_MSIX, ALLOW_LEGACY))
+		return -ENOTSUPP;
+
 	if (!pci_msi_supported(dev, nvec) || dev->current_state != PCI_D0)
 		return -EINVAL;
 
maintainer
#145 lore raw

PATCH 1/1 x86/xen: Set MSI_FLAG_PCI_MSIX support in Xen MSI domain

In reply to Thomas Gleixner

The Xen MSI → PIRQ magic does support MSI-X, so advertise it.

(In fact it's better off with MSI-X than MSI, because it's actually broken by design for 32-bit MSI, since it puts the high bits of the PIRQ# into the high 32 bits of the MSI message address, instead of the Extended Destination ID field which is in bits 4-11.

Strictly speaking, this really fixes a much older commit 2e4386eba0c0 ("x86/xen: Wrap XEN MSI management into irqdomain") which failed to set the flag. But that never really mattered until __pci_enable_msix_range() started to check and bail out early. So in 6.2-rc we see failures e.g. to bring up networking on an Amazon EC2 m4.16xlarge instance:

[   41.498694] ena 0000:00:03.0 (unnamed net_device) (uninitialized): Failed to enable MSI-X. irq_cnt -524
[   41.498705] ena 0000:00:03.0: Can not reserve msix vectors
[   41.498712] ena 0000:00:03.0: Failed to enable and set the admin interrupts

Side note: This is the first bug found, and first patch tested, by running Xen guests under QEMU/KVM instead of running under actual Xen.

Fixes 99f3d2797657 ("PCI/MSI: Reject MSI-X early")
---
 arch/x86/pci/xen.c | 1 +
 1 file changed, 1 insertion(+)
arch/x86/pci/xen.c +1 -0
index b94f727251b6..790550479831 100644
@@ -433,6 +433,7 @@ static struct msi_domain_ops xen_pci_msi_domain_ops = {
 };
 
 static struct msi_domain_info xen_pci_msi_domain_info = {
+	.flags			= MSI_FLAG_PCI_MSIX,
 	.ops			= &xen_pci_msi_domain_ops,
 };
 
Signature · 1 line
2.34.1
#146 lore raw

[tip: x86/urgent] x86/pci/xen: Set MSI_FLAG_PCI_MSIX support in Xen MSI domain

The following commit has been merged into the x86/urgent branch of tip:

The following commit has been merged into the x86/urgent branch of tip:

Commit-ID:     0a3a58de319552525507a3497da86df039a5e4e0
Gitweb:        https://git.kernel.org/tip/0a3a58de319552525507a3497da86df039a5e4e0
Author:        David Woodhouse <[email protected]>
AuthorDate:    Sun, 15 Jan 2023 22:14:19 
Committer:     Thomas Gleixner <[email protected]>
CommitterDate: Mon, 16 Jan 2023 20:40:44 +01:00

x86/pci/xen: Set MSI_FLAG_PCI_MSIX support in Xen MSI domain

The Xen MSI → PIRQ magic does support MSI-X, so advertise it.

(In fact it's better off with MSI-X than MSI, because it's actually broken by design for 32-bit MSI, since it puts the high bits of the PIRQ# into the high 32 bits of the MSI message address, instead of the Extended Destination ID field which is in bits 4-11.

Strictly speaking, this really fixes a much older commit 2e4386eba0c0 ("x86/xen: Wrap XEN MSI management into irqdomain") which failed to set the flag. But that never really mattered until __pci_enable_msix_range() started to check and bail out early. So in 6.2-rc we see failures e.g. to bring up networking on an Amazon EC2 m4.16xlarge instance:

[   41.498694] ena 0000:00:03.0 (unnamed net_device) (uninitialized): Failed to enable MSI-X. irq_cnt -524
[   41.498705] ena 0000:00:03.0: Can not reserve msix vectors
[   41.498712] ena 0000:00:03.0: Failed to enable and set the admin interrupts

Side note: This is the first bug found, and first patch tested, by running Xen guests under QEMU/KVM instead of running under actual Xen.

Fixes 99f3d2797657 ("PCI/MSI: Reject MSI-X early")
---
 arch/x86/pci/xen.c | 1 +
 1 file changed, 1 insertion(+)
arch/x86/pci/xen.c +1 -0
index b94f727..7905504 100644
@@ -433,6 +433,7 @@ static struct msi_domain_ops xen_pci_msi_domain_ops = {
 };
 
 static struct msi_domain_info xen_pci_msi_domain_info = {
+	.flags			= MSI_FLAG_PCI_MSIX,
 	.ops			= &xen_pci_msi_domain_ops,
 };
 
#147 lore raw

PATCH 36/39 PCI/MSI: Validate MSIX contiguous restriction early

In reply to Thomas Gleixner

With interrupt domains the sanity check for MSI-X vector validation can be done _before_ any allocation happens. The sanity check only applies to the allocation functions which have an 'entries' array argument. The entries array is filled by the caller with the requested MSI-X indicies. Some drivers have gaps in the index space which is not supported on all architectures.

The PCI/MSI irqdomain has a 'feature' bit to enforce this validation late during the allocation phase.

Just do it right away before doing any other work along with the other sanity checks on that array.

---
 drivers/pci/msi/msi.c |   11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)
drivers/pci/msi/msi.c +9 -2
@@ -725,13 +725,17 @@ static int msix_capability_init(struct p
 	return ret;
 }
 
-static bool pci_msix_validate_entries(struct msix_entry *entries, int nvec, int hwsize)
+static bool pci_msix_validate_entries(struct pci_dev *dev, struct msix_entry *entries,
+				      int nvec, int hwsize)
 {
+	bool nogap;
 	int i, j;
 
 	if (!entries)
 		return true;
 
+	nogap = pci_msi_domain_supports(dev, MSI_FLAG_MSIX_CONTIGUOUS, DENY_LEGACY);
+
 	for (i = 0; i < nvec; i++) {
 		/* Entry within hardware limit? */
 		if (entries[i].entry >= hwsize)
@@ -742,6 +746,9 @@ static bool pci_msix_validate_entries(st
 			if (entries[i].entry == entries[j].entry)
 				return false;
 		}
+		/* Check for unsupported gaps */
+		if (nogap && entries[i].entry != i)
+			return false;
 	}
 	return true;
 }
@@ -773,7 +780,7 @@ int __pci_enable_msix_range(struct pci_d
 	if (hwsize < 0)
 		return hwsize;
 
-	if (!pci_msix_validate_entries(entries, nvec, hwsize))
+	if (!pci_msix_validate_entries(dev, entries, nvec, hwsize))
 		return -EINVAL;
 
 	/* PCI_IRQ_VIRTUAL is a horrible hack! */
Acked-by on v1 36/39
#148 lore raw

Re: [patch 36/39] PCI/MSI: Validate MSIX contiguous restriction early

Quoted 6 lines · Thomas Gleixner

With interrupt domains the sanity check for MSI-X vector validation can be done _before_ any allocation happens. The sanity check only applies to the allocation functions which have an 'entries' array argument. The entries array is filled by the caller with the requested MSI-X indicies. Some drivers have gaps in the index space which is not supported on all architectures.

Quoted 7 lines

The PCI/MSI irqdomain has a 'feature' bit to enforce this validation late during the allocation phase.

Just do it right away before doing any other work along with the other sanity checks on that array.

s/indicies/indices/ (commit log)
s/irqdomain/irq domain/? IIRC previous logs used "irq domain" s/MSIX/MSI-X/ (subject line)

Quoted 45 lines
---
 drivers/pci/msi/msi.c |   11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)
drivers/pci/msi/msi.c +9 -2
@@ -725,13 +725,17 @@ static int msix_capability_init(struct p
 	return ret;
 }
 
-static bool pci_msix_validate_entries(struct msix_entry *entries, int nvec, int hwsize)
+static bool pci_msix_validate_entries(struct pci_dev *dev, struct msix_entry *entries,
+				      int nvec, int hwsize)
 {
+	bool nogap;
 	int i, j;
 
 	if (!entries)
 		return true;
 
+	nogap = pci_msi_domain_supports(dev, MSI_FLAG_MSIX_CONTIGUOUS, DENY_LEGACY);
+
 	for (i = 0; i < nvec; i++) {
 		/* Entry within hardware limit? */
 		if (entries[i].entry >= hwsize)
@@ -742,6 +746,9 @@ static bool pci_msix_validate_entries(st
 			if (entries[i].entry == entries[j].entry)
 				return false;
 		}
+		/* Check for unsupported gaps */
+		if (nogap && entries[i].entry != i)
+			return false;
 	}
 	return true;
 }
@@ -773,7 +780,7 @@ int __pci_enable_msix_range(struct pci_d
 	if (hwsize < 0)
 		return hwsize;
 
-	if (!pci_msix_validate_entries(entries, nvec, hwsize))
+	if (!pci_msix_validate_entries(dev, entries, nvec, hwsize))
 		return -EINVAL;
 
 	/* PCI_IRQ_VIRTUAL is a horrible hack! */
Reviewed-by on v1 36/39
#149 lore raw

Re: [patch 36/39] PCI/MSI: Validate MSIX contiguous restriction early

In reply to Thomas Gleixner
Quoted 9 lines · Thomas Gleixner

With interrupt domains the sanity check for MSI-X vector validation can be done _before_ any allocation happens. The sanity check only applies to the allocation functions which have an 'entries' array argument. The entries array is filled by the caller with the requested MSI-X indicies. Some drivers have gaps in the index space which is not supported on all architectures.

The PCI/MSI irqdomain has a 'feature' bit to enforce this validation late during the allocation phase.

Quoted 7 lines

Just do it right away before doing any other work along with the other sanity checks on that array.

---
 drivers/pci/msi/msi.c |   11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)

Jason

#150 lore raw

[tip: irq/core] PCI/MSI: Validate MSI-X contiguous restriction early

In reply to Thomas Gleixner
The following commit has been merged into the irq/core branch of tip:

The following commit has been merged into the irq/core branch of tip:

Commit-ID:     4644d22eb673f173252610a93aaaba4c2bff7b41
Gitweb:        https://git.kernel.org/tip/4644d22eb673f173252610a93aaaba4c2bff7b41
Author:        Thomas Gleixner <[email protected]>
AuthorDate:    Fri, 11 Nov 2022 14:55:12 +01:00
Committer:     Thomas Gleixner <[email protected]>
CommitterDate: Thu, 17 Nov 2022 15:15:22 +01:00

PCI/MSI: Validate MSI-X contiguous restriction early

With interrupt domains the sanity check for MSI-X vector validation can be done _before_ any allocation happens. The sanity check only applies to the allocation functions which have an 'entries' array argument. The entries array is filled by the caller with the requested MSI-X indices. Some drivers have gaps in the index space which is not supported on all architectures.

The PCI/MSI irq domain has a 'feature' bit to enforce this validation late during the allocation phase.

Just do it right away before doing any other work along with the other sanity checks on that array.

---
 drivers/pci/msi/msi.c | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)
drivers/pci/msi/msi.c +9 -2
index 0740acd..b94f6da 100644
@@ -725,13 +725,17 @@ out_disable:
 	return ret;
 }
 
-static bool pci_msix_validate_entries(struct msix_entry *entries, int nvec, int hwsize)
+static bool pci_msix_validate_entries(struct pci_dev *dev, struct msix_entry *entries,
+				      int nvec, int hwsize)
 {
+	bool nogap;
 	int i, j;
 
 	if (!entries)
 		return true;
 
+	nogap = pci_msi_domain_supports(dev, MSI_FLAG_MSIX_CONTIGUOUS, DENY_LEGACY);
+
 	for (i = 0; i < nvec; i++) {
 		/* Entry within hardware limit? */
 		if (entries[i].entry >= hwsize)
@@ -742,6 +746,9 @@ static bool pci_msix_validate_entries(struct msix_entry *entries, int nvec, int 
 			if (entries[i].entry == entries[j].entry)
 				return false;
 		}
+		/* Check for unsupported gaps */
+		if (nogap && entries[i].entry != i)
+			return false;
 	}
 	return true;
 }
@@ -773,7 +780,7 @@ int __pci_enable_msix_range(struct pci_dev *dev, struct msix_entry *entries, int
 	if (hwsize < 0)
 		return hwsize;
 
-	if (!pci_msix_validate_entries(entries, nvec, hwsize))
+	if (!pci_msix_validate_entries(dev, entries, nvec, hwsize))
 		return -EINVAL;
 
 	/* PCI_IRQ_VIRTUAL is a horrible hack! */
#151 lore raw

PATCH 37/39 PCI/MSI: Remove redundant msi_check() callback

In reply to Thomas Gleixner

All these sanity checks are now done _before_ any allocation work happens. No point in doing it twice.

---
 drivers/pci/msi/irqdomain.c |   48 --------------------------------------------
 1 file changed, 48 deletions(-)
drivers/pci/msi/irqdomain.c +0 -48
@@ -64,51 +64,6 @@ static irq_hw_number_t pci_msi_domain_ca
 		(pci_domain_nr(dev->bus) & 0xFFFFFFFF) << 27;
 }
 
-static inline bool pci_msi_desc_is_multi_msi(struct msi_desc *desc)
-{
-	return !desc->pci.msi_attrib.is_msix && desc->nvec_used > 1;
-}
-
-/**
- * pci_msi_domain_check_cap - Verify that @domain supports the capabilities
- *			      for @dev
- * @domain:	The interrupt domain to check
- * @info:	The domain info for verification
- * @dev:	The device to check
- *
- * Returns:
- *  0 if the functionality is supported
- *  1 if Multi MSI is requested, but the domain does not support it
- *  -ENOTSUPP otherwise
- */
-static int pci_msi_domain_check_cap(struct irq_domain *domain,
-				    struct msi_domain_info *info,
-				    struct device *dev)
-{
-	struct msi_desc *desc = msi_first_desc(dev, MSI_DESC_ALL);
-
-	/* Special handling to support __pci_enable_msi_range() */
-	if (pci_msi_desc_is_multi_msi(desc) &&
-	    !(info->flags & MSI_FLAG_MULTI_PCI_MSI))
-		return 1;
-
-	if (desc->pci.msi_attrib.is_msix) {
-		if (!(info->flags & MSI_FLAG_PCI_MSIX))
-			return -ENOTSUPP;
-
-		if (info->flags & MSI_FLAG_MSIX_CONTIGUOUS) {
-			unsigned int idx = 0;
-
-			/* Check for gaps in the entry indices */
-			msi_for_each_desc(desc, dev, MSI_DESC_ALL) {
-				if (desc->msi_index != idx++)
-					return -ENOTSUPP;
-			}
-		}
-	}
-	return 0;
-}
-
 static void pci_msi_domain_set_desc(msi_alloc_info_t *arg,
 				    struct msi_desc *desc)
 {
@@ -118,7 +73,6 @@ static void pci_msi_domain_set_desc(msi_
 
 static struct msi_domain_ops pci_msi_domain_ops_default = {
 	.set_desc	= pci_msi_domain_set_desc,
-	.msi_check	= pci_msi_domain_check_cap,
 };
 
 static void pci_msi_domain_update_dom_ops(struct msi_domain_info *info)
@@ -130,8 +84,6 @@ static void pci_msi_domain_update_dom_op
 	} else {
 		if (ops->set_desc == NULL)
 			ops->set_desc = pci_msi_domain_set_desc;
-		if (ops->msi_check == NULL)
-			ops->msi_check = pci_msi_domain_check_cap;
 	}
 }
 
Acked-by on v1 37/39
#152 lore raw

Re: [patch 37/39] PCI/MSI: Remove redundant msi_check() callback

Quoted 4 lines · Thomas Gleixner

All these sanity checks are now done _before_ any allocation work happens. No point in doing it twice.

Quoted 76 lines
---
 drivers/pci/msi/irqdomain.c |   48 --------------------------------------------
 1 file changed, 48 deletions(-)
drivers/pci/msi/irqdomain.c +0 -48
@@ -64,51 +64,6 @@ static irq_hw_number_t pci_msi_domain_ca
 		(pci_domain_nr(dev->bus) & 0xFFFFFFFF) << 27;
 }
 
-static inline bool pci_msi_desc_is_multi_msi(struct msi_desc *desc)
-{
-	return !desc->pci.msi_attrib.is_msix && desc->nvec_used > 1;
-}
-
-/**
- * pci_msi_domain_check_cap - Verify that @domain supports the capabilities
- *			      for @dev
- * @domain:	The interrupt domain to check
- * @info:	The domain info for verification
- * @dev:	The device to check
- *
- * Returns:
- *  0 if the functionality is supported
- *  1 if Multi MSI is requested, but the domain does not support it
- *  -ENOTSUPP otherwise
- */
-static int pci_msi_domain_check_cap(struct irq_domain *domain,
-				    struct msi_domain_info *info,
-				    struct device *dev)
-{
-	struct msi_desc *desc = msi_first_desc(dev, MSI_DESC_ALL);
-
-	/* Special handling to support __pci_enable_msi_range() */
-	if (pci_msi_desc_is_multi_msi(desc) &&
-	    !(info->flags & MSI_FLAG_MULTI_PCI_MSI))
-		return 1;
-
-	if (desc->pci.msi_attrib.is_msix) {
-		if (!(info->flags & MSI_FLAG_PCI_MSIX))
-			return -ENOTSUPP;
-
-		if (info->flags & MSI_FLAG_MSIX_CONTIGUOUS) {
-			unsigned int idx = 0;
-
-			/* Check for gaps in the entry indices */
-			msi_for_each_desc(desc, dev, MSI_DESC_ALL) {
-				if (desc->msi_index != idx++)
-					return -ENOTSUPP;
-			}
-		}
-	}
-	return 0;
-}
-
 static void pci_msi_domain_set_desc(msi_alloc_info_t *arg,
 				    struct msi_desc *desc)
 {
@@ -118,7 +73,6 @@ static void pci_msi_domain_set_desc(msi_
 
 static struct msi_domain_ops pci_msi_domain_ops_default = {
 	.set_desc	= pci_msi_domain_set_desc,
-	.msi_check	= pci_msi_domain_check_cap,
 };
 
 static void pci_msi_domain_update_dom_ops(struct msi_domain_info *info)
@@ -130,8 +84,6 @@ static void pci_msi_domain_update_dom_op
 	} else {
 		if (ops->set_desc == NULL)
 			ops->set_desc = pci_msi_domain_set_desc;
-		if (ops->msi_check == NULL)
-			ops->msi_check = pci_msi_domain_check_cap;
 	}
 }
 
Reviewed-by on v1 37/39
#153 lore raw

Re: [patch 37/39] PCI/MSI: Remove redundant msi_check() callback

In reply to Thomas Gleixner
Quoted 7 lines · Thomas Gleixner

All these sanity checks are now done _before_ any allocation work happens. No point in doing it twice.

---
 drivers/pci/msi/irqdomain.c |   48 --------------------------------------------
 1 file changed, 48 deletions(-)

Much clearer this way

Jason

#154 lore raw

[tip: irq/core] PCI/MSI: Remove redundant msi_check() callback

In reply to Thomas Gleixner
The following commit has been merged into the irq/core branch of tip:

The following commit has been merged into the irq/core branch of tip:

Commit-ID:     9c03b2589da2b2b259a960f2a325731f6b38d115
Gitweb:        https://git.kernel.org/tip/9c03b2589da2b2b259a960f2a325731f6b38d115
Author:        Thomas Gleixner <[email protected]>
AuthorDate:    Fri, 11 Nov 2022 14:55:14 +01:00
Committer:     Thomas Gleixner <[email protected]>
CommitterDate: Thu, 17 Nov 2022 15:15:22 +01:00

PCI/MSI: Remove redundant msi_check() callback

All these sanity checks are now done _before_ any allocation work happens. No point in doing it twice.

---
 drivers/pci/msi/irqdomain.c | 48 +------------------------------------
 1 file changed, 48 deletions(-)
drivers/pci/msi/irqdomain.c +0 -48
index 666ed21..dea2e8c 100644
@@ -64,51 +64,6 @@ static irq_hw_number_t pci_msi_domain_calc_hwirq(struct msi_desc *desc)
 		(pci_domain_nr(dev->bus) & 0xFFFFFFFF) << 27;
 }
 
-static inline bool pci_msi_desc_is_multi_msi(struct msi_desc *desc)
-{
-	return !desc->pci.msi_attrib.is_msix && desc->nvec_used > 1;
-}
-
-/**
- * pci_msi_domain_check_cap - Verify that @domain supports the capabilities
- *			      for @dev
- * @domain:	The interrupt domain to check
- * @info:	The domain info for verification
- * @dev:	The device to check
- *
- * Returns:
- *  0 if the functionality is supported
- *  1 if Multi MSI is requested, but the domain does not support it
- *  -ENOTSUPP otherwise
- */
-static int pci_msi_domain_check_cap(struct irq_domain *domain,
-				    struct msi_domain_info *info,
-				    struct device *dev)
-{
-	struct msi_desc *desc = msi_first_desc(dev, MSI_DESC_ALL);
-
-	/* Special handling to support __pci_enable_msi_range() */
-	if (pci_msi_desc_is_multi_msi(desc) &&
-	    !(info->flags & MSI_FLAG_MULTI_PCI_MSI))
-		return 1;
-
-	if (desc->pci.msi_attrib.is_msix) {
-		if (!(info->flags & MSI_FLAG_PCI_MSIX))
-			return -ENOTSUPP;
-
-		if (info->flags & MSI_FLAG_MSIX_CONTIGUOUS) {
-			unsigned int idx = 0;
-
-			/* Check for gaps in the entry indices */
-			msi_for_each_desc(desc, dev, MSI_DESC_ALL) {
-				if (desc->msi_index != idx++)
-					return -ENOTSUPP;
-			}
-		}
-	}
-	return 0;
-}
-
 static void pci_msi_domain_set_desc(msi_alloc_info_t *arg,
 				    struct msi_desc *desc)
 {
@@ -118,7 +73,6 @@ static void pci_msi_domain_set_desc(msi_alloc_info_t *arg,
 
 static struct msi_domain_ops pci_msi_domain_ops_default = {
 	.set_desc	= pci_msi_domain_set_desc,
-	.msi_check	= pci_msi_domain_check_cap,
 };
 
 static void pci_msi_domain_update_dom_ops(struct msi_domain_info *info)
@@ -130,8 +84,6 @@ static void pci_msi_domain_update_dom_ops(struct msi_domain_info *info)
 	} else {
 		if (ops->set_desc == NULL)
 			ops->set_desc = pci_msi_domain_set_desc;
-		if (ops->msi_check == NULL)
-			ops->msi_check = pci_msi_domain_check_cap;
 	}
 }
 
#155 lore raw

PATCH 38/39 genirq/msi: Remove msi_domain_ops::msi_check()

In reply to Thomas Gleixner

No more users.

---
 include/linux/msi.h |    4 ----
 kernel/irq/msi.c    |   17 +----------------
 2 files changed, 1 insertion(+), 20 deletions(-)
include/linux/msi.h +0 -4
@@ -288,7 +288,6 @@ struct msi_domain_info;
  * @get_hwirq:		Retrieve the resulting hw irq number
  * @msi_init:		Domain specific init function for MSI interrupts
  * @msi_free:		Domain specific function to free a MSI interrupts
- * @msi_check:		Callback for verification of the domain/info/dev data
  * @msi_prepare:	Prepare the allocation of the interrupts in the domain
  * @set_desc:		Set the msi descriptor for an interrupt
  * @domain_alloc_irqs:	Optional function to override the default allocation
@@ -326,9 +325,6 @@ struct msi_domain_ops {
 	void		(*msi_free)(struct irq_domain *domain,
 				    struct msi_domain_info *info,
 				    unsigned int virq);
-	int		(*msi_check)(struct irq_domain *domain,
-				     struct msi_domain_info *info,
-				     struct device *dev);
 	int		(*msi_prepare)(struct irq_domain *domain,
 				       struct device *dev, int nvec,
 				       msi_alloc_info_t *arg);
kernel/irq/msi.c +1 -16
@@ -617,17 +617,9 @@ static int msi_domain_ops_init(struct ir
 	return 0;
 }
 
-static int msi_domain_ops_check(struct irq_domain *domain,
-				struct msi_domain_info *info,
-				struct device *dev)
-{
-	return 0;
-}
-
 static struct msi_domain_ops msi_domain_ops_default = {
 	.get_hwirq		= msi_domain_ops_get_hwirq,
 	.msi_init		= msi_domain_ops_init,
-	.msi_check		= msi_domain_ops_check,
 	.msi_prepare		= msi_domain_ops_prepare,
 	.set_desc		= msi_domain_ops_set_desc,
 	.domain_alloc_irqs	= __msi_domain_alloc_irqs,
@@ -655,8 +647,6 @@ static void msi_domain_update_dom_ops(st
 		ops->get_hwirq = msi_domain_ops_default.get_hwirq;
 	if (ops->msi_init == NULL)
 		ops->msi_init = msi_domain_ops_default.msi_init;
-	if (ops->msi_check == NULL)
-		ops->msi_check = msi_domain_ops_default.msi_check;
 	if (ops->msi_prepare == NULL)
 		ops->msi_prepare = msi_domain_ops_default.msi_prepare;
 	if (ops->set_desc == NULL)
@@ -707,13 +697,8 @@ int msi_domain_prepare_irqs(struct irq_d
 {
 	struct msi_domain_info *info = domain->host_data;
 	struct msi_domain_ops *ops = info->ops;
-	int ret;
-
-	ret = ops->msi_check(domain, info, dev);
-	if (ret == 0)
-		ret = ops->msi_prepare(domain, dev, nvec, arg);
 
-	return ret;
+	return ops->msi_prepare(domain, dev, nvec, arg);
 }
 
 int msi_domain_populate_irqs(struct irq_domain *domain, struct device *dev,
#157 lore raw

[tip: irq/core] genirq/msi: Remove msi_domain_ops:: Msi_check()

In reply to Thomas Gleixner
The following commit has been merged into the irq/core branch of tip:

The following commit has been merged into the irq/core branch of tip:

Commit-ID:     2569f62ca473584a8ba389f455fc00805389f7da
Gitweb:        https://git.kernel.org/tip/2569f62ca473584a8ba389f455fc00805389f7da
Author:        Thomas Gleixner <[email protected]>
AuthorDate:    Fri, 11 Nov 2022 14:55:15 +01:00
Committer:     Thomas Gleixner <[email protected]>
CommitterDate: Thu, 17 Nov 2022 15:15:22 +01:00

genirq/msi: Remove msi_domain_ops:: Msi_check()

No more users.

---
 include/linux/msi.h |  4 ----
 kernel/irq/msi.c    | 17 +----------------
 2 files changed, 1 insertion(+), 20 deletions(-)
include/linux/msi.h +0 -4
index 8b28714..1ce9d5e 100644
@@ -292,7 +292,6 @@ struct msi_domain_info;
  * @get_hwirq:		Retrieve the resulting hw irq number
  * @msi_init:		Domain specific init function for MSI interrupts
  * @msi_free:		Domain specific function to free a MSI interrupts
- * @msi_check:		Callback for verification of the domain/info/dev data
  * @msi_prepare:	Prepare the allocation of the interrupts in the domain
  * @set_desc:		Set the msi descriptor for an interrupt
  * @domain_alloc_irqs:	Optional function to override the default allocation
@@ -330,9 +329,6 @@ struct msi_domain_ops {
 	void		(*msi_free)(struct irq_domain *domain,
 				    struct msi_domain_info *info,
 				    unsigned int virq);
-	int		(*msi_check)(struct irq_domain *domain,
-				     struct msi_domain_info *info,
-				     struct device *dev);
 	int		(*msi_prepare)(struct irq_domain *domain,
 				       struct device *dev, int nvec,
 				       msi_alloc_info_t *arg);
kernel/irq/msi.c +1 -16
index 4fde917..4b99f37 100644
@@ -617,17 +617,9 @@ static int msi_domain_ops_init(struct irq_domain *domain,
 	return 0;
 }
 
-static int msi_domain_ops_check(struct irq_domain *domain,
-				struct msi_domain_info *info,
-				struct device *dev)
-{
-	return 0;
-}
-
 static struct msi_domain_ops msi_domain_ops_default = {
 	.get_hwirq		= msi_domain_ops_get_hwirq,
 	.msi_init		= msi_domain_ops_init,
-	.msi_check		= msi_domain_ops_check,
 	.msi_prepare		= msi_domain_ops_prepare,
 	.set_desc		= msi_domain_ops_set_desc,
 	.domain_alloc_irqs	= __msi_domain_alloc_irqs,
@@ -655,8 +647,6 @@ static void msi_domain_update_dom_ops(struct msi_domain_info *info)
 		ops->get_hwirq = msi_domain_ops_default.get_hwirq;
 	if (ops->msi_init == NULL)
 		ops->msi_init = msi_domain_ops_default.msi_init;
-	if (ops->msi_check == NULL)
-		ops->msi_check = msi_domain_ops_default.msi_check;
 	if (ops->msi_prepare == NULL)
 		ops->msi_prepare = msi_domain_ops_default.msi_prepare;
 	if (ops->set_desc == NULL)
@@ -707,13 +697,8 @@ int msi_domain_prepare_irqs(struct irq_domain *domain, struct device *dev,
 {
 	struct msi_domain_info *info = domain->host_data;
 	struct msi_domain_ops *ops = info->ops;
-	int ret;
-
-	ret = ops->msi_check(domain, info, dev);
-	if (ret == 0)
-		ret = ops->msi_prepare(domain, dev, nvec, arg);
 
-	return ret;
+	return ops->msi_prepare(domain, dev, nvec, arg);
 }
 
 int msi_domain_populate_irqs(struct irq_domain *domain, struct device *dev,
#158 lore raw

PATCH 39/39 x86/apic: Remove X86_IRQ_ALLOC_CONTIGUOUS_VECTORS

In reply to Thomas Gleixner

Now that the PCI/MSI core code does early checking for multi-MSI support X86_IRQ_ALLOC_CONTIGUOUS_VECTORS is not required anymore.

Remove the flag and rely on MSI_FLAG_MULTI_PCI_MSI.

---
 arch/x86/include/asm/irqdomain.h    |    4 +---
 arch/x86/kernel/apic/msi.c          |    6 ++----
 arch/x86/kernel/apic/vector.c       |    4 ----
 drivers/iommu/amd/iommu.c           |    7 -------
 drivers/iommu/intel/irq_remapping.c |    7 -------
 drivers/pci/controller/pci-hyperv.c |   15 +--------------
 6 files changed, 4 insertions(+), 39 deletions(-)
arch/x86/include/asm/irqdomain.h +1 -3
@@ -7,9 +7,7 @@
 
 #ifdef CONFIG_X86_LOCAL_APIC
 enum {
-	/* Allocate contiguous CPU vectors */
-	X86_IRQ_ALLOC_CONTIGUOUS_VECTORS		= 0x1,
-	X86_IRQ_ALLOC_LEGACY				= 0x2,
+	X86_IRQ_ALLOC_LEGACY				= 0x1,
 };
 
 extern int x86_fwspec_is_ioapic(struct irq_fwspec *fwspec);
arch/x86/kernel/apic/msi.c +2 -4
@@ -161,12 +161,10 @@ int pci_msi_prepare(struct irq_domain *d
 		    msi_alloc_info_t *arg)
 {
 	init_irq_alloc_info(arg, NULL);
-	if (to_pci_dev(dev)->msix_enabled) {
+	if (to_pci_dev(dev)->msix_enabled)
 		arg->type = X86_IRQ_ALLOC_TYPE_PCI_MSIX;
-	} else {
+	else
 		arg->type = X86_IRQ_ALLOC_TYPE_PCI_MSI;
-		arg->flags |= X86_IRQ_ALLOC_CONTIGUOUS_VECTORS;
-	}
 
 	return 0;
 }
arch/x86/kernel/apic/vector.c +0 -4
@@ -539,10 +539,6 @@ static int x86_vector_alloc_irqs(struct
 	if (disable_apic)
 		return -ENXIO;
 
-	/* Currently vector allocator can't guarantee contiguous allocations */
-	if ((info->flags & X86_IRQ_ALLOC_CONTIGUOUS_VECTORS) && nr_irqs > 1)
-		return -ENOSYS;
-
 	/*
 	 * Catch any attempt to touch the cascade interrupt on a PIC
 	 * equipped system.
drivers/iommu/amd/iommu.c +0 -7
@@ -3297,13 +3297,6 @@ static int irq_remapping_alloc(struct ir
 	if (nr_irqs > 1 && info->type != X86_IRQ_ALLOC_TYPE_PCI_MSI)
 		return -EINVAL;
 
-	/*
-	 * With IRQ remapping enabled, don't need contiguous CPU vectors
-	 * to support multiple MSI interrupts.
-	 */
-	if (info->type == X86_IRQ_ALLOC_TYPE_PCI_MSI)
-		info->flags &= ~X86_IRQ_ALLOC_CONTIGUOUS_VECTORS;
-
 	sbdf = get_devid(info);
 	if (sbdf < 0)
 		return -EINVAL;
drivers/iommu/intel/irq_remapping.c +0 -7
@@ -1337,13 +1337,6 @@ static int intel_irq_remapping_alloc(str
 	if (nr_irqs > 1 && info->type != X86_IRQ_ALLOC_TYPE_PCI_MSI)
 		return -EINVAL;
 
-	/*
-	 * With IRQ remapping enabled, don't need contiguous CPU vectors
-	 * to support multiple MSI interrupts.
-	 */
-	if (info->type == X86_IRQ_ALLOC_TYPE_PCI_MSI)
-		info->flags &= ~X86_IRQ_ALLOC_CONTIGUOUS_VECTORS;
-
 	ret = irq_domain_alloc_irqs_parent(domain, virq, nr_irqs, arg);
 	if (ret < 0)
 		return ret;
drivers/pci/controller/pci-hyperv.c +1 -14
@@ -611,20 +611,7 @@ static unsigned int hv_msi_get_int_vecto
 	return cfg->vector;
 }
 
-static int hv_msi_prepare(struct irq_domain *domain, struct device *dev,
-			  int nvec, msi_alloc_info_t *info)
-{
-	int ret = pci_msi_prepare(domain, dev, nvec, info);
-
-	/*
-	 * By using the interrupt remapper in the hypervisor IOMMU, contiguous
-	 * CPU vectors is not needed for multi-MSI
-	 */
-	if (info->type == X86_IRQ_ALLOC_TYPE_PCI_MSI)
-		info->flags &= ~X86_IRQ_ALLOC_CONTIGUOUS_VECTORS;
-
-	return ret;
-}
+#define hv_msi_prepare		pci_msi_prepare
 
 /**
  * hv_arch_irq_unmask() - "Unmask" the IRQ by setting its current
Reviewed-by on v1 39/39
#159 lore raw

Re: [patch 39/39] x86/apic: Remove X86_IRQ_ALLOC_CONTIGUOUS_VECTORS

Quoted 4 lines · Thomas Gleixner

Now that the PCI/MSI core code does early checking for multi-MSI support X86_IRQ_ALLOC_CONTIGUOUS_VECTORS is not required anymore.

Remove the flag and rely on MSI_FLAG_MULTI_PCI_MSI.

Jason

#160 lore raw

Re: [patch 39/39] x86/apic: Remove X86_IRQ_ALLOC_CONTIGUOUS_VECTORS

Jason, Bjorn, Ashok!

Quoted 7 lines · Jason Gunthorpe
Quoted 4 lines · Thomas Gleixner

Now that the PCI/MSI core code does early checking for multi-MSI support X86_IRQ_ALLOC_CONTIGUOUS_VECTORS is not required anymore.

Remove the flag and rely on MSI_FLAG_MULTI_PCI_MSI.

Thanks for taking the time to go through this pile!

tglx

#161 lore raw

[tip: irq/core] x86/apic: Remove X86_IRQ_ALLOC_CONTIGUOUS_VECTORS

In reply to Thomas Gleixner
The following commit has been merged into the irq/core branch of tip:

The following commit has been merged into the irq/core branch of tip:

Commit-ID:     d474d92d70250d43e7ce0c7cb8623f31ee7c40f6
Gitweb:        https://git.kernel.org/tip/d474d92d70250d43e7ce0c7cb8623f31ee7c40f6
Author:        Thomas Gleixner <[email protected]>
AuthorDate:    Fri, 11 Nov 2022 14:55:17 +01:00
Committer:     Thomas Gleixner <[email protected]>
CommitterDate: Thu, 17 Nov 2022 15:15:22 +01:00

x86/apic: Remove X86_IRQ_ALLOC_CONTIGUOUS_VECTORS

Now that the PCI/MSI core code does early checking for multi-MSI support X86_IRQ_ALLOC_CONTIGUOUS_VECTORS is not required anymore.

Remove the flag and rely on MSI_FLAG_MULTI_PCI_MSI.

---
 arch/x86/include/asm/irqdomain.h    |  4 +---
 arch/x86/kernel/apic/msi.c          |  6 ++----
 arch/x86/kernel/apic/vector.c       |  4 ----
 drivers/iommu/amd/iommu.c           |  7 -------
 drivers/iommu/intel/irq_remapping.c |  7 -------
 drivers/pci/controller/pci-hyperv.c | 15 +--------------
 6 files changed, 4 insertions(+), 39 deletions(-)
arch/x86/include/asm/irqdomain.h +1 -3
index 125c23b..30c325c 100644
@@ -7,9 +7,7 @@
 
 #ifdef CONFIG_X86_LOCAL_APIC
 enum {
-	/* Allocate contiguous CPU vectors */
-	X86_IRQ_ALLOC_CONTIGUOUS_VECTORS		= 0x1,
-	X86_IRQ_ALLOC_LEGACY				= 0x2,
+	X86_IRQ_ALLOC_LEGACY				= 0x1,
 };
 
 extern int x86_fwspec_is_ioapic(struct irq_fwspec *fwspec);
arch/x86/kernel/apic/msi.c +2 -4
index 7517eb0..248a6a5 100644
@@ -161,12 +161,10 @@ int pci_msi_prepare(struct irq_domain *domain, struct device *dev, int nvec,
 		    msi_alloc_info_t *arg)
 {
 	init_irq_alloc_info(arg, NULL);
-	if (to_pci_dev(dev)->msix_enabled) {
+	if (to_pci_dev(dev)->msix_enabled)
 		arg->type = X86_IRQ_ALLOC_TYPE_PCI_MSIX;
-	} else {
+	else
 		arg->type = X86_IRQ_ALLOC_TYPE_PCI_MSI;
-		arg->flags |= X86_IRQ_ALLOC_CONTIGUOUS_VECTORS;
-	}
 
 	return 0;
 }
arch/x86/kernel/apic/vector.c +0 -4
index 3e6f6b4..c1efebd 100644
@@ -539,10 +539,6 @@ static int x86_vector_alloc_irqs(struct irq_domain *domain, unsigned int virq,
 	if (disable_apic)
 		return -ENXIO;
 
-	/* Currently vector allocator can't guarantee contiguous allocations */
-	if ((info->flags & X86_IRQ_ALLOC_CONTIGUOUS_VECTORS) && nr_irqs > 1)
-		return -ENOSYS;
-
 	/*
 	 * Catch any attempt to touch the cascade interrupt on a PIC
 	 * equipped system.
drivers/iommu/amd/iommu.c +0 -7
index 8ece864..72dfe57 100644
@@ -3297,13 +3297,6 @@ static int irq_remapping_alloc(struct irq_domain *domain, unsigned int virq,
 	if (nr_irqs > 1 && info->type != X86_IRQ_ALLOC_TYPE_PCI_MSI)
 		return -EINVAL;
 
-	/*
-	 * With IRQ remapping enabled, don't need contiguous CPU vectors
-	 * to support multiple MSI interrupts.
-	 */
-	if (info->type == X86_IRQ_ALLOC_TYPE_PCI_MSI)
-		info->flags &= ~X86_IRQ_ALLOC_CONTIGUOUS_VECTORS;
-
 	sbdf = get_devid(info);
 	if (sbdf < 0)
 		return -EINVAL;
drivers/iommu/intel/irq_remapping.c +0 -7
index 0b80a27..a914eba 100644
@@ -1337,13 +1337,6 @@ static int intel_irq_remapping_alloc(struct irq_domain *domain,
 	if (nr_irqs > 1 && info->type != X86_IRQ_ALLOC_TYPE_PCI_MSI)
 		return -EINVAL;
 
-	/*
-	 * With IRQ remapping enabled, don't need contiguous CPU vectors
-	 * to support multiple MSI interrupts.
-	 */
-	if (info->type == X86_IRQ_ALLOC_TYPE_PCI_MSI)
-		info->flags &= ~X86_IRQ_ALLOC_CONTIGUOUS_VECTORS;
-
 	ret = irq_domain_alloc_irqs_parent(domain, virq, nr_irqs, arg);
 	if (ret < 0)
 		return ret;
drivers/pci/controller/pci-hyperv.c +1 -14
index ba64284..1dee55d 100644
@@ -611,20 +611,7 @@ static unsigned int hv_msi_get_int_vector(struct irq_data *data)
 	return cfg->vector;
 }
 
-static int hv_msi_prepare(struct irq_domain *domain, struct device *dev,
-			  int nvec, msi_alloc_info_t *info)
-{
-	int ret = pci_msi_prepare(domain, dev, nvec, info);
-
-	/*
-	 * By using the interrupt remapper in the hypervisor IOMMU, contiguous
-	 * CPU vectors is not needed for multi-MSI
-	 */
-	if (info->type == X86_IRQ_ALLOC_TYPE_PCI_MSI)
-		info->flags &= ~X86_IRQ_ALLOC_CONTIGUOUS_VECTORS;
-
-	return ret;
-}
+#define hv_msi_prepare		pci_msi_prepare
 
 /**
  * hv_arch_irq_unmask() - "Unmask" the IRQ by setting its current
Reviewed-by on the v1 series
#162 lore raw
In reply to Thomas Gleixner
Quoted 9 lines

From: Thomas Gleixner <[email protected]>
Sent: Friday, November 11, 2022 9:54 PM

Enough of history and theory. Here comes part 1:

This is just a cleanup and a reorganisation of the PCI/MSI code which became quite an unreadable mess over time. There is no intentional functional change in this series.

Quoted 6 lines

It's just a separate step to make the subsequent changes in the infrastructure easier both to implement and to review.

Thanks,

tglx

The entire series looks good to me except a couple nits replied in individual patches: