cxl: Add general MSI-X/MSI irq support
Davidlohr Bueso
maintainerv3 cover cxl: Add general MSI-X/MSI irq support
Changes from v2 (https://lore.kernel.org/linux-cxl/[email protected]/):
- Moved vector allocation further down the line during the probing (Ira).
- Replaced the irq enum with cxlds->has_irq (Jonathan).
- Use pci_request_irq (Ira).
- Small fixlets/improvements.
Hi,
Patch 1 contains updates for the generic table handling. Patch 2 populates the table and adds mailbox support.
Applies against 'linux-cxl/pending'. Please consider for v6.3.
Thanks!
Davidlohr Bueso (2):
cxl/pci: Add generic MSI-X/MSI irq support
cxl/mbox: Wire up irq support
drivers/cxl/cxl.h | 1 + drivers/cxl/cxlmem.h | 3 ++ drivers/cxl/pci.c | 99 ++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 103 insertions(+)
Signature · 1 line
2.38.0
Davidlohr Bueso
maintainerPATCH v3 1/2 cxl/pci: Add generic MSI-X/MSI irq support
Introduce a generic irq table for CXL components/features that can have standard irq support - DOE requires dynamic vector sizing and is not considered here. For now the table is empty.
Create an infrastructure to query the max vectors required for the CXL device. Upon successful allocation, users can plug in their respective isr at any point thereafter, which is supported by a new cxlds->has_irq flag, for example, if the irq setup is not done in the PCI driver, such as the case of the CXL-PMU.
--- drivers/cxl/cxlmem.h | 3 ++ drivers/cxl/pci.c | 72 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 75 insertions(+)
drivers/cxl/cxlmem.h +3 -0
@@ -211,6 +211,7 @@ struct cxl_endpoint_dvsec_info {* @info: Cached DVSEC information about the device. * @serial: PCIe Device Serial Number * @doe_mbs: PCI DOE mailbox array +* @has_irq: PCIe MSI-X/MSI support * @mbox_send: @dev specific transport for transmitting mailbox commands * * See section 8.2.9.5.2 Capacity Configuration and Label Storage for
@@ -247,6 +248,8 @@ struct cxl_dev_state {struct xarray doe_mbs; +bool has_irq; +int (*mbox_send)(struct cxl_dev_state *cxlds,struct cxl_mbox_cmd *cmd); };
drivers/cxl/pci.c +72 -0
84 lines not on this page · raw message
Signature · 1 line
2.38.0
Re: [PATCH 1/2] cxl/pci: Add generic MSI-X/MSI irq support
Quoted 4 lines · On Mon, 17 Oct 2022 20:00:09 -0700 Davidlohr Bueso <[email protected]> wrote:
Introduce a generic irq table for CXL components/features that can have standard irq support - DOE requires dynamic vector sizing and is not considered here. For now the table is empty.
Quoted 8 lines
Create an infrastructure to query the max vectors required for the CXL device. Upon successful allocation, users can plug in their respective isr at any point thereafter, which is supported by a new
cxlds->has_irqflag, for example, if the irq setup is not done in the PCI driver, such as the case of the CXL-PMU.Reviewed-by Dave Jiang <[email protected]>Signed-off-by Davidlohr Bueso <[email protected]>
A few nitpicks inline.
With the comment one tidied up (other one optional)
I'll rebase my cpmu code on top of this shortly.
Jonathan
Quoted 50 lines
--- drivers/cxl/cxlmem.h | 3 ++ drivers/cxl/pci.c | 72 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 75 insertions(+)drivers/cxl/cxlmem.h +3
-0@@ -211,6 +211,7 @@ struct cxl_endpoint_dvsec_info {* @info: Cached DVSEC information about the device. * @serial: PCIe Device Serial Number * @doe_mbs: PCI DOE mailbox array +* @has_irq: PCIe MSI-X/MSI support * @mbox_send: @dev specific transport for transmitting mailbox commands * * See section 8.2.9.5.2 Capacity Configuration and Label Storage for @@ -247,6 +248,8 @@ struct cxl_dev_state {struct xarray doe_mbs; +bool has_irq; +int (*mbox_send)(struct cxl_dev_state *cxlds,struct cxl_mbox_cmd *cmd); };drivers/cxl/pci.c +16
-0@@ -428,6 +428,73 @@ static void devm_cxl_pci_create_doe(struct cxl_dev_state *cxlds) } } +/** +* struct cxl_irq_cap - CXL feature that is capable of receiving MSI-X/MSI irqs. +* +* @name: Name of the device/component generating this interrupt. +* @get_max_msgnum: Get the feature's largest interrupt message number. If the +* feature does not have the Interrupt Supported bit set, then +* return -1. +*/ +struct cxl_irq_cap { +const char *name; +int (*get_max_msgnum)(struct cxl_dev_state *cxlds); +}; + +static const struct cxl_irq_cap cxl_irq_cap_table[] = { +NULL +};
drivers/cxl/pci.c static void devm_cxl_pci_create_doe(struct cxl_dev_state *cxlds) · line 456
Quoted 10 lines
drivers/cxl/pci.c +10
-0+ +static void cxl_pci_free_irq_vectors (void *data) +{ +pci_free_irq_vectors (data); +} + +/* +* Attempt to allocate the largest amount of necessary vectors. +* +* Returns 0 upon a successful allocation of *all* vectors, or a
Technically not all vectors. If we wanted to do that we could just directly query that via pci_msix_vec_count() etc that gets it from the MSIX capability. That's frowned upon because it's common to stick lots of extra vectors on the end for stuff that linux never cares about (debug etc, or optional features).
All vectors up to the maximum one the code uses would be more accurate.
Quoted 45 lines
drivers/cxl/pci.c +41
-0+* negative value otherwise. +*/ +static int cxl_pci_alloc_irq_vectors (struct cxl_dev_state *cxlds) +{ +struct device *dev = cxlds->dev; +struct pci_dev *pdev =to_pci_dev (dev); +int rc, i, vectors =-1 ; + +for (i =0 ; i <ARRAY_SIZE (cxl_irq_cap_table); i++) { +int irq; + +if (!cxl_irq_cap_table[i].get_max_msgnum) +continue ; + + irq = cxl_irq_cap_table[i].get_max_msgnum (cxlds); + vectors =max_t (int, irq, vectors); + } + +/* +* Semantically lack of irq support is not an error, but we +* still fail to allocate, so return negative. +*/ +if (vectors ==-1 ) +return -1 ; + + vectors++; + rc =pci_alloc_irq_vectors (pdev, vectors, vectors, +PCI_IRQ_MSIX |PCI_IRQ_MSI ); +if (rc <0 ) +return rc; + +if (rc != vectors) { +dev_dbg (dev,"Not enough interrupts; use polling instead.\n" ); +/* some got allocated, clean them up */ +cxl_pci_free_irq_vectors (pdev); +return -ENOSPC ; + } + +return devm_add_action_or_reset (dev, cxl_pci_free_irq_vectors, pdev); +} +static int cxl_pci_probe (struct pci_dev *pdev,const struct pci_device_id *id) {struct cxl_register_map map;
drivers/cxl/pci.c static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id) · line 568
Quoted 8 lines
drivers/cxl/pci.c +5
-0@@ -494,6 +561,11 @@ static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)if (rc)return rc; +if (!cxl_pci_alloc_irq_vectors (cxlds)) { + cxlds->has_irq = true; + }else + cxlds->has_irq = false; +
cxlds->has_irq = !(cxl_pci_aloc_irq_vectors(cxlds) < 0);
maybe...
Quoted 3 lines
drivers/cxl/pci.c +0
-0cxlmd =devm_cxl_add_memdev (cxlds);if (IS_ERR (cxlmd))return PTR_ERR (cxlmd);
Re: [PATCH 1/2] cxl/pci: Add generic MSI-X/MSI irq support
Quoted 16 lines · On Tue, 18 Oct 2022 10:36:19 +0100 Jonathan Cameron…
Quoted 4 lines · On Mon, 17 Oct 2022 20:00:09 -0700 Davidlohr Bueso <[email protected]> wrote:
Introduce a generic irq table for CXL components/features that can have standard irq support - DOE requires dynamic vector sizing and is not considered here. For now the table is empty.
Quoted 8 lines
Create an infrastructure to query the max vectors required for the CXL device. Upon successful allocation, users can plug in their respective isr at any point thereafter, which is supported by a new
cxlds->has_irqflag, for example, if the irq setup is not done in the PCI driver, such as the case of the CXL-PMU.Reviewed-by Dave Jiang <[email protected]>Signed-off-by Davidlohr Bueso <[email protected]>
Quoted 6 lines
A few nitpicks inline.
With the comment one tidied up (other one optional)
Reviewed-by Jonathan Cameron <[email protected]>I'll rebase my cpmu code on top of this shortly.
Hi Davidlohr,
Doing the CPMU rebase has shown up that using this generic infrastructure ends up rather ugly.
Previously I had a local array to manage the required register maps that was then freed. Now I have to move that into the cxl device state just so I can get at it from the irq finding callback.
So I have an extra step to be able to use this generic framework.
1. Query how many CPMU devices there are. Stash that and register map info in cxlds. I could do this in the callback but that's really really horrible layering issue as most of what is done has nothing to do with finding the vector numbers.
2. The callback below to find those numbers
3. Registration of the cpmu devices.
Reality is that it is cleaner to more or less ignore the infrastructure proposed in this patch.
1. Query how many CPMU devices there are. Whilst there stash the maximim cpmu vector number in the cxlds.
2. Run a stub in this infrastructure that does max(irq, cxlds->irq_num);
3. Carry on as before.
Thus destroying the point of this infrastructure for that usecase at least and leaving an extra bit of state in the cxl_dev_state that is just to squirt a value into the callback...
So with that in mind I'm withdrawing the RB above. This looks to be an idea that with hindsight doesn't necessarily pan out.
Long hand equivalent with the specific handling needed for each case is probably going to be neater than walking a table of much more restricted callbacks. Maybe there is a nice way to fit the CPMU registration into this infrastructure, but I'm not immediately seeing it.
One other note inline via a compiler warning.
Jonathan
Quoted 38 lines
Jonathan
Quoted 34 lines
--- drivers/cxl/cxlmem.h | 3 ++ drivers/cxl/pci.c | 72 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 75 insertions(+)drivers/cxl/cxlmem.h +3
-0@@ -211,6 +211,7 @@ struct cxl_endpoint_dvsec_info {* @info: Cached DVSEC information about the device. * @serial: PCIe Device Serial Number * @doe_mbs: PCI DOE mailbox array +* @has_irq: PCIe MSI-X/MSI support * @mbox_send: @dev specific transport for transmitting mailbox commands * * See section 8.2.9.5.2 Capacity Configuration and Label Storage for @@ -247,6 +248,8 @@ struct cxl_dev_state {struct xarray doe_mbs; +bool has_irq; +int (*mbox_send)(struct cxl_dev_state *cxlds,struct cxl_mbox_cmd *cmd); };drivers/cxl/pci.c +0
-0@@ -428,6 +428,73 @@ static void devm_cxl_pci_create_doe(struct cxl_dev_state *cxlds) } }
Quoted 15 lines
Quoted 5 lines
drivers/cxl/pci.c +5
-0+/** +* struct cxl_irq_cap - CXL feature that is capable of receiving MSI-X/MSI irqs. +* +* @name: Name of the device/component generating this interrupt. +* @get_max_msgnum: Get the feature's largest interrupt message number. If the drivers/cxl/pci.c static void devm_cxl_pci_create_doe(struct cxl_dev_state *cxlds) · line 445
Quoted 10 lines
drivers/cxl/pci.c +10
-0+* feature does not have the Interrupt Supported bit set, then +* return -1. +*/ +struct cxl_irq_cap { +const char *name; +int (*get_max_msgnum)(struct cxl_dev_state *cxlds); +}; + +static const struct cxl_irq_cap cxl_irq_cap_table[] = { +NULL
That's not valid, just make it empty instead.
Quoted 81 lines
drivers/cxl/pci.c static void devm_cxl_pci_create_doe(struct cxl_dev_state *cxlds) · line 456
Quoted 11 lines
drivers/cxl/pci.c +11
-0+}; + +static void cxl_pci_free_irq_vectors (void *data) +{ +pci_free_irq_vectors (data); +} + +/* +* Attempt to allocate the largest amount of necessary vectors. +* +* Returns 0 upon a successful allocation of *all* vectors, or a Technically not all vectors. If we wanted to do that we could just directly query that via
pci_msix_vec_count()etc that gets it from the MSIX capability. That's frowned upon because it's common to stick lots of extra vectors on the end for stuff that linux never cares about (debug etc, or optional features).All vectors up to the maximum one the code uses would be more accurate.
Quoted 45 lines
drivers/cxl/pci.c +41
-0+* negative value otherwise. +*/ +static int cxl_pci_alloc_irq_vectors (struct cxl_dev_state *cxlds) +{ +struct device *dev = cxlds->dev; +struct pci_dev *pdev =to_pci_dev (dev); +int rc, i, vectors =-1 ; + +for (i =0 ; i <ARRAY_SIZE (cxl_irq_cap_table); i++) { +int irq; + +if (!cxl_irq_cap_table[i].get_max_msgnum) +continue ; + + irq = cxl_irq_cap_table[i].get_max_msgnum (cxlds); + vectors =max_t (int, irq, vectors); + } + +/* +* Semantically lack of irq support is not an error, but we +* still fail to allocate, so return negative. +*/ +if (vectors ==-1 ) +return -1 ; + + vectors++; + rc =pci_alloc_irq_vectors (pdev, vectors, vectors, +PCI_IRQ_MSIX |PCI_IRQ_MSI ); +if (rc <0 ) +return rc; + +if (rc != vectors) { +dev_dbg (dev,"Not enough interrupts; use polling instead.\n" ); +/* some got allocated, clean them up */ +cxl_pci_free_irq_vectors (pdev); +return -ENOSPC ; + } + +return devm_add_action_or_reset (dev, cxl_pci_free_irq_vectors, pdev); +} +static int cxl_pci_probe (struct pci_dev *pdev,const struct pci_device_id *id) {struct cxl_register_map map;drivers/cxl/pci.c static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id) · line 568
Quoted 8 lines
drivers/cxl/pci.c +5
-0@@ -494,6 +561,11 @@ static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)if (rc)return rc; +if (!cxl_pci_alloc_irq_vectors (cxlds)) { + cxlds->has_irq = true; + }else + cxlds->has_irq = false; +cxlds->has_irq = !(cxl_pci_aloc_irq_vectors(cxlds) < 0);maybe...
Quoted 3 lines
drivers/cxl/pci.c +0
-0cxlmd =devm_cxl_add_memdev (cxlds);if (IS_ERR (cxlmd))return PTR_ERR (cxlmd);
Davidlohr Bueso
maintainerRe: [PATCH 1/2] cxl/pci: Add generic MSI-X/MSI irq support
Quoted 11 lines · Jonathan Cameron
Reality is that it is cleaner to more or less ignore the infrastructure proposed in this patch.
1. Query how many CPMU devices there are. Whilst there stash the maximim cpmu vector number in the cxlds.
2. Run a stub in this infrastructure that doesmax(irq, cxlds->irq_num);
3. Carry on as before.Thus destroying the point of this infrastructure for that usecase at least and leaving an extra bit of state in the
cxl_dev_statethat is just to squirt a value into the callback...
If it doesn't fit, then it doesn't fit.
However, while I was expecting pass one to be in the callback, I wasn't expecting that both pass 1 and 2 shared the cpmu_regs_array. If the array could be reconstructed during pass 2, then it would fit a bit better; albeit the extra allocation, cycles etc., but this is probing phase, so overhead isn't that important (and cpmu_count isn't big enough to matter).
But if we're going to go with a free-for-all approach, can we establish who goes for the initial pci_alloc_irq_vectors()? I think perhaps mbox since it's the most straightforward and with least requirements, I'm also unsure of the status yet to merge events and pmu, but regardless they are still larger patchsets. If folks agree I can send a new mbox-only patch.
Thanks,
Davidlohr
Re: [PATCH 1/2] cxl/pci: Add generic MSI-X/MSI irq support
Quoted 22 lines · Davidlohr Bueso
Quoted 11 lines · Jonathan Cameron
Reality is that it is cleaner to more or less ignore the infrastructure proposed in this patch.
1. Query how many CPMU devices there are. Whilst there stash the maximim cpmu vector number in the cxlds.
2. Run a stub in this infrastructure that doesmax(irq, cxlds->irq_num);
3. Carry on as before.Thus destroying the point of this infrastructure for that usecase at least and leaving an extra bit of state in the
cxl_dev_statethat is just to squirt a value into the callback...If it doesn't fit, then it doesn't fit.
However, while I was expecting pass one to be in the callback, I wasn't expecting that both pass 1 and 2 shared the
cpmu_regs_array. If the array could be reconstructed during pass 2, then it would fit a bit better; albeit the extra allocation, cycles etc., but this is probing phase, so overhead isn't that important (andcpmu_countisn't big enough to matter).
Quoted 6 lines
But if we're going to go with a free-for-all approach, can we establish who goes for the initial
pci_alloc_irq_vectors()? I think perhaps mbox since it's the most straightforward and with least requirements, I'm also unsure of the status yet to merge events and pmu, but regardless they are still larger patchsets. If folks agree I can send a new mbox-only patch.
I think there needs to be some mechanism for all of the sub-device-functions to report their max required vectors.
I don't think that the mbox code is necessarily the code which should need to know about all those other sub-device-thingys. But it could certainly take some 'max vectors' value that probe passed to it.
I'm still not sure how dropping this infrastructure makes Jonathan's code cleaner. I still think there will need to be 2 passes over the number of CPMU's.
Ira
Quoted 3 lines
Thanks,
Davidlohr
Re: [PATCH 1/2] cxl/pci: Add generic MSI-X/MSI irq support
Quoted 15 lines · On Thu, 20 Oct 2022 21:18:58 -0700 Ira Weiny <[email protected]> wrote:
Quoted 14 lines · Davidlohr Bueso
Quoted 11 lines · Jonathan Cameron
Reality is that it is cleaner to more or less ignore the infrastructure proposed in this patch.
1. Query how many CPMU devices there are. Whilst there stash the maximim cpmu vector number in the cxlds.
2. Run a stub in this infrastructure that doesmax(irq, cxlds->irq_num);
3. Carry on as before.Thus destroying the point of this infrastructure for that usecase at least and leaving an extra bit of state in the
cxl_dev_statethat is just to squirt a value into the callback...
Quoted 7 lines
Quoted 7 lines
If it doesn't fit, then it doesn't fit.
However, while I was expecting pass one to be in the callback, I wasn't expecting that both pass 1 and 2 shared the
cpmu_regs_array. If the array could be reconstructed during pass 2, then it would fit a bit better; albeit the extra allocation, cycles etc., but this is probing phase, so overhead isn't that important (andcpmu_countisn't big enough to matter).
I thought about that approach, but it's really ugly to have to do
1) For the IRQ number gathering.
a) Parse 1 to count CPMUs
b) Parse 2 to get the register maps - grab the irq numbers and unmap them again
2) For the CPMU registration
a) Parse 3 to count CPMUs (we could stash the number of CPMUS form 1a) but that's no advantage over stashing the max irq in current proposal. Both are putting state where it's not relevant or wanted just to make it available in a callback. This way is even worse because it's getting stashed as a side effect of a parse in a function doing something different.
b) Parse 4 to get the register maps and actually create the devices. Could have stashed this earlier as well, but same 'side effects' argument applies.
Sure, can move to this however with appropriate comments on why we are playing these games because otherwise I suspect a future 'cleanup' would remove double, double pass.
To allow for an irq registration wrapper that turns a series of straight line calls into callbacks in an array. The straight line calls aren't exactly complex in the first place.
//find cpmu filling in cxl_cpmu_reg_maps.
max_irq = -1
rc = cxl_mailbox_get_irq()
if (rc < 0)
return rc;
max_irq = max(max_irq, rc);
rc = cxl_events_get_irq()
if (rc < 0)
return rc;
max_irq = max(max_irq, rc);
rc = cxl_cpmus_get_irq(cxl_cpmu_reg_maps);
if (rc < 0)
return rc;
max_irq = max(max_irq, rC);
...
if (irq > 0) {
pci_get... }
//create all the devices...
Quoted 11 lines
Quoted 7 lines
But if we're going to go with a free-for-all approach, can we establish who goes for the initial
pci_alloc_irq_vectors()? I think perhaps mbox since it's the most straightforward and with least requirements, I'm also unsure of the status yet to merge events and pmu, but regardless they are still larger patchsets. If folks agree I can send a new mbox-only patch.I think there needs to be some mechanism for all of the sub-device-functions to report their max required vectors.
Quoted 8 lines
I don't think that the mbox code is necessarily the code which should need to know about all those other sub-device-thingys. But it could certainly take some 'max vectors' value that probe passed to it.
I'm still not sure how dropping this infrastructure makes Jonathan's code cleaner. I still think there will need to be 2 passes over the number of CPMU's.
Primarily that there is no need to stash anything about the CPMUs in the cxl_device_state (option 1) or repeat all the counting and discovery logic twice (option 2).
I can live with it (it's what we have to do in pcie port for the equivalent) but the wrapped up version feels like a false optimization.
Saves a few lines of code and adds a bunch of complexity elsewhere that looks to me to outweigh that saving.
If people are convinced this is the way to go then fair enough, but be prepared for the ugly corners!
Jonathan
Quoted 6 lines
Ira
Quoted 3 lines
Thanks,
Davidlohr
Davidlohr Bueso
maintainerRe: [PATCH 1/2] cxl/pci: Add generic MSI-X/MSI irq support
Quoted 65 lines · Jonathan Cameron
Quoted 15 lines · On Thu, 20 Oct 2022 21:18:58 -0700 Ira Weiny <[email protected]> wrote:
Quoted 14 lines · Davidlohr Bueso
Quoted 11 lines · Jonathan Cameron
Reality is that it is cleaner to more or less ignore the infrastructure proposed in this patch.
1. Query how many CPMU devices there are. Whilst there stash the maximim cpmu vector number in the cxlds.
2. Run a stub in this infrastructure that doesmax(irq, cxlds->irq_num);
3. Carry on as before.Thus destroying the point of this infrastructure for that usecase at least and leaving an extra bit of state in the
cxl_dev_statethat is just to squirt a value into the callback...Quoted 7 lines
Quoted 7 lines
If it doesn't fit, then it doesn't fit.
However, while I was expecting pass one to be in the callback, I wasn't expecting that both pass 1 and 2 shared the
cpmu_regs_array. If the array could be reconstructed during pass 2, then it would fit a bit better; albeit the extra allocation, cycles etc., but this is probing phase, so overhead isn't that important (andcpmu_countisn't big enough to matter).I thought about that approach, but it's really ugly to have to do
1) For the IRQ number gathering.
a) Parse 1 to count CPMUs
b) Parse 2 to get the register maps - grab the irq numbers and unmap them again
2) For the CPMU registration
a) Parse 3 to count CPMUs (we could stash the number of CPMUS form 1a) but that's no advantage over stashing the max irq in current proposal. Both are putting state where it's not relevant or wanted just to make it available in a callback. This way is even worse because it's getting stashed as a side effect of a parse in a function doing something different.
b) Parse 4 to get the register maps and actually create the devices. Could have stashed this earlier as well, but same 'side effects' argument applies.Sure, can move to this however with appropriate comments on why we are playing these games because otherwise I suspect a future 'cleanup' would remove double, double pass.
To allow for an irq registration wrapper that turns a series of straight line calls into callbacks in an array. The straight line calls aren't exactly complex in the first place.
//find cpmu filling incxl_cpmu_reg_maps.
max_irq= -1
rc =cxl_mailbox_get_irq()
if (rc < 0)return rc;
max_irq=max(max_irq, rc);rc =
cxl_events_get_irq()
if (rc < 0)return rc;
max_irq=max(max_irq, rc);rc =
cxl_cpmus_get_irq(cxl_cpmu_reg_maps);
if (rc < 0)return rc;
max_irq=max(max_irq, rC);
Quoted 8 lines
...
if (irq > 0) {
pci_get... }//create all the devices...
Yes, this was sort of what I pictured if we go this way. It doesn't make my eyes sore.
Quoted 25 lines
Quoted 12 lines
Quoted 8 lines
But if we're going to go with a free-for-all approach, can we establish who goes for the initial
pci_alloc_irq_vectors()? I think perhaps mbox since it's the most straightforward and with least requirements, I'm also unsure of the status yet to merge events and pmu, but regardless they are still larger patchsets. If folks agree I can send a new mbox-only patch.I think there needs to be some mechanism for all of the sub-device-functions to report their max required vectors.
Quoted 8 lines
I don't think that the mbox code is necessarily the code which should need to know about all those other sub-device-thingys. But it could certainly take some 'max vectors' value that probe passed to it.
I'm still not sure how dropping this infrastructure makes Jonathan's code cleaner. I still think there will need to be 2 passes over the number of CPMU's.
Primarily that there is no need to stash anything about the CPMUs in the
cxl_device_state(option 1) or repeat all the counting and discovery logic twice (option 2).
Quoted 5 lines
I can live with it (it's what we have to do in pcie port for the equivalent) but the wrapped up version feels like a false optimization.
Saves a few lines of code and adds a bunch of complexity elsewhere that looks to me to outweigh that saving.
Yeah it's hard to justify the extra complexity here when the alternative isn't even that bad.
Thanks,
Davidlohr
Re: [PATCH 1/2] cxl/pci: Add generic MSI-X/MSI irq support
Quoted 71 lines · Davidlohr Bueso
Quoted 69 lines · Jonathan Cameron
Quoted 15 lines · On Thu, 20 Oct 2022 21:18:58 -0700 Ira Weiny <[email protected]> wrote:
Quoted 14 lines · Davidlohr Bueso
Quoted 11 lines · Jonathan Cameron
Reality is that it is cleaner to more or less ignore the infrastructure proposed in this patch.
1. Query how many CPMU devices there are. Whilst there stash the maximim cpmu vector number in the cxlds.
2. Run a stub in this infrastructure that doesmax(irq, cxlds->irq_num);
3. Carry on as before.Thus destroying the point of this infrastructure for that usecase at least and leaving an extra bit of state in the
cxl_dev_statethat is just to squirt a value into the callback...Quoted 7 lines
Quoted 7 lines
If it doesn't fit, then it doesn't fit.
However, while I was expecting pass one to be in the callback, I wasn't expecting that both pass 1 and 2 shared the
cpmu_regs_array. If the array could be reconstructed during pass 2, then it would fit a bit better; albeit the extra allocation, cycles etc., but this is probing phase, so overhead isn't that important (andcpmu_countisn't big enough to matter).I thought about that approach, but it's really ugly to have to do
1) For the IRQ number gathering.
a) Parse 1 to count CPMUs
b) Parse 2 to get the register maps - grab the irq numbers and unmap them again
2) For the CPMU registration
a) Parse 3 to count CPMUs (we could stash the number of CPMUS form 1a) but that's no advantage over stashing the max irq in current proposal. Both are putting state where it's not relevant or wanted just to make it available in a callback. This way is even worse because it's getting stashed as a side effect of a parse in a function doing something different.
b) Parse 4 to get the register maps and actually create the devices. Could have stashed this earlier as well, but same 'side effects' argument applies.Sure, can move to this however with appropriate comments on why we are playing these games because otherwise I suspect a future 'cleanup' would remove double, double pass.
To allow for an irq registration wrapper that turns a series of straight line calls into callbacks in an array. The straight line calls aren't exactly complex in the first place.
//find cpmu filling incxl_cpmu_reg_maps.
max_irq= -1
rc =cxl_mailbox_get_irq()
if (rc < 0)return rc;
max_irq=max(max_irq, rc);rc =
cxl_events_get_irq()
if (rc < 0)return rc;
max_irq=max(max_irq, rc);rc =
cxl_cpmus_get_irq(cxl_cpmu_reg_maps);
if (rc < 0)return rc;
max_irq=max(max_irq, rC);...
if (irq > 0) {
Quoted 7 lines
Quoted 4 lines
pci_get... }//create all the devices...
Yes, this was sort of what I pictured if we go this way. It doesn't make my eyes sore.
Ok
Quoted 17 lines
Quoted 17 lines
Quoted 17 lines
Quoted 9 lines
But if we're going to go with a free-for-all approach, can we establish who goes for the initial
pci_alloc_irq_vectors()? I think perhaps mbox since it's the most straightforward and with least requirements, I'm also unsure of the status yet to merge events and pmu, but regardless they are still larger patchsets. If folks agree I can send a new mbox-only patch.I think there needs to be some mechanism for all of the sub-device-functions to report their max required vectors.
I don't think that the mbox code is necessarily the code which should need to know about all those other sub-device-thingys. But it could certainly take some 'max vectors' value that probe passed to it.
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
I'm still not sure how dropping this infrastructure makes Jonathan's code cleaner. I still think there will need to be 2 passes over the number of CPMU's.
Primarily that there is no need to stash anything about the CPMUs in the
cxl_device_state(option 1) or repeat all the counting and discovery logic twice (option 2).
I see what you are driving at now. But I don't think having irq information in cxlds is a layering violation.
Quoted 9 lines
Quoted 6 lines
I can live with it (it's what we have to do in pcie port for the equivalent) but the wrapped up version feels like a false optimization.
Saves a few lines of code and adds a bunch of complexity elsewhere that looks to me to outweigh that saving.
Yeah it's hard to justify the extra complexity here when the alternative isn't even that bad.
Lets just do this. I don't think it matters much either and I need something to land before the event stuff.
Ira
Quoted 3 lines
Thanks,
Davidlohr
Re: [PATCH 1/2] cxl/pci: Add generic MSI-X/MSI irq support
In reply to Jonathan CameronQuoted 34 lines · Jonathan Cameron
Quoted 16 lines · On Tue, 18 Oct 2022 10:36:19 +0100 Jonathan Cameron…
Quoted 4 lines · On Mon, 17 Oct 2022 20:00:09 -0700 Davidlohr Bueso <[email protected]> wrote:
Introduce a generic irq table for CXL components/features that can have standard irq support - DOE requires dynamic vector sizing and is not considered here. For now the table is empty.
Quoted 8 lines
Create an infrastructure to query the max vectors required for the CXL device. Upon successful allocation, users can plug in their respective isr at any point thereafter, which is supported by a new
cxlds->has_irqflag, for example, if the irq setup is not done in the PCI driver, such as the case of the CXL-PMU.Reviewed-by Dave Jiang <[email protected]>Signed-off-by Davidlohr Bueso <[email protected]>Quoted 6 lines
A few nitpicks inline.
With the comment one tidied up (other one optional)
Reviewed-by Jonathan Cameron <[email protected]>I'll rebase my cpmu code on top of this shortly.
Hi Davidlohr,
Doing the CPMU rebase has shown up that using this generic infrastructure ends up rather ugly.
Previously I had a local array to manage the required register maps that was then freed. Now I have to move that into the cxl device state just so I can get at it from the irq finding callback.
Quoted 6 lines
So I have an extra step to be able to use this generic framework.
1. Query how many CPMU devices there are. Stash that and register map info in cxlds. I could do this in the callback but that's really really horrible layering issue as most of what is done has nothing to do with finding the vector numbers.
FWIW I did this for the event stuff and did not find it so distasteful... :-/
However the information I am stashing in the cxlds is all interrupt information. So I think it is different from what I see in the CPMU stuff.
Quoted 6 lines
2. The callback below to find those numbers
3. Registration of the cpmu devices.Reality is that it is cleaner to more or less ignore the infrastructure proposed in this patch.
Quoted 8 lines
1. Query how many CPMU devices there are. Whilst there stash the maximim cpmu vector number in the cxlds.
2. Run a stub in this infrastructure that doesmax(irq, cxlds->irq_num);
3. Carry on as before.Thus destroying the point of this infrastructure for that usecase at least and leaving an extra bit of state in the
cxl_dev_statethat is just to squirt a value into the callback...
I'm not sure I follow? Do you mean this?
static int cxl_cpmu_get_max_msgnum(struct cxl_dev_state *cxlds)
{
return cxlds->cpmu_max_vector;
} Quoted 6 lines
So with that in mind I'm withdrawing the RB above. This looks to be an idea that with hindsight doesn't necessarily pan out. Long hand equivalent with the specific handling needed for each case is probably going to be neater than walking a table of much more restricted callbacks.
I'm not married to the idea of the array of callbacks but I'm not sure how this solves having to iterate on the CPMU devices twice?
Ira
Quoted 146 lines
Maybe there is a nice way to fit the CPMU
registration into this infrastructure, but I'm not immediately seeing it.One other note inline via a compiler warning.
Jonathan
Quoted 38 lines
Jonathan
Quoted 34 lines
--- drivers/cxl/cxlmem.h | 3 ++ drivers/cxl/pci.c | 72 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 75 insertions(+)drivers/cxl/cxlmem.h +3
-0@@ -211,6 +211,7 @@ struct cxl_endpoint_dvsec_info {* @info: Cached DVSEC information about the device. * @serial: PCIe Device Serial Number * @doe_mbs: PCI DOE mailbox array +* @has_irq: PCIe MSI-X/MSI support * @mbox_send: @dev specific transport for transmitting mailbox commands * * See section 8.2.9.5.2 Capacity Configuration and Label Storage for @@ -247,6 +248,8 @@ struct cxl_dev_state {struct xarray doe_mbs; +bool has_irq; +int (*mbox_send)(struct cxl_dev_state *cxlds,struct cxl_mbox_cmd *cmd); };drivers/cxl/pci.c +0
-0@@ -428,6 +428,73 @@ static void devm_cxl_pci_create_doe(struct cxl_dev_state *cxlds) } }Quoted 15 lines
Quoted 5 lines
drivers/cxl/pci.c +5
-0+/** +* struct cxl_irq_cap - CXL feature that is capable of receiving MSI-X/MSI irqs. +* +* @name: Name of the device/component generating this interrupt. +* @get_max_msgnum: Get the feature's largest interrupt message number. If the drivers/cxl/pci.c static void devm_cxl_pci_create_doe(struct cxl_dev_state *cxlds) · line 445
Quoted 10 lines
drivers/cxl/pci.c +10
-0+* feature does not have the Interrupt Supported bit set, then +* return -1. +*/ +struct cxl_irq_cap { +const char *name; +int (*get_max_msgnum)(struct cxl_dev_state *cxlds); +}; + +static const struct cxl_irq_cap cxl_irq_cap_table[] = { +NULL That's not valid, just make it empty instead.
Quoted 81 lines
drivers/cxl/pci.c static void devm_cxl_pci_create_doe(struct cxl_dev_state *cxlds) · line 456
Quoted 11 lines
drivers/cxl/pci.c +11
-0+}; + +static void cxl_pci_free_irq_vectors (void *data) +{ +pci_free_irq_vectors (data); +} + +/* +* Attempt to allocate the largest amount of necessary vectors. +* +* Returns 0 upon a successful allocation of *all* vectors, or a Technically not all vectors. If we wanted to do that we could just directly query that via
pci_msix_vec_count()etc that gets it from the MSIX capability. That's frowned upon because it's common to stick lots of extra vectors on the end for stuff that linux never cares about (debug etc, or optional features).All vectors up to the maximum one the code uses would be more accurate.
Quoted 45 lines
drivers/cxl/pci.c +41
-0+* negative value otherwise. +*/ +static int cxl_pci_alloc_irq_vectors (struct cxl_dev_state *cxlds) +{ +struct device *dev = cxlds->dev; +struct pci_dev *pdev =to_pci_dev (dev); +int rc, i, vectors =-1 ; + +for (i =0 ; i <ARRAY_SIZE (cxl_irq_cap_table); i++) { +int irq; + +if (!cxl_irq_cap_table[i].get_max_msgnum) +continue ; + + irq = cxl_irq_cap_table[i].get_max_msgnum (cxlds); + vectors =max_t (int, irq, vectors); + } + +/* +* Semantically lack of irq support is not an error, but we +* still fail to allocate, so return negative. +*/ +if (vectors ==-1 ) +return -1 ; + + vectors++; + rc =pci_alloc_irq_vectors (pdev, vectors, vectors, +PCI_IRQ_MSIX |PCI_IRQ_MSI ); +if (rc <0 ) +return rc; + +if (rc != vectors) { +dev_dbg (dev,"Not enough interrupts; use polling instead.\n" ); +/* some got allocated, clean them up */ +cxl_pci_free_irq_vectors (pdev); +return -ENOSPC ; + } + +return devm_add_action_or_reset (dev, cxl_pci_free_irq_vectors, pdev); +} +static int cxl_pci_probe (struct pci_dev *pdev,const struct pci_device_id *id) {struct cxl_register_map map;drivers/cxl/pci.c static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id) · line 568
Quoted 8 lines
drivers/cxl/pci.c +5
-0@@ -494,6 +561,11 @@ static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)if (rc)return rc; +if (!cxl_pci_alloc_irq_vectors (cxlds)) { + cxlds->has_irq = true; + }else + cxlds->has_irq = false; +cxlds->has_irq = !(cxl_pci_aloc_irq_vectors(cxlds) < 0);maybe...
Quoted 3 lines
drivers/cxl/pci.c +0
-0cxlmd =devm_cxl_add_memdev (cxlds);if (IS_ERR (cxlmd))return PTR_ERR (cxlmd);
Re: [PATCH 1/2] cxl/pci: Add generic MSI-X/MSI irq support
Quoted 42 lines · On Thu, 20 Oct 2022 21:14:29 -0700 Ira Weiny <[email protected]> wrote:
Quoted 34 lines · Jonathan Cameron
Quoted 16 lines · On Tue, 18 Oct 2022 10:36:19 +0100 Jonathan Cameron…
Quoted 4 lines · On Mon, 17 Oct 2022 20:00:09 -0700 Davidlohr Bueso <[email protected]> wrote:
Introduce a generic irq table for CXL components/features that can have standard irq support - DOE requires dynamic vector sizing and is not considered here. For now the table is empty.
Quoted 8 lines
Create an infrastructure to query the max vectors required for the CXL device. Upon successful allocation, users can plug in their respective isr at any point thereafter, which is supported by a new
cxlds->has_irqflag, for example, if the irq setup is not done in the PCI driver, such as the case of the CXL-PMU.Reviewed-by Dave Jiang <[email protected]>Signed-off-by Davidlohr Bueso <[email protected]>Quoted 6 lines
A few nitpicks inline.
With the comment one tidied up (other one optional)
Reviewed-by Jonathan Cameron <[email protected]>I'll rebase my cpmu code on top of this shortly.
Hi Davidlohr,
Doing the CPMU rebase has shown up that using this generic infrastructure ends up rather ugly.
Previously I had a local array to manage the required register maps that was then freed. Now I have to move that into the cxl device state just so I can get at it from the irq finding callback.
Quoted 6 lines
So I have an extra step to be able to use this generic framework.
1. Query how many CPMU devices there are. Stash that and register map info in cxlds. I could do this in the callback but that's really really horrible layering issue as most of what is done has nothing to do with finding the vector numbers.
Quoted 4 lines
FWIW I did this for the event stuff and did not find it so distasteful... :-/
However the information I am stashing in the cxlds is all interrupt information. So I think it is different from what I see in the CPMU stuff.
Right now I'm just stashing the max interrupt number to squirt into a callback a few lines later. That feels like a hack to get around parsing the structures 4 times. If it's an acceptable hack then fair enough.
Quoted 16 lines
Quoted 7 lines
2. The callback below to find those numbers
3. Registration of the cpmu devices.Reality is that it is cleaner to more or less ignore the infrastructure proposed in this patch.
Quoted 8 lines
1. Query how many CPMU devices there are. Whilst there stash the maximim cpmu vector number in the cxlds.
2. Run a stub in this infrastructure that doesmax(irq, cxlds->irq_num);
3. Carry on as before.Thus destroying the point of this infrastructure for that usecase at least and leaving an extra bit of state in the
cxl_dev_statethat is just to squirt a value into the callback...
Quoted 6 lines
I'm not sure I follow? Do you mean this?
static int
cxl_cpmu_get_max_msgnum(struct cxl_dev_state *cxlds){ return cxlds->cpmu_max_vector; }
Yup. That state is no relevance to the cxl_dev_state outside of this tiny block of code. Hence I really don't like putting it in there.
Quoted 10 lines
Quoted 7 lines
So with that in mind I'm withdrawing the RB above. This looks to be an idea that with hindsight doesn't necessarily pan out. Long hand equivalent with the specific handling needed for each case is probably going to be neater than walking a table of much more restricted callbacks.
I'm not married to the idea of the array of callbacks but I'm not sure how this solves having to iterate on the CPMU devices twice?
Laid that out in the other branch of the thread but basically either
1) We stash irrelevant information in cxl_dev_state just to get it into the callback It's not used for anything else and this makes a fiddly and non obvious tie up between different registration steps that appear somewhat independent.
2) We do the whole double parse twice (so 4 times in total) which is the right option to keep the layering if using this array of callbacks approach, but really ugly. If we flatten it to straight line code there is no implication of layering and the state being parsed on is right there in a local variable.
I can live with it either way, but it's definitely not as pretty as it looks for the mailbox case.
Jonathan
Quoted 150 lines
Ira
Quoted 146 lines
Maybe there is a nice way to fit the CPMU
registration into this infrastructure, but I'm not immediately seeing it.One other note inline via a compiler warning.
Jonathan
Quoted 38 lines
Jonathan
Quoted 34 lines
--- drivers/cxl/cxlmem.h | 3 ++ drivers/cxl/pci.c | 72 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 75 insertions(+)drivers/cxl/cxlmem.h +3
-0@@ -211,6 +211,7 @@ struct cxl_endpoint_dvsec_info {* @info: Cached DVSEC information about the device. * @serial: PCIe Device Serial Number * @doe_mbs: PCI DOE mailbox array +* @has_irq: PCIe MSI-X/MSI support * @mbox_send: @dev specific transport for transmitting mailbox commands * * See section 8.2.9.5.2 Capacity Configuration and Label Storage for @@ -247,6 +248,8 @@ struct cxl_dev_state {struct xarray doe_mbs; +bool has_irq; +int (*mbox_send)(struct cxl_dev_state *cxlds,struct cxl_mbox_cmd *cmd); };drivers/cxl/pci.c +0
-0@@ -428,6 +428,73 @@ static void devm_cxl_pci_create_doe(struct cxl_dev_state *cxlds) } }Quoted 15 lines
Quoted 5 lines
drivers/cxl/pci.c +5
-0+/** +* struct cxl_irq_cap - CXL feature that is capable of receiving MSI-X/MSI irqs. +* +* @name: Name of the device/component generating this interrupt. +* @get_max_msgnum: Get the feature's largest interrupt message number. If the drivers/cxl/pci.c static void devm_cxl_pci_create_doe(struct cxl_dev_state *cxlds) · line 445
Quoted 10 lines
drivers/cxl/pci.c +10
-0+* feature does not have the Interrupt Supported bit set, then +* return -1. +*/ +struct cxl_irq_cap { +const char *name; +int (*get_max_msgnum)(struct cxl_dev_state *cxlds); +}; + +static const struct cxl_irq_cap cxl_irq_cap_table[] = { +NULL That's not valid, just make it empty instead.
Quoted 81 lines
drivers/cxl/pci.c static void devm_cxl_pci_create_doe(struct cxl_dev_state *cxlds) · line 456
Quoted 11 lines
drivers/cxl/pci.c +11
-0+}; + +static void cxl_pci_free_irq_vectors (void *data) +{ +pci_free_irq_vectors (data); +} + +/* +* Attempt to allocate the largest amount of necessary vectors. +* +* Returns 0 upon a successful allocation of *all* vectors, or a Technically not all vectors. If we wanted to do that we could just directly query that via
pci_msix_vec_count()etc that gets it from the MSIX capability. That's frowned upon because it's common to stick lots of extra vectors on the end for stuff that linux never cares about (debug etc, or optional features).All vectors up to the maximum one the code uses would be more accurate.
Quoted 45 lines
drivers/cxl/pci.c +41
-0+* negative value otherwise. +*/ +static int cxl_pci_alloc_irq_vectors (struct cxl_dev_state *cxlds) +{ +struct device *dev = cxlds->dev; +struct pci_dev *pdev =to_pci_dev (dev); +int rc, i, vectors =-1 ; + +for (i =0 ; i <ARRAY_SIZE (cxl_irq_cap_table); i++) { +int irq; + +if (!cxl_irq_cap_table[i].get_max_msgnum) +continue ; + + irq = cxl_irq_cap_table[i].get_max_msgnum (cxlds); + vectors =max_t (int, irq, vectors); + } + +/* +* Semantically lack of irq support is not an error, but we +* still fail to allocate, so return negative. +*/ +if (vectors ==-1 ) +return -1 ; + + vectors++; + rc =pci_alloc_irq_vectors (pdev, vectors, vectors, +PCI_IRQ_MSIX |PCI_IRQ_MSI ); +if (rc <0 ) +return rc; + +if (rc != vectors) { +dev_dbg (dev,"Not enough interrupts; use polling instead.\n" ); +/* some got allocated, clean them up */ +cxl_pci_free_irq_vectors (pdev); +return -ENOSPC ; + } + +return devm_add_action_or_reset (dev, cxl_pci_free_irq_vectors, pdev); +} +static int cxl_pci_probe (struct pci_dev *pdev,const struct pci_device_id *id) {struct cxl_register_map map;drivers/cxl/pci.c static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id) · line 568
Quoted 8 lines
drivers/cxl/pci.c +5
-0@@ -494,6 +561,11 @@ static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)if (rc)return rc; +if (!cxl_pci_alloc_irq_vectors (cxlds)) { + cxlds->has_irq = true; + }else + cxlds->has_irq = false; +cxlds->has_irq = !(cxl_pci_aloc_irq_vectors(cxlds) < 0);maybe...
Quoted 3 lines
drivers/cxl/pci.c +0
-0cxlmd =devm_cxl_add_memdev (cxlds);if (IS_ERR (cxlmd))return PTR_ERR (cxlmd);
Davidlohr Bueso
maintainerRe: [PATCH 1/2] cxl/pci: Add generic MSI-X/MSI irq support
Quoted 27 lines · Jonathan Cameron
Quoted 4 lines
FWIW I did this for the event stuff and did not find it so distasteful... :-/
However the information I am stashing in the cxlds is all interrupt information. So I think it is different from what I see in the CPMU stuff.
Right now I'm just stashing the max interrupt number to squirt into a callback a few lines later. That feels like a hack to get around parsing the structures 4 times. If it's an acceptable hack then fair enough.
Quoted 18 lines
Quoted 7 lines
2. The callback below to find those numbers
3. Registration of the cpmu devices.Reality is that it is cleaner to more or less ignore the infrastructure proposed in this patch.
Quoted 8 lines
1. Query how many CPMU devices there are. Whilst there stash the maximim cpmu vector number in the cxlds.
2. Run a stub in this infrastructure that doesmax(irq, cxlds->irq_num);
3. Carry on as before.Thus destroying the point of this infrastructure for that usecase at least and leaving an extra bit of state in the
cxl_dev_statethat is just to squirt a value into the callback...I'm not sure I follow? Do you mean this?
Quoted 7 lines
Quoted 4 lines
static int
cxl_cpmu_get_max_msgnum(struct cxl_dev_state *cxlds){ return cxlds->cpmu_max_vector; }Yup. That state is no relevance to the
cxl_dev_stateoutside of this tiny block of code. Hence I really don't like putting it in there.
Oh absolutely, this is ugly as sin. And if there is anything even worth stashing the max would only be mbox, as Ira suggested earlier in v1, iirc. So no, we should not be doing this sort of thing. And if pass one were done in the callback the need for this would disappear.
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
So with that in mind I'm withdrawing the RB above. This looks to be an idea that with hindsight doesn't necessarily pan out. Long hand equivalent with the specific handling needed for each case is probably going to be neater than walking a table of much more restricted callbacks.
Quoted 7 lines
Quoted 2 lines
I'm not married to the idea of the array of callbacks but I'm not sure how this solves having to iterate on the CPMU devices twice?
Laid that out in the other branch of the thread but basically either
1) We stash irrelevant information incxl_dev_statejust to get it into the callback It's not used for anything else and this makes a fiddly and non obvious tie up between different registration steps that appear somewhat independent.
Yeah anything _but_ this.
Quoted 5 lines
2) We do the whole double parse twice (so 4 times in total) which is the right option to keep the layering if using this array of callbacks approach, but really ugly. If we flatten it to straight line code there is no implication of layering and the state being parsed on is right there in a local variable.
If we are keeping this patch, then as mentioned before, I would prefer this. imo this is better than both 1 above and the open-coding approach.
Quoted 2 lines
I can live with it either way, but it's definitely not as pretty as it looks for the mailbox case.
Agreed.
Thanks,
Davidlohr
Re: [PATCH 1/2] cxl/pci: Add generic MSI-X/MSI irq support
In reply to Jonathan CameronQuoted 72 lines · Jonathan Cameron
Quoted 42 lines · On Thu, 20 Oct 2022 21:14:29 -0700 Ira Weiny <[email protected]> wrote:
Quoted 34 lines · Jonathan Cameron
Quoted 16 lines · On Tue, 18 Oct 2022 10:36:19 +0100 Jonathan Cameron…
Quoted 4 lines · On Mon, 17 Oct 2022 20:00:09 -0700 Davidlohr Bueso <[email protected]> wrote:
Introduce a generic irq table for CXL components/features that can have standard irq support - DOE requires dynamic vector sizing and is not considered here. For now the table is empty.
Quoted 8 lines
Create an infrastructure to query the max vectors required for the CXL device. Upon successful allocation, users can plug in their respective isr at any point thereafter, which is supported by a new
cxlds->has_irqflag, for example, if the irq setup is not done in the PCI driver, such as the case of the CXL-PMU.Reviewed-by Dave Jiang <[email protected]>Signed-off-by Davidlohr Bueso <[email protected]>Quoted 6 lines
A few nitpicks inline.
With the comment one tidied up (other one optional)
Reviewed-by Jonathan Cameron <[email protected]>I'll rebase my cpmu code on top of this shortly.
Hi Davidlohr,
Doing the CPMU rebase has shown up that using this generic infrastructure ends up rather ugly.
Previously I had a local array to manage the required register maps that was then freed. Now I have to move that into the cxl device state just so I can get at it from the irq finding callback.
Quoted 6 lines
So I have an extra step to be able to use this generic framework.
1. Query how many CPMU devices there are. Stash that and register map info in cxlds. I could do this in the callback but that's really really horrible layering issue as most of what is done has nothing to do with finding the vector numbers.
Quoted 4 lines
FWIW I did this for the event stuff and did not find it so distasteful... :-/
However the information I am stashing in the cxlds is all interrupt information. So I think it is different from what I see in the CPMU stuff.
Right now I'm just stashing the max interrupt number to squirt into a callback a few lines later. That feels like a hack to get around parsing the structures 4 times. If it's an acceptable hack then fair enough.
Quoted 18 lines
Quoted 7 lines
2. The callback below to find those numbers
3. Registration of the cpmu devices.Reality is that it is cleaner to more or less ignore the infrastructure proposed in this patch.
Quoted 8 lines
1. Query how many CPMU devices there are. Whilst there stash the maximim cpmu vector number in the cxlds.
2. Run a stub in this infrastructure that doesmax(irq, cxlds->irq_num);
3. Carry on as before.Thus destroying the point of this infrastructure for that usecase at least and leaving an extra bit of state in the
cxl_dev_statethat is just to squirt a value into the callback...I'm not sure I follow? Do you mean this?
Quoted 7 lines
Quoted 4 lines
static int
cxl_cpmu_get_max_msgnum(struct cxl_dev_state *cxlds){ return cxlds->cpmu_max_vector; }Yup. That state is no relevance to the
cxl_dev_stateoutside of this tiny block of code. Hence I really don't like putting it in there.
Yeah, I tend to agree. cxl_dev_state is the catch-all of last resort, but if there is a more appropriate / less-abstract object to carry a given property it should.
Re: [PATCH 1/2] cxl/pci: Add generic MSI-X/MSI irq support
In reply to Davidlohr BuesoQuoted 13 lines · On Mon, 17 Oct 2022 20:00:09 -0700 Davidlohr Bueso <[email protected]> wrote:
Introduce a generic irq table for CXL components/features that can have standard irq support - DOE requires dynamic vector sizing and is not considered here. For now the table is empty.
Create an infrastructure to query the max vectors required for the CXL device. Upon successful allocation, users can plug in their respective isr at any point thereafter, which is supported by a new
cxlds->has_irqflag, for example, if the irq setup is not done in the PCI driver, such as the case of the CXL-PMU.Reviewed-by Dave Jiang <[email protected]>Signed-off-by Davidlohr Bueso <[email protected]>
drivers/cxl/pci.c static void devm_cxl_pci_create_doe(struct cxl_dev_state *cxlds) · line 488
Quoted 7 lines
drivers/cxl/pci.c +7
-0+ vectors++; + rc =pci_alloc_irq_vectors (pdev, vectors, vectors, +PCI_IRQ_MSIX |PCI_IRQ_MSI ); +if (rc <0 ) +return rc; + +if (rc != vectors) {
Just catching up with David Jiang's review of the CPMU driver. He correctly points out that we won't hit this condition because we provide minvec to the pci_alloc_irq_vectors() call:
Quoted 16 lines
I don't think you'll hit here since you passed in vectors for min and max. You'll get
-ENOSPCand return from the earlier check.https://elixir.bootlin.com/linux/v6.0-rc3/source/drivers/pci/msi/msi.c#L1005
DJ
drivers/cxl/pci.c +8
-0+dev_dbg (dev,"Not enough interrupts; use polling instead.\n" ); +/* some got allocated, clean them up */ +cxl_pci_free_irq_vectors (pdev); +return -ENOSPC ; + } + +return devm_add_action_or_reset (dev, cxl_pci_free_irq_vectors, pdev); +}
RE: [PATCH 1/2] cxl/pci: Add generic MSI-X/MSI irq support
In reply to Davidlohr BuesoQuoted 47 lines · Davidlohr Bueso
Introduce a generic irq table for CXL components/features that can have standard irq support - DOE requires dynamic vector sizing and is not considered here. For now the table is empty.
Create an infrastructure to query the max vectors required for the CXL device. Upon successful allocation, users can plug in their respective isr at any point thereafter, which is supported by a new
cxlds->has_irqflag, for example, if the irq setup is not done in the PCI driver, such as the case of the CXL-PMU.Reviewed-by Dave Jiang <[email protected]>Signed-off-by Davidlohr Bueso <[email protected]>--- drivers/cxl/cxlmem.h | 3 ++ drivers/cxl/pci.c | 72 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 75 insertions(+)drivers/cxl/cxlmem.h +3
-0@@ -211,6 +211,7 @@ struct cxl_endpoint_dvsec_info {* @info: Cached DVSEC information about the device. * @serial: PCIe Device Serial Number * @doe_mbs: PCI DOE mailbox array +* @has_irq: PCIe MSI-X/MSI support * @mbox_send: @dev specific transport for transmitting mailbox commands * * See section 8.2.9.5.2 Capacity Configuration and Label Storage for @@ -247,6 +248,8 @@ struct cxl_dev_state {struct xarray doe_mbs; +bool has_irq; +int (*mbox_send)(struct cxl_dev_state *cxlds,struct cxl_mbox_cmd *cmd); };drivers/cxl/pci.c +1
-0@@ -428,6 +428,73 @@ static void devm_cxl_pci_create_doe(struct cxl_dev_state *cxlds) } } +/**
drivers/cxl/pci.c static void devm_cxl_pci_create_doe(struct cxl_dev_state *cxlds) · line 441
Quoted 10 lines
drivers/cxl/pci.c +10
-0+* struct cxl_irq_cap - CXL feature that is capable of receiving MSI-X/MSI irqs. +* +* @name: Name of the device/component generating this interrupt. +* @get_max_msgnum: Get the feature's largest interrupt message number. If the +* feature does not have the Interrupt Supported bit set, then +* return -1. +*/ +struct cxl_irq_cap { +const char *name; +int (*get_max_msgnum)(struct cxl_dev_state *cxlds);
Why is this a callback, why not just have the features populate their irq numbers?
Quoted 22 lines
drivers/cxl/pci.c +22
-0+}; + +static const struct cxl_irq_cap cxl_irq_cap_table[] = { +NULL +}; + +static void cxl_pci_free_irq_vectors (void *data) +{ +pci_free_irq_vectors (data); +} + +/* +* Attempt to allocate the largest amount of necessary vectors. +* +* Returns 0 upon a successful allocation of *all* vectors, or a +* negative value otherwise. +*/ +static int cxl_pci_alloc_irq_vectors (struct cxl_dev_state *cxlds) +{ +struct device *dev = cxlds->dev; +struct pci_dev *pdev =to_pci_dev (dev); +int rc, i, vectors =-1 ;
drivers/cxl/pci.c static void devm_cxl_pci_create_doe(struct cxl_dev_state *cxlds) · line 473
Quoted 10 lines
drivers/cxl/pci.c +10
-0+ +for (i =0 ; i <ARRAY_SIZE (cxl_irq_cap_table); i++) { +int irq; + +if (!cxl_irq_cap_table[i].get_max_msgnum) +continue ; + + irq = cxl_irq_cap_table[i].get_max_msgnum (cxlds); + vectors =max_t (int, irq, vectors); + }
Forgive me if I have missed something, I only look at interrupt enable code once every few years, and the APIs are always a bit different, but is this not too early to read the message number? The number is not stable until either MSI or MSI-X has been selected below at pci_alloc_irq_vectors() time?
Quoted 42 lines
drivers/cxl/pci.c +29
-0+ +/* +* Semantically lack of irq support is not an error, but we +* still fail to allocate, so return negative. +*/ +if (vectors ==-1 ) +return -1 ; + + vectors++; + rc =pci_alloc_irq_vectors (pdev, vectors, vectors, +PCI_IRQ_MSIX |PCI_IRQ_MSI ); +if (rc <0 ) +return rc; + +if (rc != vectors) { +dev_dbg (dev,"Not enough interrupts; use polling instead.\n" ); +/* some got allocated, clean them up */ +cxl_pci_free_irq_vectors (pdev); +return -ENOSPC ; + } + +return devm_add_action_or_reset (dev, cxl_pci_free_irq_vectors, pdev); +} +static int cxl_pci_probe (struct pci_dev *pdev,const struct pci_device_id *id) {struct cxl_register_map map;@@ -494,6 +561,11 @@ static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)if (rc)return rc; +if (!cxl_pci_alloc_irq_vectors (cxlds)) { + cxlds->has_irq = true; + }else + cxlds->has_irq = false; + cxlmd =devm_cxl_add_memdev (cxlds);if (IS_ERR (cxlmd))return PTR_ERR (cxlmd);Signature · 1 line
2.38.0
Re: [PATCH 1/2] cxl/pci: Add generic MSI-X/MSI irq support
Quoted 47 lines · Dan Williams
Quoted 46 lines · Davidlohr Bueso
Introduce a generic irq table for CXL components/features that can have standard irq support - DOE requires dynamic vector sizing and is not considered here. For now the table is empty.
Create an infrastructure to query the max vectors required for the CXL device. Upon successful allocation, users can plug in their respective isr at any point thereafter, which is supported by a new
cxlds->has_irqflag, for example, if the irq setup is not done in the PCI driver, such as the case of the CXL-PMU.Reviewed-by Dave Jiang <[email protected]>Signed-off-by Davidlohr Bueso <[email protected]>--- drivers/cxl/cxlmem.h | 3 ++ drivers/cxl/pci.c | 72 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 75 insertions(+)drivers/cxl/cxlmem.h +3
-0@@ -211,6 +211,7 @@ struct cxl_endpoint_dvsec_info {* @info: Cached DVSEC information about the device. * @serial: PCIe Device Serial Number * @doe_mbs: PCI DOE mailbox array +* @has_irq: PCIe MSI-X/MSI support * @mbox_send: @dev specific transport for transmitting mailbox commands * * See section 8.2.9.5.2 Capacity Configuration and Label Storage for @@ -247,6 +248,8 @@ struct cxl_dev_state {struct xarray doe_mbs; +bool has_irq; +int (*mbox_send)(struct cxl_dev_state *cxlds,struct cxl_mbox_cmd *cmd); };drivers/cxl/pci.c +0
-0@@ -428,6 +428,73 @@ static void devm_cxl_pci_create_doe(struct cxl_dev_state *cxlds) } }
Quoted 14 lines
drivers/cxl/pci.c static void devm_cxl_pci_create_doe(struct cxl_dev_state *cxlds) · line 441
Quoted 11 lines
drivers/cxl/pci.c +11
-0+/** +* struct cxl_irq_cap - CXL feature that is capable of receiving MSI-X/MSI irqs. +* +* @name: Name of the device/component generating this interrupt. +* @get_max_msgnum: Get the feature's largest interrupt message number. If the +* feature does not have the Interrupt Supported bit set, then +* return -1. +*/ +struct cxl_irq_cap { +const char *name; +int (*get_max_msgnum)(struct cxl_dev_state *cxlds);Why is this a callback, why not just have the features populate their irq numbers?
I think we have decided to forgo the callback but I'm not sure what you mean by 'populate their irq numbers'?
Quoted 31 lines
Quoted 31 lines
drivers/cxl/pci.c +30
-0+}; + +static const struct cxl_irq_cap cxl_irq_cap_table[] = { +NULL +}; + +static void cxl_pci_free_irq_vectors (void *data) +{ +pci_free_irq_vectors (data); +} + +/* +* Attempt to allocate the largest amount of necessary vectors. +* +* Returns 0 upon a successful allocation of *all* vectors, or a +* negative value otherwise. +*/ +static int cxl_pci_alloc_irq_vectors (struct cxl_dev_state *cxlds) +{ +struct device *dev = cxlds->dev; +struct pci_dev *pdev =to_pci_dev (dev); +int rc, i, vectors =-1 ; + +for (i =0 ; i <ARRAY_SIZE (cxl_irq_cap_table); i++) { +int irq; + +if (!cxl_irq_cap_table[i].get_max_msgnum) +continue ; + + irq = cxl_irq_cap_table[i].get_max_msgnum (cxlds);
Quoted 8 lines
drivers/cxl/pci.c static void devm_cxl_pci_create_doe(struct cxl_dev_state *cxlds) · line 473
Quoted 2 lines
drivers/cxl/pci.c +2
-0+ vectors =max_t (int, irq, vectors); + }Forgive me if I have missed something, I only look at interrupt enable code once every few years, and the APIs are always a bit different, but is this not too early to read the message number? The number is not stable until either MSI or MSI-X has been selected below at
pci_alloc_irq_vectors()time?
Well I keep getting wrapped around the axle on this one too.
This all started back when Jonathan originally attempted to allocate the maximum number of vectors a device _could_ allocate. But it was recommended that we determine the max number first then allocate that number.
This seems like a chicken and egg issue. How is the number not stable before calling pci_alloc_irq_vectors() when you need the max msg number in that call?
The Event code seems pretty simple because we use a mailbox command to configure which logs to use irq's and the device reports the message number for each. Thus we can determine the max message number Events need without enabling anything. But your comment about them not being stable now has me more worried... :-(
Confused,
Ira
Quoted 44 lines
Quoted 42 lines
drivers/cxl/pci.c +29
-0+ +/* +* Semantically lack of irq support is not an error, but we +* still fail to allocate, so return negative. +*/ +if (vectors ==-1 ) +return -1 ; + + vectors++; + rc =pci_alloc_irq_vectors (pdev, vectors, vectors, +PCI_IRQ_MSIX |PCI_IRQ_MSI ); +if (rc <0 ) +return rc; + +if (rc != vectors) { +dev_dbg (dev,"Not enough interrupts; use polling instead.\n" ); +/* some got allocated, clean them up */ +cxl_pci_free_irq_vectors (pdev); +return -ENOSPC ; + } + +return devm_add_action_or_reset (dev, cxl_pci_free_irq_vectors, pdev); +} +static int cxl_pci_probe (struct pci_dev *pdev,const struct pci_device_id *id) {struct cxl_register_map map;@@ -494,6 +561,11 @@ static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)if (rc)return rc; +if (!cxl_pci_alloc_irq_vectors (cxlds)) { + cxlds->has_irq = true; + }else + cxlds->has_irq = false; + cxlmd =devm_cxl_add_memdev (cxlds);if (IS_ERR (cxlmd))return PTR_ERR (cxlmd);Signature · 1 line
2.38.0
Re: [PATCH 1/2] cxl/pci: Add generic MSI-X/MSI irq support
Quoted 106 lines · Ira Weiny
Quoted 47 lines · Dan Williams
Quoted 46 lines · Davidlohr Bueso
Introduce a generic irq table for CXL components/features that can have standard irq support - DOE requires dynamic vector sizing and is not considered here. For now the table is empty.
Create an infrastructure to query the max vectors required for the CXL device. Upon successful allocation, users can plug in their respective isr at any point thereafter, which is supported by a new
cxlds->has_irqflag, for example, if the irq setup is not done in the PCI driver, such as the case of the CXL-PMU.Reviewed-by Dave Jiang <[email protected]>Signed-off-by Davidlohr Bueso <[email protected]>--- drivers/cxl/cxlmem.h | 3 ++ drivers/cxl/pci.c | 72 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 75 insertions(+)drivers/cxl/cxlmem.h +3
-0@@ -211,6 +211,7 @@ struct cxl_endpoint_dvsec_info {* @info: Cached DVSEC information about the device. * @serial: PCIe Device Serial Number * @doe_mbs: PCI DOE mailbox array +* @has_irq: PCIe MSI-X/MSI support * @mbox_send: @dev specific transport for transmitting mailbox commands * * See section 8.2.9.5.2 Capacity Configuration and Label Storage for @@ -247,6 +248,8 @@ struct cxl_dev_state {struct xarray doe_mbs; +bool has_irq; +int (*mbox_send)(struct cxl_dev_state *cxlds,struct cxl_mbox_cmd *cmd); };drivers/cxl/pci.c +0
-0@@ -428,6 +428,73 @@ static void devm_cxl_pci_create_doe(struct cxl_dev_state *cxlds) } }Quoted 14 lines
drivers/cxl/pci.c static void devm_cxl_pci_create_doe(struct cxl_dev_state *cxlds) · line 441
Quoted 11 lines
drivers/cxl/pci.c +11
-0+/** +* struct cxl_irq_cap - CXL feature that is capable of receiving MSI-X/MSI irqs. +* +* @name: Name of the device/component generating this interrupt. +* @get_max_msgnum: Get the feature's largest interrupt message number. If the +* feature does not have the Interrupt Supported bit set, then +* return -1. +*/ +struct cxl_irq_cap { +const char *name; +int (*get_max_msgnum)(struct cxl_dev_state *cxlds);Why is this a callback, why not just have the features populate their irq numbers?
I think we have decided to forgo the callback but I'm not sure what you mean by 'populate their irq numbers'?
Quoted 31 lines
Quoted 31 lines
drivers/cxl/pci.c +30
-0+}; + +static const struct cxl_irq_cap cxl_irq_cap_table[] = { +NULL +}; + +static void cxl_pci_free_irq_vectors (void *data) +{ +pci_free_irq_vectors (data); +} + +/* +* Attempt to allocate the largest amount of necessary vectors. +* +* Returns 0 upon a successful allocation of *all* vectors, or a +* negative value otherwise. +*/ +static int cxl_pci_alloc_irq_vectors (struct cxl_dev_state *cxlds) +{ +struct device *dev = cxlds->dev; +struct pci_dev *pdev =to_pci_dev (dev); +int rc, i, vectors =-1 ; + +for (i =0 ; i <ARRAY_SIZE (cxl_irq_cap_table); i++) { +int irq; + +if (!cxl_irq_cap_table[i].get_max_msgnum) +continue ; + + irq = cxl_irq_cap_table[i].get_max_msgnum (cxlds);Quoted 8 lines
drivers/cxl/pci.c static void devm_cxl_pci_create_doe(struct cxl_dev_state *cxlds) · line 473
Quoted 2 lines
drivers/cxl/pci.c +2
-0+ vectors =max_t (int, irq, vectors); + }Forgive me if I have missed something, I only look at interrupt enable code once every few years, and the APIs are always a bit different, but is this not too early to read the message number? The number is not stable until either MSI or MSI-X has been selected below at
pci_alloc_irq_vectors()time?
Quoted 8 lines
Well I keep getting wrapped around the axle on this one too.
This all started back when Jonathan originally attempted to allocate the maximum number of vectors a device _could_ allocate. But it was recommended that we determine the max number first then allocate that number.
This seems like a chicken and egg issue. How is the number not stable before calling
pci_alloc_irq_vectors()when you need the max msg number in that call?
Are we talking about the same thing? I am talking about the value in the "Interrupt Message Number" field. That depends on whether MSI or MSI-X gets enabled. The number of vectors the device can support is static.
Since CXL is such an a la carte spec I think this is situation to just specify a large number of amx vectors to pci_alloc_irq_vectors() and then find out after the fact if all of the interrupt generators that today's cxl_pci knows about in the device each got their own vector.
Re: [PATCH 1/2] cxl/pci: Add generic MSI-X/MSI irq support
Quoted 116 lines · On Sun, 23 Oct 2022 19:08:57 -0700 Dan Williams <[email protected]>…
Quoted 106 lines · Ira Weiny
Quoted 47 lines · Dan Williams
Quoted 46 lines · Davidlohr Bueso
Introduce a generic irq table for CXL components/features that can have standard irq support - DOE requires dynamic vector sizing and is not considered here. For now the table is empty.
Create an infrastructure to query the max vectors required for the CXL device. Upon successful allocation, users can plug in their respective isr at any point thereafter, which is supported by a new
cxlds->has_irqflag, for example, if the irq setup is not done in the PCI driver, such as the case of the CXL-PMU.Reviewed-by Dave Jiang <[email protected]>Signed-off-by Davidlohr Bueso <[email protected]>--- drivers/cxl/cxlmem.h | 3 ++ drivers/cxl/pci.c | 72 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 75 insertions(+)drivers/cxl/cxlmem.h +3
-0@@ -211,6 +211,7 @@ struct cxl_endpoint_dvsec_info {* @info: Cached DVSEC information about the device. * @serial: PCIe Device Serial Number * @doe_mbs: PCI DOE mailbox array +* @has_irq: PCIe MSI-X/MSI support * @mbox_send: @dev specific transport for transmitting mailbox commands * * See section 8.2.9.5.2 Capacity Configuration and Label Storage for @@ -247,6 +248,8 @@ struct cxl_dev_state {struct xarray doe_mbs; +bool has_irq; +int (*mbox_send)(struct cxl_dev_state *cxlds,struct cxl_mbox_cmd *cmd); };drivers/cxl/pci.c +0
-0@@ -428,6 +428,73 @@ static void devm_cxl_pci_create_doe(struct cxl_dev_state *cxlds) } }Quoted 14 lines
drivers/cxl/pci.c static void devm_cxl_pci_create_doe(struct cxl_dev_state *cxlds) · line 441
Quoted 11 lines
drivers/cxl/pci.c +11
-0+/** +* struct cxl_irq_cap - CXL feature that is capable of receiving MSI-X/MSI irqs. +* +* @name: Name of the device/component generating this interrupt. +* @get_max_msgnum: Get the feature's largest interrupt message number. If the +* feature does not have the Interrupt Supported bit set, then +* return -1. +*/ +struct cxl_irq_cap { +const char *name; +int (*get_max_msgnum)(struct cxl_dev_state *cxlds);Why is this a callback, why not just have the features populate their irq numbers?
I think we have decided to forgo the callback but I'm not sure what you mean by 'populate their irq numbers'?
Quoted 31 lines
Quoted 31 lines
drivers/cxl/pci.c +30
-0+}; + +static const struct cxl_irq_cap cxl_irq_cap_table[] = { +NULL +}; + +static void cxl_pci_free_irq_vectors (void *data) +{ +pci_free_irq_vectors (data); +} + +/* +* Attempt to allocate the largest amount of necessary vectors. +* +* Returns 0 upon a successful allocation of *all* vectors, or a +* negative value otherwise. +*/ +static int cxl_pci_alloc_irq_vectors (struct cxl_dev_state *cxlds) +{ +struct device *dev = cxlds->dev; +struct pci_dev *pdev =to_pci_dev (dev); +int rc, i, vectors =-1 ; + +for (i =0 ; i <ARRAY_SIZE (cxl_irq_cap_table); i++) { +int irq; + +if (!cxl_irq_cap_table[i].get_max_msgnum) +continue ; + + irq = cxl_irq_cap_table[i].get_max_msgnum (cxlds);Quoted 8 lines
drivers/cxl/pci.c static void devm_cxl_pci_create_doe(struct cxl_dev_state *cxlds) · line 473
Quoted 2 lines
drivers/cxl/pci.c +2
-0+ vectors =max_t (int, irq, vectors); + }Forgive me if I have missed something, I only look at interrupt enable code once every few years, and the APIs are always a bit different, but is this not too early to read the message number? The number is not stable until either MSI or MSI-X has been selected below at
pci_alloc_irq_vectors()time?Quoted 8 lines
Well I keep getting wrapped around the axle on this one too.
This all started back when Jonathan originally attempted to allocate the maximum number of vectors a device _could_ allocate. But it was recommended that we determine the max number first then allocate that number.
This seems like a chicken and egg issue. How is the number not stable before calling
pci_alloc_irq_vectors()when you need the max msg number in that call?
Quoted 8 lines
Are we talking about the same thing? I am talking about the value in the "Interrupt Message Number" field. That depends on whether MSI or MSI-X gets enabled. The number of vectors the device can support is static.
Since CXL is such an a la carte spec I think this is situation to just specify a large number of amx vectors to
pci_alloc_irq_vectors()and then find out after the fact if all of the interrupt generators that today'scxl_pciknows about in the device each got their own vector.
I'd misunderstood how this worked and not read the spec :( I wrongly thought portdrv did the query first and allocated the vectors after, but that's not the case. It first allocates max entries, then frees them all and then allocates the ones that we find present.
We should probably look to do something similar to that though I'm not sure even that code is always optimal.
https://elixir.bootlin.com/linux/v6.1-rc2/source/drivers/pci/pcie/portdrv_core.c#L101
In short that calls:
/* Allocate the maximum possible number of MSI/MSI-X vectors */ nr_entries = pci_alloc_irq_vectors(dev, 1, PCIE_PORT_MAX_MSI_ENTRIES,
PCI_IRQ_MSIX | PCI_IRQ_MSI);
/* See how many and which Interrupt Message Numbers we actually use */ nvec = pcie_message_numbers(dev, mask, &pme, &aer, &dpc);
if (nvec != nr_entries) {
pci_free_irq_vectors(dev);
nr_entries = pci_alloc_irq_vectors(dev, nvec, nvec,
PCI_IRQ_MSIX | PCI_IRQ_MSI);
}My worry here is that the implicit assumption is that the vectors won't move if we reduce the overall number of vectors we are asking for...
However, imagine the case that we have a feature the driver doesn't know about that was previously at a higher vector. After reducing the vectors allocated the hardware might decide that feature needs it's own vector whereas some others can be combined. Hence we'd end up with a less than ideal packing for the features we actually support.
Could do something iterative to solve this if it actually matters (increase number of vectors until the layout matches what we get with max possible vectors).
+CC linux-pci and Bjorn for their take on this. Maybe I'm over thinking things and in reality this never happens.
Jonathan
Re: [PATCH 1/2] cxl/pci: Add generic MSI-X/MSI irq support
[+cc Christoph, beginning of thread https://lore.kernel.org/r/[email protected]]
Quoted 164 lines · Jonathan Cameron
Quoted 118 lines · On Sun, 23 Oct 2022 19:08:57 -0700 Dan Williams <[email protected]>…
Quoted 108 lines · Ira Weiny
Quoted 47 lines · Dan Williams
Quoted 46 lines · Davidlohr Bueso
Introduce a generic irq table for CXL components/features that can have standard irq support - DOE requires dynamic vector sizing and is not considered here. For now the table is empty.
Create an infrastructure to query the max vectors required for the CXL device. Upon successful allocation, users can plug in their respective isr at any point thereafter, which is supported by a new
cxlds->has_irqflag, for example, if the irq setup is not done in the PCI driver, such as the case of the CXL-PMU.Reviewed-by Dave Jiang <[email protected]>Signed-off-by Davidlohr Bueso <[email protected]>--- drivers/cxl/cxlmem.h | 3 ++ drivers/cxl/pci.c | 72 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 75 insertions(+)drivers/cxl/cxlmem.h +3
-0@@ -211,6 +211,7 @@ struct cxl_endpoint_dvsec_info {* @info: Cached DVSEC information about the device. * @serial: PCIe Device Serial Number * @doe_mbs: PCI DOE mailbox array +* @has_irq: PCIe MSI-X/MSI support * @mbox_send: @dev specific transport for transmitting mailbox commands * * See section 8.2.9.5.2 Capacity Configuration and Label Storage for @@ -247,6 +248,8 @@ struct cxl_dev_state {struct xarray doe_mbs; +bool has_irq; +int (*mbox_send)(struct cxl_dev_state *cxlds,struct cxl_mbox_cmd *cmd); };drivers/cxl/pci.c +0
-0@@ -428,6 +428,73 @@ static void devm_cxl_pci_create_doe(struct cxl_dev_state *cxlds) } }Quoted 14 lines
drivers/cxl/pci.c static void devm_cxl_pci_create_doe(struct cxl_dev_state *cxlds) · line 441
Quoted 11 lines
drivers/cxl/pci.c +11
-0+/** +* struct cxl_irq_cap - CXL feature that is capable of receiving MSI-X/MSI irqs. +* +* @name: Name of the device/component generating this interrupt. +* @get_max_msgnum: Get the feature's largest interrupt message number. If the +* feature does not have the Interrupt Supported bit set, then +* return -1. +*/ +struct cxl_irq_cap { +const char *name; +int (*get_max_msgnum)(struct cxl_dev_state *cxlds);Why is this a callback, why not just have the features populate their irq numbers?
I think we have decided to forgo the callback but I'm not sure what you mean by 'populate their irq numbers'?
Quoted 31 lines
Quoted 31 lines
drivers/cxl/pci.c +30
-0+}; + +static const struct cxl_irq_cap cxl_irq_cap_table[] = { +NULL +}; + +static void cxl_pci_free_irq_vectors (void *data) +{ +pci_free_irq_vectors (data); +} + +/* +* Attempt to allocate the largest amount of necessary vectors. +* +* Returns 0 upon a successful allocation of *all* vectors, or a +* negative value otherwise. +*/ +static int cxl_pci_alloc_irq_vectors (struct cxl_dev_state *cxlds) +{ +struct device *dev = cxlds->dev; +struct pci_dev *pdev =to_pci_dev (dev); +int rc, i, vectors =-1 ; + +for (i =0 ; i <ARRAY_SIZE (cxl_irq_cap_table); i++) { +int irq; + +if (!cxl_irq_cap_table[i].get_max_msgnum) +continue ; + + irq = cxl_irq_cap_table[i].get_max_msgnum (cxlds);Quoted 8 lines
drivers/cxl/pci.c static void devm_cxl_pci_create_doe(struct cxl_dev_state *cxlds) · line 473
Quoted 2 lines
drivers/cxl/pci.c +2
-0+ vectors =max_t (int, irq, vectors); + }Forgive me if I have missed something, I only look at interrupt enable code once every few years, and the APIs are always a bit different, but is this not too early to read the message number? The number is not stable until either MSI or MSI-X has been selected below at
pci_alloc_irq_vectors()time?Well I keep getting wrapped around the axle on this one too.
Quoted 8 lines
This all started back when Jonathan originally attempted to allocate the maximum number of vectors a device _could_ allocate. But it was recommended that we determine the max number first then allocate that number.
This seems like a chicken and egg issue. How is the number not stable before calling
pci_alloc_irq_vectors()when you need the max msg number in that call?Quoted 8 lines
Are we talking about the same thing? I am talking about the value in the "Interrupt Message Number" field. That depends on whether MSI or MSI-X gets enabled. The number of vectors the device can support is static.
Since CXL is such an a la carte spec I think this is situation to just specify a large number of amx vectors to
pci_alloc_irq_vectors()and then find out after the fact if all of the interrupt generators that today'scxl_pciknows about in the device each got their own vector.I'd misunderstood how this worked and not read the spec :( I wrongly thought portdrv did the query first and allocated the vectors after, but that's not the case. It first allocates max entries, then frees them all and then allocates the ones that we find present. We should probably look to do something similar to that though I'm not sure even that code is always optimal.
https://elixir.bootlin.com/linux/v6.1-rc2/source/drivers/pci/pcie/portdrv_core.c#L101
In short that calls:
/* Allocate the maximum possible number of MSI/MSI-X vectors */nr_entries= pci_alloc_irq_vectors(dev, 1,PCIE_PORT_MAX_MSI_ENTRIES,PCI_IRQ_MSIX | PCI_IRQ_MSI);/* See how many and which Interrupt Message Numbers we actually use */ nvec =
pcie_message_numbers(dev, mask, &pme, &aer, &dpc);if (nvec != nr_entries) { pci_free_irq_vectors(dev); nr_entries = pci_alloc_irq_vectors(dev, nvec, nvec, PCI_IRQ_MSIX | PCI_IRQ_MSI); }My worry here is that the implicit assumption is that the vectors won't move if we reduce the overall number of vectors we are asking for...
However, imagine the case that we have a feature the driver doesn't know about that was previously at a higher vector. After reducing the vectors allocated the hardware might decide that feature needs its own vector whereas some others can be combined. Hence we'd end up with a less than ideal packing for the features we actually support.
Quoted 3 lines
Could do something iterative to solve this if it actually matters (increase number of vectors until the layout matches what we get with max possible vectors).
Is this cxl code allocating vectors for devices that might also be claimed by portdrv? I assume not because that sounds like a problem.
Ugh. I always feel like the portdrv design must be sub-optimal because this seems so hard to do cleanly.
pci_alloc_irq_vectors() has a lot of magic inside it and is great for most drivers, but the PCIe service IRQs are definitely unusual and maybe it's not the best fit for this situation.
If I understand correctly, Interrupt Message Numbers for all these PCIe services (hotplug, AER, DPC, etc) are restricted to 0-31 for both MSI and MSI-X, and the reason we don't just allocate 32 vectors all the time is to avoid consuming too many IRQs.
The MSI case is ugly because the Interrupt Message Number can change when we set Multiple Message Enable. Maybe we can separate it out and have a less than optimal solution for this case, like allocating one or two vectors and polling if that's not enough. I expect most devices will support MSI-X.
For MSI-X, the vector table is in a BAR, so IIUC there's no resource pressure until we actually assign an IRQ to a vector. Would it be feasible to always allocate vectors 0-31 of the MSI-X Table, and then selectively allocate IRQs for the services we need, so we might consume 32 MSI-X vectors but only a handful of IRQs?
Bjorn
Re: [PATCH 1/2] cxl/pci: Add generic MSI-X/MSI irq support
Quoted 5 lines · Bjorn Helgaas
Is this cxl code allocating vectors for devices that might also be claimed by portdrv? I assume not because that sounds like a problem.
Ugh. I always feel like the portdrv design must be sub-optimal because this seems so hard to do cleanly.
Yes, portdrv is a mess. And I fear we really need to bite the bullet rather sooner than later to sort much of this out by lifting all the logic to the core and just keep the "drivers" around for sysfs pretence.
And I think CXL is trying to run into a similar (but not quiete as bad) mess with it's overly modular approach. In either case the right thing would be to do anough early setup to find the requird number of interrupts and highest interrupt number and just request that once.
Davidlohr Bueso
maintainerRe: [PATCH 1/2] cxl/pci: Add generic MSI-X/MSI irq support
In reply to Bjorn HelgaasQuoted 11 lines · Bjorn Helgaas
Quoted 11 lines
In short that calls:
/* Allocate the maximum possible number of MSI/MSI-X vectors */nr_entries= pci_alloc_irq_vectors(dev, 1,PCIE_PORT_MAX_MSI_ENTRIES,PCI_IRQ_MSIX | PCI_IRQ_MSI);/* See how many and which Interrupt Message Numbers we actually use */ nvec =
pcie_message_numbers(dev, mask, &pme, &aer, &dpc);if (nvec != nr_entries) { pci_free_irq_vectors(dev);
Quoted 7 lines
Quoted 7 lines
nr_entries = pci_alloc_irq_vectors(dev, nvec, nvec, PCI_IRQ_MSIX | PCI_IRQ_MSI); }My worry here is that the implicit assumption is that the vectors won't move if we reduce the overall number of vectors we are asking for...
This would also apply to what is currently in portdrv machinery, no?
Quoted 11 lines
Quoted 11 lines
However, imagine the case that we have a feature the driver doesn't know about that was previously at a higher vector. After reducing the vectors allocated the hardware might decide that feature needs its own vector whereas some others can be combined. Hence we'd end up with a less than ideal packing for the features we actually support.
Could do something iterative to solve this if it actually matters (increase number of vectors until the layout matches what we get with max possible vectors).
Maybe do a bounded retry loop until we get stable value?
retry = 1;
do {
pci_alloc_irq_vectors(1, 32);
nvecs = get_max_msgnum(); // max(pmu, events, mbox, isolation)
pci_free_irq_vectors();
pci_alloc_irq_vectors(nvecs, nvecs);
new_nvecs = get_max_msgnum();
if (likely(new_nvecs == nvecs))
return 0;
pci_free_irq_vectors();
} while (retry--);return -1; // no irq support
But yeah I'm not sure how much we actually care about this. But if so, it also might be worth re-visiting the generic table thing, as if nothing else it can standalone co-exist and avoid allocating any irqs altogether if we know a-priori that there is no irq support.
Quoted 7 lines
Is this cxl code allocating vectors for devices that might also be claimed by portdrv? I assume not because that sounds like a problem.
Ugh. I always feel like the portdrv design must be sub-optimal because this seems so hard to do cleanly.
Quoted 8 lines
pci_alloc_irq_vectors()has a lot of magic inside it and is great for most drivers, but the PCIe service IRQs are definitely unusual and maybe it's not the best fit for this situation.If I understand correctly, Interrupt Message Numbers for all these PCIe services (hotplug, AER, DPC, etc) are restricted to 0-31 for both MSI and MSI-X, and the reason we don't just allocate 32 vectors all the time is to avoid consuming too many IRQs.
Most CXL features that can have irqs will normally use only the first 16, with the exception of isolation (cxl 3.0), which per the spec is up to 32.
Quoted 5 lines
The MSI case is ugly because the Interrupt Message Number can change when we set Multiple Message Enable. Maybe we can separate it out and have a less than optimal solution for this case, like allocating one or two vectors and polling if that's not enough. I expect most devices will support MSI-X.
Would only supporting MSI-X be so terrible?
Thanks,
Davidlohr
Re: [PATCH 1/2] cxl/pci: Add generic MSI-X/MSI irq support
Quoted 10 lines · Davidlohr Bueso
On Tue, 25 Oct 2022, Bjorn Helgaas wrote:
...Quoted 5 lines
The MSI case is ugly because the Interrupt Message Number can change when we set Multiple Message Enable. Maybe we can separate it out and have a less than optimal solution for this case, like allocating one or two vectors and polling if that's not enough. I expect most devices will support MSI-X.
Would only supporting MSI-X be so terrible?
My gut feeling is that polling when MSI-X isn't supported wouldn't be terrible, but I have no data on how common devices that only support MSI are.
Bjorn
Re: [PATCH 1/2] cxl/pci: Add generic MSI-X/MSI irq support
In reply to Davidlohr BuesoQuoted 70 lines · Davidlohr Bueso
Quoted 11 lines · Bjorn Helgaas
Quoted 11 lines
In short that calls:
/* Allocate the maximum possible number of MSI/MSI-X vectors */nr_entries= pci_alloc_irq_vectors(dev, 1,PCIE_PORT_MAX_MSI_ENTRIES,PCI_IRQ_MSIX | PCI_IRQ_MSI);/* See how many and which Interrupt Message Numbers we actually use */ nvec =
pcie_message_numbers(dev, mask, &pme, &aer, &dpc);if (nvec != nr_entries) { pci_free_irq_vectors(dev);Quoted 7 lines
Quoted 7 lines
nr_entries = pci_alloc_irq_vectors(dev, nvec, nvec, PCI_IRQ_MSIX | PCI_IRQ_MSI); }My worry here is that the implicit assumption is that the vectors won't move if we reduce the overall number of vectors we are asking for...
This would also apply to what is currently in portdrv machinery, no?
Quoted 11 lines
Quoted 11 lines
However, imagine the case that we have a feature the driver doesn't know about that was previously at a higher vector. After reducing the vectors allocated the hardware might decide that feature needs its own vector whereas some others can be combined. Hence we'd end up with a less than ideal packing for the features we actually support.
Could do something iterative to solve this if it actually matters (increase number of vectors until the layout matches what we get with max possible vectors).
Maybe do a bounded retry loop until we get stable value?
retry = 1;
do { pci_alloc_irq_vectors(1, 32); nvecs = get_max_msgnum(); // max(pmu, events, mbox, isolation) pci_free_irq_vectors(); pci_alloc_irq_vectors(nvecs, nvecs); new_nvecs = get_max_msgnum(); if (likely(new_nvecs == nvecs)) return 0; pci_free_irq_vectors(); } while (retry--);return -1; // no irq support
But yeah I'm not sure how much we actually care about this. But if so, it also might be worth re-visiting the generic table thing, as if nothing else it can standalone co-exist and avoid allocating any irqs altogether if we know a-priori that there is no irq support.
Quoted 11 lines
Is this cxl code allocating vectors for devices that might also be claimed by portdrv? I assume not because that sounds like a problem.
Ugh. I always feel like the portdrv design must be sub-optimal because this seems so hard to do cleanly.
pci_alloc_irq_vectors()has a lot of magic inside it and is great for most drivers, but the PCIe service IRQs are definitely unusual and maybe it's not the best fit for this situation.
Quoted 7 lines
Quoted 4 lines
If I understand correctly, Interrupt Message Numbers for all these PCIe services (hotplug, AER, DPC, etc) are restricted to 0-31 for both MSI and MSI-X, and the reason we don't just allocate 32 vectors all the time is to avoid consuming too many IRQs.
Most CXL features that can have irqs will normally use only the first 16, with the exception of isolation (cxl 3.0), which per the spec is up to 32.
Dan, Dave, and I were discussing this and we agree. For now the only things people are working on are within the first 16 so why not just request 16 as the max for now?
Ira
Quoted 11 lines
Quoted 6 lines
The MSI case is ugly because the Interrupt Message Number can change when we set Multiple Message Enable. Maybe we can separate it out and have a less than optimal solution for this case, like allocating one or two vectors and polling if that's not enough. I expect most devices will support MSI-X.
Would only supporting MSI-X be so terrible?
Thanks,
Davidlohr
Davidlohr Bueso
maintainerRe: [PATCH 1/2] cxl/pci: Add generic MSI-X/MSI irq support
Quoted 7 lines · Ira Weiny
Quoted 2 lines · Davidlohr Bueso
Most CXL features that can have irqs will normally use only the first 16, with the exception of isolation (cxl 3.0), which per the spec is up to 32.
Dan, Dave, and I were discussing this and we agree. For now the only things people are working on are within the first 16 so why not just request 16 as the max for now?
It is a fair compromise, yes.
Re: [PATCH 1/2] cxl/pci: Add generic MSI-X/MSI irq support
Quoted 11 lines · On Wed, 2 Nov 2022 17:18:33 -0700 Davidlohr Bueso <[email protected]> wrote:
Quoted 7 lines · Ira Weiny
Quoted 2 lines · Davidlohr Bueso
Most CXL features that can have irqs will normally use only the first 16, with the exception of isolation (cxl 3.0), which per the spec is up to 32.
Dan, Dave, and I were discussing this and we agree. For now the only things people are working on are within the first 16 so why not just request 16 as the max for now?
It is a fair compromise, yes.
works for me.
Re: [PATCH 1/2] cxl/pci: Add generic MSI-X/MSI irq support
Quoted 9 lines · Jonathan Cameron
Quoted 6 lines · On Wed, 2 Nov 2022 17:18:33 -0700 Davidlohr Bueso <[email protected]> wrote:
Quoted 4 lines · Ira Weiny
Quoted 2 lines · Davidlohr Bueso
Most CXL features that can have irqs will normally use only the first 16, with the exception of isolation (cxl 3.0), which per the spec is up to 32.
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Dan, Dave, and I were discussing this and we agree. For now the only things people are working on are within the first 16 so why not just request 16 as the max for now?
It is a fair compromise, yes.
works for me.
I made what I thought would be a simple change to your patch and built this into my series.
Unfortunately the following does not work with the current Qemu.
/*
* NOTE: Currently all the functions which are enabled for CXL require their
* vectors to be in the first 16. Allocate this number as the min/max. */
#define CXL_PCI_REQUIRED_VECTORS 16
...
rc = pci_alloc_irq_vectors(pdev, CXL_PCI_REQUIRED_VECTORS,
CXL_PCI_REQUIRED_VECTORS, PCI_IRQ_MSIX | PCI_IRQ_MSI);
This is because Qemu CXL devices only support (with the event changes I have made) 8 msg numbers. So the code fails to allocate any vectors.
I guess I should have known better. But allocating something less than 16 I guess needs to be allowed.
But that also means that beyond knowing _if_ irq's have been enabled I think each CXL feature needs to know the number of vectors allocated so they can ensure their msg numbers are going to work.
So how about the following as a diff to this patch?
In the event code I have then used the nr_irq_vecs field to determine if I should enable the irq for each log.
If you are ok with it I'm going to squash it into your patch and send out a new version of the event log series.
Thanks,
Ira
From 105561243c800442a2b7ff39b931e73b0a89bc34 Mon Sep 17 00:00:00 2001
From: Ira Weiny <[email protected]>
Date: Wed, 9 Nov 2022 12:35:07 -0800
Subject: [PATCH] squash: Allocate up to a static 16 vectors.
This covers the current desired features which CXL needs now.
--- drivers/cxl/cxlmem.h | 4 +-- drivers/cxl/cxlpci.h | 6 ++++ drivers/cxl/pci.c | 68 +++++++++----------------------------------- 3 files changed, 22 insertions(+), 56 deletions(-)
drivers/cxl/cxlmem.h +2 -2
@@ -212,7 +212,7 @@ struct cxl_endpoint_dvsec_info {* @info: Cached DVSEC information about the device. * @serial: PCIe Device Serial Number * @doe_mbs: PCI DOE mailbox array -* @has_irq: PCIe MSI-X/MSI support +* @nr_irq_vecs: Number of MSI-X/MSI vectors available * @mbox_send: @dev specific transport for transmitting mailbox commands * * See section 8.2.9.5.2 Capacity Configuration and Label Storage for
@@ -249,7 +249,7 @@ struct cxl_dev_state {struct xarray doe_mbs; -bool has_irq; +int nr_irq_vecs;int (*mbox_send)(struct cxl_dev_state *cxlds,struct cxl_mbox_cmd *cmd); };
drivers/cxl/cxlpci.h +6 -0
@@ -53,6 +53,12 @@#define CXL_DVSEC_REG_LOCATOR_BLOCK_ID_MASK GENMASK(15, 8)#define CXL_DVSEC_REG_LOCATOR_BLOCK_OFF_LOW_MASK GENMASK(31, 16) +/* +* NOTE: Currently all the functions which are enabled for CXL require their +* vectors to be in the first 16. Use this as the max. +*/ +#define CXL_PCI_REQUIRED_VECTORS 16 +/* Register Block Identifier (RBI) */ enum cxl_regloc_type {CXL_REGLOC_RBI_EMPTY =0 ,
drivers/cxl/pci.c +14 -54
95 lines not on this page · raw message
base-commit: aae703b02f92bde9264366c545e87cec451de471
prerequisite-patch-id: e3c882f3fce0872c2259538d00ad798236ec251f
prerequisite-patch-id: a8faa71e6d79cb30eae9b863349f0cb5ffa55b05
prerequisite-patch-id: f8e6edeb4a1d8bc4b34a509cb3d4a625becdf1b3
prerequisite-patch-id: 665a2b5af761a3f50e20da2fa8e5fdc9df13969d
prerequisite-patch-id: 5cd9f56597f9c8637201193849f68811a94d2309
prerequisite-patch-id: 89be9f2bd84118682b9b37e4f3a6e057fd4ad0d6
prerequisite-patch-id: 3de731a18a28c32572bf65f38e8eb2fb927e4f56
Signature · 1 line
2.37.2
Davidlohr Bueso
maintainerRe: [PATCH 1/2] cxl/pci: Add generic MSI-X/MSI irq support
Quoted 24 lines · Ira Weiny
Unfortunately the following does not work with the current Qemu.
/*
* NOTE: Currently all the functions which are enabled for CXL require their
* vectors to be in the first 16. Allocate this number as the min/max. */
#defineCXL_PCI_REQUIRED_VECTORS16...
rc = pci_alloc_irq_vectors(pdev,
CXL_PCI_REQUIRED_VECTORS,
CXL_PCI_REQUIRED_VECTORS,PCI_IRQ_MSIX|PCI_IRQ_MSI);This is because Qemu CXL devices only support (with the event changes I have made) 8 msg numbers. So the code fails to allocate any vectors.
I guess I should have known better. But allocating something less than 16 I guess needs to be allowed.
But that also means that beyond knowing _if_ irq's have been enabled I think each CXL feature needs to know the number of vectors allocated so they can ensure their msg numbers are going to work.
Quoted 7 lines
So how about the following as a diff to this patch?
In the event code I have then used the
nr_irq_vecsfield to determine if I should enable the irq for each log.If you are ok with it I'm going to squash it into your patch and send out a new version of the event log series.
LGTM, thanks.
Re: [PATCH 1/2] cxl/pci: Add generic MSI-X/MSI irq support
In reply to Davidlohr BuesoQuoted 52 lines · On Wed, 2 Nov 2022 10:15:24 -0700 Davidlohr Bueso <[email protected]> wrote:
Quoted 11 lines · Bjorn Helgaas
Quoted 11 lines
In short that calls:
/* Allocate the maximum possible number of MSI/MSI-X vectors */nr_entries= pci_alloc_irq_vectors(dev, 1,PCIE_PORT_MAX_MSI_ENTRIES,PCI_IRQ_MSIX | PCI_IRQ_MSI);/* See how many and which Interrupt Message Numbers we actually use */ nvec =
pcie_message_numbers(dev, mask, &pme, &aer, &dpc);if (nvec != nr_entries) { pci_free_irq_vectors(dev);Quoted 7 lines
Quoted 7 lines
nr_entries = pci_alloc_irq_vectors(dev, nvec, nvec, PCI_IRQ_MSIX | PCI_IRQ_MSI); }My worry here is that the implicit assumption is that the vectors won't move if we reduce the overall number of vectors we are asking for...
This would also apply to what is currently in portdrv machinery, no?
Quoted 11 lines
Quoted 11 lines
However, imagine the case that we have a feature the driver doesn't know about that was previously at a higher vector. After reducing the vectors allocated the hardware might decide that feature needs its own vector whereas some others can be combined. Hence we'd end up with a less than ideal packing for the features we actually support.
Could do something iterative to solve this if it actually matters (increase number of vectors until the layout matches what we get with max possible vectors).
Maybe do a bounded retry loop until we get stable value?
retry = 1;
do { pci_alloc_irq_vectors(1, 32); nvecs = get_max_msgnum(); // max(pmu, events, mbox, isolation) pci_free_irq_vectors(); pci_alloc_irq_vectors(nvecs, nvecs); new_nvecs = get_max_msgnum(); if (likely(new_nvecs == nvecs)) return 0; pci_free_irq_vectors(); } while (retry--);
Quoted 1 line
return -1; // no irq support
Yup. That's pretty much what I was thinking - if we care :)
Quoted 2 lines
But yeah I'm not sure how much we actually care about this.
That was my feeling. This might be worth a comment to say that it's not guaranteed to be optimal (in portdrv), but probably a won't fix.
Jonathan
Davidlohr Bueso
maintainerPATCH v3 2/2 cxl/mbox: Wire up irq support
In reply to Davidlohr BuesoWith enough vectors properly allocated, this adds support for (the primary) mailbox interrupt, which is needed for background completion handling, beyond polling.
--- drivers/cxl/cxl.h | 1 + drivers/cxl/pci.c | 29 ++++++++++++++++++++++++++++- 2 files changed, 29 insertions(+), 1 deletion(-)
drivers/cxl/cxl.h +1 -0
@@ -135,6 +135,7 @@ static inline int ways_to_cxl(unsigned int ways, u8 *iw)/* CXL 2.0 8.2.8.4 Mailbox Registers */ #define CXLDEV_MBOX_CAPS_OFFSET 0x00#define CXLDEV_MBOX_CAP_PAYLOAD_SIZE_MASK GENMASK(4, 0) +#define CXLDEV_MBOX_CAP_IRQ_MSGNUM_MASK GENMASK(10, 7)#define CXLDEV_MBOX_CTRL_OFFSET 0x04#define CXLDEV_MBOX_CTRL_DOORBELL BIT(0)#define CXLDEV_MBOX_CMD_OFFSET 0x08
drivers/cxl/pci.c +28 -1
@@ -274,6 +274,32 @@ static int cxl_pci_setup_mailbox(struct cxl_dev_state *cxlds)return 0 ; } +static int cxl_pci_mbox_get_max_msgnum (struct cxl_dev_state *cxlds) +{ +int cap; + + cap =readl (cxlds->regs.mbox +CXLDEV_MBOX_CAPS_OFFSET ); +return FIELD_GET (CXLDEV_MBOX_CAP_IRQ_MSGNUM_MASK , cap); +} + +static irqreturn_t cxl_pci_mbox_irq (int irq,void *id) +{ +/* TODO: handle completion of background commands */ +return IRQ_HANDLED ; +} + +static void cxl_pci_mbox_irqsetup (struct cxl_dev_state *cxlds) +{ +struct device *dev = cxlds->dev; +struct pci_dev *pdev =to_pci_dev (dev); +int irq; + + irq =cxl_pci_mbox_get_max_msgnum (cxlds); +if (!pci_request_irq (pdev, irq, cxl_pci_mbox_irq,NULL , + cxlds,"%s-mailbox" ,dev_name (dev))) +dev_dbg (dev,"Mailbox irq (%d) supported" , irq); +} +static int cxl_map_regblock (struct pci_dev *pdev,struct cxl_register_map *map) {void __iomem *addr;
@@ -442,7 +468,7 @@ struct cxl_irq_cap { };static const struct cxl_irq_cap cxl_irq_cap_table[] = { -NULL + {"mailbox" , cxl_pci_mbox_get_max_msgnum }, };static void cxl_pci_free_irq_vectors (void *data)
@@ -562,6 +588,7 @@ static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)return rc;if (!cxl_pci_alloc_irq_vectors (cxlds)) { +cxl_pci_mbox_irqsetup (cxlds); cxlds->has_irq = true; }else cxlds->has_irq = false;
Signature · 1 line
2.38.0
Re: [PATCH 2/2] cxl/mbox: Wire up irq support
Quoted 5 lines · On Mon, 17 Oct 2022 20:00:10 -0700 Davidlohr Bueso <[email protected]> wrote:
With enough vectors properly allocated, this adds support for (the primary) mailbox interrupt, which is needed for background completion handling, beyond polling.
Signed-off-by Davidlohr Bueso <[email protected]>
Whilst I get the need for an example, I'd rather this didn't go in until we have that background handler in place.
Unused infrastructure tends to rot or not be quite what is needed.
Follow up comment below.
Jonathan
drivers/cxl/pci.c static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id) · line 591
Quoted 6 lines
static void
cxl_pci_free_irq_vectors(void *data)drivers/cxl/pci.c +1
-0@@ -562,6 +588,7 @@ static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)return rc;if (!cxl_pci_alloc_irq_vectors (cxlds)) { +cxl_pci_mbox_irqsetup (cxlds);
Ah. That rather wrecks my previous suggestion :)
Quoted 3 lines
drivers/cxl/pci.c +0
-0cxlds->has_irq = true; }else cxlds->has_irq = false;
Davidlohr Bueso
maintainerRe: [PATCH 2/2] cxl/mbox: Wire up irq support
Quoted 2 lines · Jonathan Cameron
Whilst I get the need for an example, I'd rather this didn't go in until we have that background handler in place.
One of the reasons why the bg stuff hasn't been re-posted is because the only user currently is the sanitation work (and hopefully maybe scan media soon), which in turn depends on the cache-flushing thing to be picked up (unless someone starts ranting again against wbinvd). And that is in Dave's pmem security series which I'm hoping will be picked up for v6.3 at some point.
So I guess we're a while away from the irq thing. I was mostly suggesting sending an mbox-only just to layout the pci_alloc_irq_vectors if we're not using the common table, but per the above this might not be sooner than the pmu or events stuff...
Quoted 2 lines
Unused infrastructure tends to rot or not be quite what is needed.
No arguing there.
Thanks,
Davidlohr
Re: [PATCH 2/2] cxl/mbox: Wire up irq support
Quoted 19 lines · Davidlohr Bueso
Quoted 2 lines · Jonathan Cameron
Whilst I get the need for an example, I'd rather this didn't go in until we have that background handler in place.
One of the reasons why the bg stuff hasn't been re-posted is because the only user currently is the sanitation work (and hopefully maybe scan media soon), which in turn depends on the cache-flushing thing to be picked up (unless someone starts ranting again against wbinvd). And that is in Dave's pmem security series which I'm hoping will be picked up for v6.3 at some point.
So I guess we're a while away from the irq thing. I was mostly suggesting sending an mbox-only just to layout the
pci_alloc_irq_vectorsif we're not using the common table, but per the above this might not be sooner than the pmu or events stuff...
Quoted 4 lines
Quoted 2 lines
Unused infrastructure tends to rot or not be quite what is needed.
No arguing there.
I do feel that event notifications are likely the nearest consumer because the whole reason to do the RCD work is to support OS native error handling for these CXL 1.1+ devices that people can actually get their hands on today. Error handling and other RAS flows need to be able to hear the device's screams for help.
RE: [PATCH 2/2] cxl/mbox: Wire up irq support
In reply to Davidlohr BuesoQuoted 41 lines · Davidlohr Bueso
With enough vectors properly allocated, this adds support for (the primary) mailbox interrupt, which is needed for background completion handling, beyond polling.
Signed-off-by Davidlohr Bueso <[email protected]>--- drivers/cxl/cxl.h | 1 + drivers/cxl/pci.c | 29 ++++++++++++++++++++++++++++- 2 files changed, 29 insertions(+), 1 deletion(-)drivers/cxl/cxl.h +1
-0@@ -135,6 +135,7 @@ static inline int ways_to_cxl(unsigned int ways, u8 *iw)/* CXL 2.0 8.2.8.4 Mailbox Registers */ #define CXLDEV_MBOX_CAPS_OFFSET 0x00#define CXLDEV_MBOX_CAP_PAYLOAD_SIZE_MASK GENMASK(4, 0) +#define CXLDEV_MBOX_CAP_IRQ_MSGNUM_MASK GENMASK(10, 7)#define CXLDEV_MBOX_CTRL_OFFSET 0x04#define CXLDEV_MBOX_CTRL_DOORBELL BIT(0)#define CXLDEV_MBOX_CMD_OFFSET 0x08drivers/cxl/pci.c +11
-0@@ -274,6 +274,32 @@ static int cxl_pci_setup_mailbox(struct cxl_dev_state *cxlds)return 0 ; } +static int cxl_pci_mbox_get_max_msgnum (struct cxl_dev_state *cxlds) +{ +int cap; + + cap =readl (cxlds->regs.mbox +CXLDEV_MBOX_CAPS_OFFSET ); +return FIELD_GET (CXLDEV_MBOX_CAP_IRQ_MSGNUM_MASK , cap); +} + +static irqreturn_t cxl_pci_mbox_irq (int irq,void *id) +{ +/* TODO: handle completion of background commands */
drivers/cxl/pci.c static int cxl_pci_setup_mailbox(struct cxl_dev_state *cxlds) · line 297
Quoted 10 lines
drivers/cxl/pci.c +10
-0+return IRQ_HANDLED ; +} + +static void cxl_pci_mbox_irqsetup (struct cxl_dev_state *cxlds) +{ +struct device *dev = cxlds->dev; +struct pci_dev *pdev =to_pci_dev (dev); +int irq; + + irq =cxl_pci_mbox_get_max_msgnum (cxlds);
In this context cxl_pci_mbox_get_max_msgnum() looks misnamed. There is no max, it's only *the* message number for the mailbox.
Quoted 28 lines
drivers/cxl/pci.c +7
-1+if (!pci_request_irq (pdev, irq, cxl_pci_mbox_irq,NULL , + cxlds,"%s-mailbox" ,dev_name (dev))) +dev_dbg (dev,"Mailbox irq (%d) supported" , irq); +} +static int cxl_map_regblock (struct pci_dev *pdev,struct cxl_register_map *map) {void __iomem *addr;@@ -442,7 +468,7 @@ struct cxl_irq_cap { };static const struct cxl_irq_cap cxl_irq_cap_table[] = { -NULL + {"mailbox" , cxl_pci_mbox_get_max_msgnum }, };static void cxl_pci_free_irq_vectors (void *data)@@ -562,6 +588,7 @@ static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)return rc;if (!cxl_pci_alloc_irq_vectors (cxlds)) { +cxl_pci_mbox_irqsetup (cxlds); cxlds->has_irq = true; }else cxlds->has_irq = false;Signature · 1 line
2.38.0