PCI: endpoint: pci-epf-vntb: using platform MSI as doorbell
Subsystems (4)
Frank Li
maintainerv13 cover PCI: endpoint: pci-epf-vntb: using platform MSI as doorbell
This patch continue https://lore.kernel.org/linux-pci/[email protected]/
The above patch series fork to two parts. clean up and use msi as
doorbell.
clean up patch already been in https://git.kernel.org/pub/scm/linux/kernel/git/lpieralisi/pci.git/commit/?h=pci/endpoint
These patches based on the above git branch.
Change from v12 to v13:
fixed Lorenzo Pieralisi comments in https://lore.kernel.org/imx/Yz%2FuMiElbqB3ThGd@lpieralisi/T/#u
- update diagram
- Add platform_msi_domain_free_irqs at failure path
- sizeof(u32) is because hardcode by ntb_hw_epf.c
Frank Li (2):
PCI: endpoint: pci-epf-vntb: change doorbell register offset calc
mathod
PCI: endpoint: pci-epf-vntb: using platform MSI as doorbell
drivers/pci/endpoint/functions/pci-epf-vntb.c | 150 +++++++++++++++--- 1 file changed, 127 insertions(+), 23 deletions(-)
Signature · 1 line
2.34.1
Frank Li
maintainerPATCH v13 1/2 PCI: endpoint: pci-epf-vntb: change doorbell register offset calc mathod
In drivers/ntb/hw/epf/ntb_hw_epf.cntb_epf_peer_db_set()
{
...
db_offset = readl(ndev->ctrl_reg + NTB_EPF_DB_OFFSET(interrupt_num)); writel(db_data, ndev->db_reg + (db_entry_size * interrupt_num) +
db_offset);
...
}
The door register offset's formular is
offset = db_entry_size * interrupt_num + db_offset[interrupt_number]
Previous db_entry_size is 4, all db_offset is 0.
irq | offset
--------------
0 0
1 4
2 8
...Change to db_entry_size is 0 and db_offset is 0, 4, 8, ... So we can get the same map value between irq and offset. This will be convenience for hardware doorbell register memory map.
--- drivers/pci/endpoint/functions/pci-epf-vntb.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/pci/endpoint/functions/pci-epf-vntb.c +2 -2
@@ -461,11 +461,11 @@ static int epf_ntb_config_spad_bar_alloc(struct epf_ntb *ntb) ctrl->num_mws = ntb->num_mws; ntb->spad_size = spad_size; - ctrl->db_entry_size =sizeof (u32); + ctrl->db_entry_size =0 ;for (i =0 ; i < ntb->db_count; i++) { ntb->reg->db_data[i] =1 + i; - ntb->reg->db_offset[i] =0 ; + ntb->reg->db_offset[i] =sizeof (u32) * i; }return 0 ;
Signature · 1 line
2.34.1
Re: [PATCH v13 1/2] PCI: endpoint: pci-epf-vntb: change doorbell register offset calc mathod
Quoted 10 lines · Frank Li
In
drivers/ntb/hw/epf/ntb_hw_epf.cntb_epf_peer_db_set(){ ... db_offset = readl(ndev->ctrl_reg + NTB_EPF_DB_OFFSET(interrupt_num)); writel(db_data, ndev->db_reg + (db_entry_size * interrupt_num) + db_offset); ... }
Quoted 2 lines
The door register offset's formular is
offset =db_entry_size*interrupt_num+ db_offset[interrupt_number]
You did not mention the DB BAR here. Without that, this calculation doesn't make sense.
Quoted 2 lines
Previous
db_entry_sizeis 4, alldb_offsetis 0.
s/Previous/Previously
Quoted 11 lines
irq | offset -------------- 0 0 1 4 2 8 ...Change to
db_entry_sizeis 0 anddb_offsetis 0, 4, 8, ... So we can get the same map value between irq and offset. This will be convenience for hardware doorbell register memory map.
In your irq-imx-mu-msi.c driver, the msi_address is calculated as:
``` u64 addr = msi_data->msiir_addr + 4 * data->hwirq; ```
So the MSI addresses itself are of 4 bytes width. So the offsets will be separated by 8 bytes like, 0, 8, 16,... and this won't match the MSI addresses as they are 4 bytes apart.
So you want to change the offset to 0, 4, 8,... by zeroing db_entry_size, right?
Quoted 10 lines
Signed-off-by Frank Li <[email protected]>--- drivers/pci/endpoint/functions/pci-epf-vntb.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)drivers/pci/endpoint/functions/pci-epf-vntb.c +0
-0
drivers/pci/endpoint/functions/pci-epf-vntb.c static int epf_ntb_config_spad_bar_alloc(struct epf_ntb *ntb) · line 468
Quoted 10 lines
drivers/pci/endpoint/functions/pci-epf-vntb.c +2
-2@@ -461,11 +461,11 @@ static int epf_ntb_config_spad_bar_alloc(struct epf_ntb *ntb) ctrl->num_mws = ntb->num_mws; ntb->spad_size = spad_size; - ctrl->db_entry_size =sizeof (u32); + ctrl->db_entry_size =0 ;for (i =0 ; i < ntb->db_count; i++) { ntb->reg->db_data[i] =1 + i; - ntb->reg->db_offset[i] =0 ; + ntb->reg->db_offset[i] =sizeof (u32) * i;
If my above understanding is correct, then you could just reassign "db_entry_size" in epf_ntb_epc_msi_init().
Thanks,
Mani
Quoted 6 lines
drivers/pci/endpoint/functions/pci-epf-vntb.c +0
-0}return 0 ;Signature · 1 line
2.34.1
Signature · 1 line
மணிவண்ணன் சதாசிவம்
Frank Li
maintainerRE: [EXT] Re: [PATCH v13 1/2] PCI: endpoint: pci-epf-vntb: change doorbell register offset calc mathod
Quoted 31 lines
-----Original Message-----
From: Manivannan Sadhasivam <[email protected]>
Sent: Thursday, November 24, 2022 3:19 AM
To: Frank Li <[email protected]>
Cc: [email protected]; Aisheng Dong <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; linux- [email protected]; dl-linux-imx <[email protected]>; linux- [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Peng Fan <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Subject: [EXT] Re: [PATCH v13 1/2] PCI: endpoint: pci-epf-vntb: change doorbell register offset calc mathodCaution: EXT Email
Quoted 10 lines · Frank Li
In
drivers/ntb/hw/epf/ntb_hw_epf.cntb_epf_peer_db_set(){ ... db_offset = readl(ndev->ctrl_reg + NTB_EPF_DB_OFFSET(interrupt_num)); writel(db_data, ndev->db_reg + (db_entry_size * interrupt_num) + db_offset); ... }
Quoted 5 lines
Quoted 2 lines
The door register offset's formular is
offset =db_entry_size*interrupt_num+ db_offset[interrupt_number]You did not mention the DB BAR here. Without that, this calculation doesn't make sense.
Doorbell register offset should means Base on DB BAR. How about "The formula of door register offset refer to DB BAR"?
Quoted 20 lines
Quoted 3 lines
Previous
db_entry_sizeis 4, alldb_offsetis 0.s/Previous/Previously
Quoted 11 lines
irq | offset -------------- 0 0 1 4 2 8 ...Change to
db_entry_sizeis 0 anddb_offsetis 0, 4, 8, ... So we can get the same map value between irq and offset. This will be convenience for hardware doorbell register memory map.In your
irq-imx-mu-msi.cdriver, themsi_addressis calculated as:
Quoted 7 lines
``` u64 addr = msi_data->msiir_addr + 4 * data->hwirq; ```So the MSI addresses itself are of 4 bytes width. So the offsets will be separated by 8 bytes like, 0, 8, 16,... and this won't match the MSI addresses as they are 4 bytes apart.
Addr is absolute physical IO address, which increased by 4. But it doesn't matter. It should be okay if range is between 2^32.
Quoted 3 lines
So you want to change the offset to 0, 4, 8,... by zeroing
db_entry_size, right?
I want to directly using db_offset[irq] value as offset. It will be simple.
I am not sure why ntb_hw_epf.c use below formular.
"Db_offset[irq] + irq * db_entry_size"
Db_entry_size = 0 will be simple, all offset will be controlled by db_offset[]
You can save db_offset[] as 0, 4, 8... or 0, 8, 16 as needs.
Quoted 19 lines
Review
Quoted 7 lines
Signed-off-by Frank Li <[email protected]>--- drivers/pci/endpoint/functions/pci-epf-vntb.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)(quoted diff) +0
-0
b/drivers/pci/endpoint/functions/pci-epf-vntb.cdrivers/pci/endpoint/functions/pci-epf-vntb.c static int epf_ntb_config_spad_bar_alloc(struct
Quoted 4 lines
index
04698e7995a5..0d744975f815100644drivers/pci/endpoint/functions/pci-epf-vntb.c +0
-0@@ -461,11 +461,11 @@ static int epf_ntb_config_spad_bar_alloc(struct
epf_ntb*ntb)Quoted 6 lines
drivers/pci/endpoint/functions/pci-epf-vntb.c +1
-1ctrl->num_mws = ntb->num_mws; ntb->spad_size = spad_size; - ctrl->db_entry_size =sizeof (u32); + ctrl->db_entry_size =0 ;
Quoted 7 lines
drivers/pci/endpoint/functions/pci-epf-vntb.c static int epf_ntb_config_spad_bar_alloc(struct epf_ntb *ntb) · line 468
Quoted 4 lines
drivers/pci/endpoint/functions/pci-epf-vntb.c +1
-1for (i =0 ; i < ntb->db_count; i++) { ntb->reg->db_data[i] =1 + i; - ntb->reg->db_offset[i] =0 ; + ntb->reg->db_offset[i] =sizeof (u32) * i;If my above understanding is correct, then you could just reassign "
db_entry_size" inepf_ntb_epc_msi_init().
Yes, that's one method.
I want to use one method to calc db offset for both software polling and MSI. So overall logic should be simple.
Frank Li
Quoted 13 lines
Thanks,
ManiQuoted 6 lines
drivers/pci/endpoint/functions/pci-epf-vntb.c +0
-1}return 0 ; --2.34.1
--
மணிவண்ணன் சதாசிவம்
Re: [EXT] Re: [PATCH v13 1/2] PCI: endpoint: pci-epf-vntb: change doorbell register offset calc mathod
Quoted 31 lines · Frank Li
Quoted 31 lines
-----Original Message-----
From: Manivannan Sadhasivam <[email protected]>
Sent: Thursday, November 24, 2022 3:19 AM
To: Frank Li <[email protected]>
Cc: [email protected]; Aisheng Dong <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; linux- [email protected]; dl-linux-imx <[email protected]>; linux- [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Peng Fan <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Subject: [EXT] Re: [PATCH v13 1/2] PCI: endpoint: pci-epf-vntb: change doorbell register offset calc mathodCaution: EXT Email
Quoted 10 lines · Frank Li
In
drivers/ntb/hw/epf/ntb_hw_epf.cntb_epf_peer_db_set(){ ... db_offset = readl(ndev->ctrl_reg + NTB_EPF_DB_OFFSET(interrupt_num)); writel(db_data, ndev->db_reg + (db_entry_size * interrupt_num) + db_offset); ... }
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
The door register offset's formular is
offset =db_entry_size*interrupt_num+ db_offset[interrupt_number]You did not mention the DB BAR here. Without that, this calculation doesn't make sense.
Doorbell register offset should means Base on DB BAR. How about "The formula of door register offset refer to DB BAR"?
"Doobell register offset in DB BAR is calculated using:"
Quoted 69 lines
Quoted 21 lines
Quoted 4 lines
Previous
db_entry_sizeis 4, alldb_offsetis 0.s/Previous/Previously
Quoted 11 lines
irq | offset -------------- 0 0 1 4 2 8 ...Change to
db_entry_sizeis 0 anddb_offsetis 0, 4, 8, ... So we can get the same map value between irq and offset. This will be convenience for hardware doorbell register memory map.In your
irq-imx-mu-msi.cdriver, themsi_addressis calculated as:Quoted 7 lines
``` u64 addr = msi_data->msiir_addr + 4 * data->hwirq; ```So the MSI addresses itself are of 4 bytes width. So the offsets will be separated by 8 bytes like, 0, 8, 16,... and this won't match the MSI addresses as they are 4 bytes apart.
Addr is absolute physical IO address, which increased by 4. But it doesn't matter. It should be okay if range is between 2^32.
Quoted 3 lines
So you want to change the offset to 0, 4, 8,... by zeroing
db_entry_size, right?I want to directly using db_offset[irq] value as offset. It will be simple.
I am not sure why
ntb_hw_epf.cuse below formular.
"Db_offset[irq] + irq *db_entry_size"
Db_entry_size= 0 will be simple, all offset will be controlled by db_offset[]You can save db_offset[] as 0, 4, 8... or 0, 8, 16 as needs.
Quoted 24 lines
Review
Quoted 7 lines
Signed-off-by Frank Li <[email protected]>--- drivers/pci/endpoint/functions/pci-epf-vntb.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)(quoted diff) +0
-0
b/drivers/pci/endpoint/functions/pci-epf-vntb.cdrivers/pci/endpoint/functions/pci-epf-vntb.c static int epf_ntb_config_spad_bar_alloc(struct
Quoted 4 lines
index
04698e7995a5..0d744975f815100644drivers/pci/endpoint/functions/pci-epf-vntb.c +0
-0@@ -461,11 +461,11 @@ static int epf_ntb_config_spad_bar_alloc(struct
epf_ntb*ntb)Quoted 10 lines
drivers/pci/endpoint/functions/pci-epf-vntb.c +2
-2ctrl->num_mws = ntb->num_mws; ntb->spad_size = spad_size; - ctrl->db_entry_size =sizeof (u32); + ctrl->db_entry_size =0 ;for (i =0 ; i < ntb->db_count; i++) { ntb->reg->db_data[i] =1 + i; - ntb->reg->db_offset[i] =0 ; + ntb->reg->db_offset[i] =sizeof (u32) * i;
Quoted 7 lines
Quoted 2 lines
If my above understanding is correct, then you could just reassign "
db_entry_size" inepf_ntb_epc_msi_init().Yes, that's one method.
I want to use one method to calc db offset for both software polling and MSI. So overall logic should be simple.
I think it is better to leave db_entry_size for polling as it is and modify it for MSI alone.
Thanks,
Mani
Quoted 15 lines
Frank Li
Quoted 13 lines
Thanks,
ManiQuoted 6 lines
drivers/pci/endpoint/functions/pci-epf-vntb.c +0
-1}return 0 ; --2.34.1
--
மணிவண்ணன் சதாசிவம்
Signature · 1 line
மணிவண்ணன் சதாசிவம்
Frank Li
maintainerRE: [EXT] Re: [PATCH v13 1/2] PCI: endpoint: pci-epf-vntb: change doorbell register offset calc mathod
Quoted 144 lines
-----Original Message-----
From: Manivannan Sadhasivam <[email protected]>
Sent: Thursday, November 24, 2022 12:51 PM
To: Frank Li <[email protected]>
Cc: [email protected]; Aisheng Dong <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; linux-arm- [email protected]; dl-linux-imx <[email protected]>; linux- [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Peng Fan <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [EXT] Re: [PATCH v13 1/2] PCI: endpoint: pci-epf-vntb: change doorbell register offset calc mathodCaution: EXT Email
Quoted 8 lines · Frank Li
Quoted 8 lines
-----Original Message-----
From: Manivannan Sadhasivam <[email protected]>
Sent: Thursday, November 24, 2022 3:19 AM
To: Frank Li <[email protected]>
Cc: [email protected]; Aisheng Dong <[email protected]>; [email protected]; [email protected];Quoted 3 lines
Quoted 3 lines
[email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; linux- [email protected]; dl-linux-imx <[email protected]>;
linux-
Quoted 7 lines
Quoted 7 lines
[email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Peng Fan <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Subject: [EXT] Re: [PATCH v13 1/2] PCI: endpoint: pci-epf-vntb: change doorbell register offset calc mathodQuoted 8 lines
Quoted 8 lines
Caution: EXT Email
Quoted 5 lines · Frank Li
In
drivers/ntb/hw/epf/ntb_hw_epf.cntb_epf_peer_db_set(){ ... db_offset = readl(ndev->ctrl_reg +
NTB_EPF_DB_OFFSET(interrupt_num));Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
writel(db_data, ndev->db_reg + (db_entry_size * interrupt_num) + db_offset); ... }The door register offset's formular is
offset =db_entry_size*interrupt_num+db_offset[interrupt_number]
Quoted 2 lines
Quoted 2 lines
You did not mention the DB BAR here. Without that, this calculation
doesn't
Quoted 4 lines
Quoted 1 line
make sense.
Doorbell register offset should means Base on DB BAR. How about "The formula of door register offset refer to DB BAR"?
"Doobell register offset in DB BAR is calculated using:"
Quoted 19 lines
Quoted 19 lines
Quoted 4 lines
Previous
db_entry_sizeis 4, alldb_offsetis 0.s/Previous/Previously
Quoted 11 lines
irq | offset -------------- 0 0 1 4 2 8 ...Change to
db_entry_sizeis 0 anddb_offsetis 0, 4, 8, ... So we can get the same map value between irq and offset. This will be convenience for hardware doorbell register memory map.Quoted 8 lines
Quoted 8 lines
In your
irq-imx-mu-msi.cdriver, themsi_addressis calculated as:``` u64 addr = msi_data->msiir_addr + 4 * data->hwirq; ```So the MSI addresses itself are of 4 bytes width. So the offsets will be separated by 8 bytes like, 0, 8, 16,... and this won't match the MSI
addresses
Quoted 3 lines
Quoted 1 line
as they are 4 bytes apart.
Addr is absolute physical IO address, which increased by 4. But it doesn't
matter.
Quoted 21 lines
It should be okay if range is between 2^32.
Quoted 3 lines
So you want to change the offset to 0, 4, 8,... by zeroing
db_entry_size, right?I want to directly using db_offset[irq] value as offset. It will be simple.
I am not sure why
ntb_hw_epf.cuse below formular.
"Db_offset[irq] + irq *db_entry_size"
Db_entry_size= 0 will be simple, all offset will be controlled by db_offset[]You can save db_offset[] as 0, 4, 8... or 0, 8, 16 as needs.
Quoted 6 lines
Quoted 6 lines
Signed-off-by Frank Li <[email protected]>--- drivers/pci/endpoint/functions/pci-epf-vntb.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)Quoted 6 lines
Quoted 6 lines
Review
Quoted 1 line
(quoted diff) +0
-0
b/drivers/pci/endpoint/functions/pci-epf-vntb.cdrivers/pci/endpoint/functions/pci-epf-vntb.c static int
Quoted 4 lines
index
04698e7995a5..0d744975f815100644drivers/pci/endpoint/functions/pci-epf-vntb.c +0
-0@@ -461,11 +461,11 @@ static intepf_ntb_config_spad_bar_alloc(struct
Quoted 15 lines
Quoted 14 lines
epf_ntb*ntb)drivers/pci/endpoint/functions/pci-epf-vntb.c static int epf_ntb_config_spad_bar_alloc(struct epf_ntb *ntb) · line 468
Quoted 10 lines
drivers/pci/endpoint/functions/pci-epf-vntb.c +2
-2ctrl->num_mws = ntb->num_mws; ntb->spad_size = spad_size; - ctrl->db_entry_size =sizeof (u32); + ctrl->db_entry_size =0 ;for (i =0 ; i < ntb->db_count; i++) { ntb->reg->db_data[i] =1 + i; - ntb->reg->db_offset[i] =0 ; + ntb->reg->db_offset[i] =sizeof (u32) * i;If my above understanding is correct, then you could just reassign "
db_entry_size" inepf_ntb_epc_msi_init().
Quoted 7 lines
Quoted 4 lines
Yes, that's one method.
I want to use one method to calc db offset for both software polling and MSI. So overall logic should be simple.I think it is better to leave
db_entry_sizefor polling as it is and modify it for MSI alone.
Okay, that means needn't this patch at all.
Quoted 22 lines
Thanks,
ManiQuoted 15 lines
Frank Li
Quoted 13 lines
Thanks,
ManiQuoted 6 lines
drivers/pci/endpoint/functions/pci-epf-vntb.c +0
-1}return 0 ; --2.34.1
--
மணிவண்ணன் சதாசிவம்--
மணிவண்ணன் சதாசிவம்
Frank Li
maintainerPATCH v13 2/2 PCI: endpoint: pci-epf-vntb: using platform MSI as doorbell
In reply to Frank Li┌────────────┐ ┌───────────────────────────────────┐ ┌────────────────┐ │ │ │ │ │ │ │ │ │ PCI Endpoint │ │ PCI Host │ │ │ │ │ │ │ │ │◄──┤ 1.platform_msi_domain_alloc_irqs()│ │ │ │ │ │ │ │ │ │ MSI ├──►│ 2.write_msi_msg() ├──►├─BAR<n> │ │ Controller │ │ update doorbell register address│ │ │ │ │ │ for BAR │ │ │ │ │ │ │ │ 3. Write BAR<n>│ │ │◄──┼───────────────────────────────────┼───┤ │ │ │ │ │ │ │ │ ├──►│ 4.Irq Handle │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ └────────────┘ └───────────────────────────────────┘ └────────────────┘
Using platform MSI interrupt controller as endpoint(EP)'s doorbell.
The memory assigned for BAR region by the PCI host is mapped to the message address of platform msi interrupt controller in PCI Endpoint. Such that, whenever the PCI host writes to the BAR region, it will trigger an IRQ in the EP.
Basic working follow as
1. EP function driver call platform_msi_domain_alloc_irqs() alloc a MSI irq from MSI controller with call back function write_msi_msg();
2. write_msg_msg will config BAR and map to address defined in msi_msg;
3. Host side trigger an IRQ at Endpoint by write to BAR region.
Add MSI doorbell support for pci-epf-vntb. Query if system has an MSI controller. Set up doorbell address according to struct msi_msg.
So PCI host can write this doorbell address to trigger EP side's IRQ.
If no MSI controller exists, fall back to software polling.
--- drivers/pci/endpoint/functions/pci-epf-vntb.c | 146 +++++++++++++++--- 1 file changed, 125 insertions(+), 21 deletions(-)
drivers/pci/endpoint/functions/pci-epf-vntb.c +125 -21
@@ -44,6 +44,7 @@#include <linux/pci-epc.h> #include <linux/pci-epf.h> #include <linux/ntb.h> +#include <linux/msi.h> static struct workqueue_struct *kpcintb_workqueue;
@@ -137,11 +138,14 @@ struct epf_ntb {struct epf_ntb_ctrl *reg;u32 *epf_db; +phys_addr_t epf_db_phys;phys_addr_t vpci_mw_phy[MAX_MW ];void __iomem *vpci_mw_addr[MAX_MW ];struct delayed_work cmd_handler; + +int msi_virqbase; };#define to_epf_ntb (epf_group) container_of((epf_group), struct epf_ntb, group)
@@ -256,11 +260,13 @@ static void epf_ntb_cmd_handler(struct work_struct *work) ntb = container_of(work, struct epf_ntb, cmd_handler.work); -for (i =1 ; i < ntb->db_count; i++) { -if (ntb->epf_db[i]) { - ntb->db |=1 << (i -1 ); -ntb_db_event (&ntb->ntb, i); - ntb->epf_db[i] =0 ; +if (!ntb->epf_db_phys) { +for (i =1 ; i < ntb->db_count; i++) { +if (ntb->epf_db[i]) { + ntb->db |=1 << (i -1 ); +ntb_db_event (&ntb->ntb, i); + ntb->epf_db[i] =0 ; + } } }
@@ -518,6 +524,28 @@ static int epf_ntb_configure_interrupt(struct epf_ntb *ntb)return 0 ; } +static int epf_ntb_db_size (struct epf_ntb *ntb) +{ +const struct pci_epc_features *epc_features; +size_t size =sizeof (u32) * ntb->db_count; +u32 align; + + epc_features =pci_epc_get_features (ntb->epf->epc, + ntb->epf->func_no, + ntb->epf->vfunc_no); + align = epc_features->align; + +if (size <128 ) + size =128 ; + +if (align) + size =ALIGN (size, align); +else + size =roundup_pow_of_two (size); + +return size; +} +/** * epf_ntb_db_bar_init() - Configure Doorbell window BARs * @ntb: NTB device that facilitates communication between HOST and VHOST
@@ -539,27 +567,26 @@ static int epf_ntb_db_bar_init(struct epf_ntb *ntb) ntb->epf->func_no, ntb->epf->vfunc_no); align = epc_features->align; - -if (size <128 ) - size =128 ; - -if (align) - size =ALIGN (size, align); -else - size =roundup_pow_of_two (size); + size =epf_ntb_db_size (ntb); barno = ntb->epf_ntb_bar[BAR_DB ]; + epf_bar = &ntb->epf->bar[barno]; - mw_addr =pci_epf_alloc_space (ntb->epf, size, barno, align,0 ); -if (!mw_addr) { -dev_err (dev,"Failed to allocate OB address\n" ); -return -ENOMEM ; +if (ntb->epf_db_phys) { + mw_addr =NULL ; + epf_bar->phys_addr = ntb->epf_db_phys; + epf_bar->barno = barno; + epf_bar->size = size; + }else { + mw_addr =pci_epf_alloc_space (ntb->epf, size, barno, align,0 ); +if (!mw_addr) { +dev_err (dev,"Failed to allocate doorbell address\n" ); +return -ENOMEM ; + } } ntb->epf_db = mw_addr; - epf_bar = &ntb->epf->bar[barno]; - ret =pci_epc_set_bar (ntb->epf->epc, ntb->epf->func_no, ntb->epf->vfunc_no, epf_bar);if (ret) {dev_err (dev,"Doorbell BAR set failed\n" );
@@ -728,6 +755,82 @@ static int epf_ntb_init_epc_bar(struct epf_ntb *ntb)return 0 ; } +static irqreturn_t epf_ntb_interrupt_handler (int irq,void *data) +{ +struct epf_ntb *ntb = data; +int index; + + index = irq - ntb->msi_virqbase; + ntb->db |=1 << (index -1 ); +ntb_db_event (&ntb->ntb, index); + +return IRQ_HANDLED ; +} + +#ifdef CONFIG_GENERIC_MSI_IRQ_DOMAIN +static void epf_ntb_write_msi_msg (struct msi_desc *desc,struct msi_msg *msg) +{ +struct epf_ntb *ntb =dev_get_drvdata (desc->dev); +struct epf_ntb_ctrl *reg = ntb->reg; +int size =epf_ntb_db_size (ntb); +u64 addr; + + addr = msg->address_hi; + addr <<=32 ; + addr |= msg->address_lo; + + reg->db_data[desc->msi_index] = msg->data; + +if (!desc->msi_index) + ntb->epf_db_phys =round_down (addr, size); + + reg->db_offset[desc->msi_index] = addr - ntb->epf_db_phys; +} + +static void epf_ntb_epc_msi_init (struct epf_ntb *ntb) +{ +struct device *dev = &ntb->epf->dev; +struct irq_domain *domain; +int virq; +int ret; +int i; + + domain =dev_get_msi_domain (ntb->epf->epc->dev.parent); +if (!domain) +return ; + +dev_set_msi_domain (dev, domain); + +if (platform_msi_domain_alloc_irqs (&ntb->epf->dev, + ntb->db_count, + epf_ntb_write_msi_msg)) { +dev_err (dev,"Can't allocate MSI, falling back to polling mode\n" ); +return ; + } +dev_dbg (dev,"Using MSI as doorbell\n" ); + +for (i =0 ; i < ntb->db_count; i++) { + virq =msi_get_virq (dev, i); + ret =devm_request_irq (dev, virq, + epf_ntb_interrupt_handler,0 , +"pci_epf_vntb" , ntb); + +if (ret) { +dev_err (dev,"Failed to request doorbell IRQ! Falling back to polling mode" ); + ntb->epf_db_phys =0 ; +platform_msi_domain_free_irqs (&ntb->epf->dev); +break ; + } + +if (!i) + ntb->msi_virqbase = virq;/* msi start virq number */ + } +} +#else +static void epf_ntb_epc_msi_init (struct epf_ntb *ntb) +{ +} +#endif /* CONFIG_GENERIC_MSI_IRQ_DOMAIN */ /** * epf_ntb_epc_init() - Initialize NTB interface * @ntb: NTB device that facilitates communication between HOST and VHOST
@@ -1336,14 +1439,15 @@ static int epf_ntb_bind(struct pci_epf *epf) goto err_bar_alloc; } +epf_set_drvdata (epf, ntb); +epf_ntb_epc_msi_init (ntb); + ret =epf_ntb_epc_init (ntb);if (ret) {dev_err (dev,"Failed to initialize EPC\n" ); goto err_bar_alloc; } -epf_set_drvdata (epf, ntb); - pci_space[0 ] = (ntb->vntb_pid <<16 ) | ntb->vntb_vid; pci_vntb_table[0 ].vendor = ntb->vntb_vid; pci_vntb_table[0 ].device = ntb->vntb_pid;
Signature · 1 line
2.34.1
Quoted 8 lines · Frank Li
┌────────────┐ ┌───────────────────────────────────┐ ┌────────────────┐ │ │ │ │ │ │ │ │ │ PCI Endpoint │ │ PCI Host │ │ │ │ │ │ │ │ │◄──┤ 1.platform_msi_domain_alloc_irqs()│ │ │ │ │ │ │ │ │ │ MSI ├──►│ 2.write_msi_msg() ├──►├─BAR<n> │ │ Controller │ │ update doorbell register address│ │ │
Quoted 9 lines
│ │ │ for BAR │ │ │ │ │ │ │ │ 3. Write BAR<n>│ │ │◄──┼───────────────────────────────────┼───┤ │ │ │ │ │ │ │ │ ├──►│ 4.Irq Handle │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ └────────────┘ └───────────────────────────────────┘ └────────────────┘
There are at least couple of BAR regions used in this patch but they were not mentioned in the above diagram.
The subject should be:
"PCI: endpoint: pci-epf-vntb: Use EP MSI controller to handle DB from host"
Quoted 2 lines
Using platform MSI interrupt controller as endpoint(EP)'s doorbell.
Above line is not needed.
Quoted 1 line
The memory assigned for BAR region by the PCI host is mapped to the
Which BAR? (BAR 1 aka. DB BAR)? There are multiple BAR regions exposed by this function driver.
Quoted 1 line
message address of platform msi interrupt controller in PCI Endpoint.
s/msi/MSI. Also, use either Endpoint or EP, pick one but not both.
Quoted 4 lines
Such that, whenever the PCI host writes to the BAR region, it will trigger an IRQ in the EP.
Basic working follow as
"work flow is"?
Quoted 1 line
1. EP function driver call
platform_msi_domain_alloc_irqs()alloc a
pci-epf-vntb function driver calls platform_msi_domain_alloc_irqs() to allocate MSI's from the platform MSI controller.
Quoted 2 lines
MSI irq from MSI controller with call back function
write_msi_msg();
2.write_msg_msgwill config BAR and map to address defined inmsi_msg;
The epf_ntb_write_msi_msg() passed as a callback will write the offset of the MSI controller's MSI address dedicated for each MSI to the doorbell register db_offset and also writes the MSI data to db_data register in the CTRL BAR region.
Quoted 2 lines
3. Host side trigger an IRQ at Endpoint by write to BAR region.
Finally, the host can trigger doorbell by reading the offset of the doorbell from db_offset register and writing the data read from db_data register in CTRL BAR region to the computed address in the DB BAR region.
Quoted 7 lines
Add MSI doorbell support for pci-epf-vntb. Query if system has an MSI controller. Set up doorbell address according to struct
msi_msg.So PCI host can write this doorbell address to trigger EP side's IRQ.
If no MSI controller exists, fall back to software polling.
"Add doorbell support to pci-epf-vntb function driver making use of the platform MSI controller. If the MSI controller is not available, fallback to the polling method."
Also, please move this paragraph to the beginning of the description.
Quoted 20 lines
Signed-off-by Frank Li <[email protected]>--- drivers/pci/endpoint/functions/pci-epf-vntb.c | 146 +++++++++++++++--- 1 file changed, 125 insertions(+), 21 deletions(-)drivers/pci/endpoint/functions/pci-epf-vntb.c +1
-0@@ -44,6 +44,7 @@#include <linux/pci-epc.h> #include <linux/pci-epf.h> #include <linux/ntb.h> +#include <linux/msi.h> static struct workqueue_struct *kpcintb_workqueue;@@ -137,11 +138,14 @@ struct epf_ntb {struct epf_ntb_ctrl *reg;
drivers/pci/endpoint/functions/pci-epf-vntb.c struct epf_ntb { · line 149
Quoted 10 lines
drivers/pci/endpoint/functions/pci-epf-vntb.c +3
-0u32 *epf_db; +phys_addr_t epf_db_phys;phys_addr_t vpci_mw_phy[MAX_MW ];void __iomem *vpci_mw_addr[MAX_MW ];struct delayed_work cmd_handler; + +int msi_virqbase; };
You should add kernel doc comments for this struct in a separate patch. It will help in understanding the driver better.
drivers/pci/endpoint/functions/pci-epf-vntb.c static void epf_ntb_cmd_handler(struct work_struct *work) · line 263
Quoted 11 lines
drivers/pci/endpoint/functions/pci-epf-vntb.c +0
-5#define to_epf_ntb (epf_group) container_of((epf_group), struct epf_ntb, group)@@ -256,11 +260,13 @@ static void epf_ntb_cmd_handler(struct work_struct *work) ntb = container_of(work, struct epf_ntb, cmd_handler.work); -for (i =1 ; i < ntb->db_count; i++) { -if (ntb->epf_db[i]) { - ntb->db |=1 << (i -1 ); -ntb_db_event (&ntb->ntb, i); - ntb->epf_db[i] =0 ;
A comment here stating that polling is implemented would be better.
Quoted 16 lines
drivers/pci/endpoint/functions/pci-epf-vntb.c +9
-0+if (!ntb->epf_db_phys) { +for (i =1 ; i < ntb->db_count; i++) { +if (ntb->epf_db[i]) { + ntb->db |=1 << (i -1 ); +ntb_db_event (&ntb->ntb, i); + ntb->epf_db[i] =0 ; + } } }@@ -518,6 +524,28 @@ static int epf_ntb_configure_interrupt(struct epf_ntb *ntb)return 0 ; } +static int epf_ntb_db_size (struct epf_ntb *ntb) +{
drivers/pci/endpoint/functions/pci-epf-vntb.c static int epf_ntb_configure_interrupt(struct epf_ntb *ntb) · line 538
Quoted 10 lines
drivers/pci/endpoint/functions/pci-epf-vntb.c +10
-0+const struct pci_epc_features *epc_features; +size_t size =sizeof (u32) * ntb->db_count; +u32 align; + + epc_features =pci_epc_get_features (ntb->epf->epc, + ntb->epf->func_no, + ntb->epf->vfunc_no); + align = epc_features->align; + +if (size <128 )
Shouldn't this be (size > 128)?
Quoted 55 lines
drivers/pci/endpoint/functions/pci-epf-vntb.c +23
-14+ size =128 ; + +if (align) + size =ALIGN (size, align); +else + size =roundup_pow_of_two (size); + +return size; +} +/** * epf_ntb_db_bar_init() - Configure Doorbell window BARs * @ntb: NTB device that facilitates communication between HOST and VHOST @@ -539,27 +567,26 @@ static int epf_ntb_db_bar_init(struct epf_ntb *ntb) ntb->epf->func_no, ntb->epf->vfunc_no); align = epc_features->align; - -if (size <128 ) - size =128 ; - -if (align) - size =ALIGN (size, align); -else - size =roundup_pow_of_two (size); + size =epf_ntb_db_size (ntb); barno = ntb->epf_ntb_bar[BAR_DB ]; + epf_bar = &ntb->epf->bar[barno]; - mw_addr =pci_epf_alloc_space (ntb->epf, size, barno, align,0 ); -if (!mw_addr) { -dev_err (dev,"Failed to allocate OB address\n" ); -return -ENOMEM ; +if (ntb->epf_db_phys) { + mw_addr =NULL ; + epf_bar->phys_addr = ntb->epf_db_phys; + epf_bar->barno = barno; + epf_bar->size = size; + }else { + mw_addr =pci_epf_alloc_space (ntb->epf, size, barno, align,0 ); +if (!mw_addr) { +dev_err (dev,"Failed to allocate doorbell address\n" ); +return -ENOMEM ; + } } ntb->epf_db = mw_addr; - epf_bar = &ntb->epf->bar[barno]; - ret =pci_epc_set_bar (ntb->epf->epc, ntb->epf->func_no, ntb->epf->vfunc_no, epf_bar);if (ret) {dev_err (dev,"Doorbell BAR set failed\n" );
drivers/pci/endpoint/functions/pci-epf-vntb.c static int epf_ntb_init_epc_bar(struct epf_ntb *ntb) · line 758
Quoted 4 lines
drivers/pci/endpoint/functions/pci-epf-vntb.c +1
-0@@ -728,6 +755,82 @@ static int epf_ntb_init_epc_bar(struct epf_ntb *ntb)return 0 ; } +static irqreturn_t epf_ntb_interrupt_handler (int irq,void *data)
Shouldn't this function also be guarded?
drivers/pci/endpoint/functions/pci-epf-vntb.c static int epf_ntb_init_epc_bar(struct epf_ntb *ntb) · line 770
Quoted 12 lines
drivers/pci/endpoint/functions/pci-epf-vntb.c +12
-0+{ +struct epf_ntb *ntb = data; +int index; + + index = irq - ntb->msi_virqbase; + ntb->db |=1 << (index -1 ); +ntb_db_event (&ntb->ntb, index); + +return IRQ_HANDLED ; +} + +#ifdef CONFIG_GENERIC_MSI_IRQ_DOMAIN
It'd be better to use the relevant commit description as a comment here.
drivers/pci/endpoint/functions/pci-epf-vntb.c static int epf_ntb_init_epc_bar(struct epf_ntb *ntb) · line 783
Quoted 13 lines
drivers/pci/endpoint/functions/pci-epf-vntb.c +13
-0+static void epf_ntb_write_msi_msg (struct msi_desc *desc,struct msi_msg *msg) +{ +struct epf_ntb *ntb =dev_get_drvdata (desc->dev); +struct epf_ntb_ctrl *reg = ntb->reg; +int size =epf_ntb_db_size (ntb); +u64 addr; + + addr = msg->address_hi; + addr <<=32 ; + addr |= msg->address_lo; + + reg->db_data[desc->msi_index] = msg->data; +
A comment stating that the base address to be used as the DB BAR is set here would be useful too.
Quoted 13 lines
drivers/pci/endpoint/functions/pci-epf-vntb.c +13
-0+if (!desc->msi_index) + ntb->epf_db_phys =round_down (addr, size); + + reg->db_offset[desc->msi_index] = addr - ntb->epf_db_phys; +} + +static void epf_ntb_epc_msi_init (struct epf_ntb *ntb) +{ +struct device *dev = &ntb->epf->dev; +struct irq_domain *domain; +int virq; +int ret; +int i;
drivers/pci/endpoint/functions/pci-epf-vntb.c static int epf_ntb_init_epc_bar(struct epf_ntb *ntb) · line 806
Quoted 10 lines
drivers/pci/endpoint/functions/pci-epf-vntb.c +10
-0+ + domain =dev_get_msi_domain (ntb->epf->epc->dev.parent); +if (!domain) +return ; + +dev_set_msi_domain (dev, domain); + +if (platform_msi_domain_alloc_irqs (&ntb->epf->dev, + ntb->db_count, + epf_ntb_write_msi_msg)) {
Please wrap above two in a single line till 100 column limit.
drivers/pci/endpoint/functions/pci-epf-vntb.c static int epf_ntb_init_epc_bar(struct epf_ntb *ntb) · line 807
Quoted 1 line
drivers/pci/endpoint/functions/pci-epf-vntb.c +1
-0+dev_err (dev,"Can't allocate MSI, falling back to polling mode\n" );
This should be dev_dbg().
drivers/pci/endpoint/functions/pci-epf-vntb.c static int epf_ntb_init_epc_bar(struct epf_ntb *ntb) · line 819
Quoted 12 lines
drivers/pci/endpoint/functions/pci-epf-vntb.c +12
-0+return ; + } +dev_dbg (dev,"Using MSI as doorbell\n" ); + +for (i =0 ; i < ntb->db_count; i++) { + virq =msi_get_virq (dev, i); + ret =devm_request_irq (dev, virq, + epf_ntb_interrupt_handler,0 , +"pci_epf_vntb" , ntb); + +if (ret) { +dev_err (dev,"Failed to request doorbell IRQ! Falling back to polling mode" );
Again, dev_dbg()
drivers/pci/endpoint/functions/pci-epf-vntb.c static int epf_ntb_init_epc_bar(struct epf_ntb *ntb) · line 829
Quoted 10 lines
drivers/pci/endpoint/functions/pci-epf-vntb.c +10
-0+ ntb->epf_db_phys =0 ; +platform_msi_domain_free_irqs (&ntb->epf->dev); + break; + } + +if (!i) + ntb->msi_virqbase = virq;/* msi start virq number */ + } +} +#else
Since this is not exposed as an API, just end the ifdef here and...
drivers/pci/endpoint/functions/pci-epf-vntb.c static int epf_ntb_bind(struct pci_epf *epf) · line 1443
Quoted 13 lines
drivers/pci/endpoint/functions/pci-epf-vntb.c +6
-0+static void epf_ntb_epc_msi_init (struct epf_ntb *ntb) +{ +} +#endif /* CONFIG_GENERIC_MSI_IRQ_DOMAIN */ /** * epf_ntb_epc_init() - Initialize NTB interface * @ntb: NTB device that facilitates communication between HOST and VHOST @@ -1336,14 +1439,15 @@ static int epf_ntb_bind(struct pci_epf *epf) goto err_bar_alloc; } +epf_set_drvdata (epf, ntb); +epf_ntb_epc_msi_init (ntb);
Guard this function instead:
#ifdef CONFIG_GENERIC_MSI_IRQ_DOMAIN epf_ntb_epc_msi_init(ntb);
#endif
Thanks,
Mani
Quoted 15 lines
drivers/pci/endpoint/functions/pci-epf-vntb.c +1
-2+ ret =epf_ntb_epc_init (ntb);if (ret) {dev_err (dev,"Failed to initialize EPC\n" ); goto err_bar_alloc; } -epf_set_drvdata (epf, ntb); - pci_space[0 ] = (ntb->vntb_pid <<16 ) | ntb->vntb_vid; pci_vntb_table[0 ].vendor = ntb->vntb_vid; pci_vntb_table[0 ].device = ntb->vntb_pid;Signature · 1 line
2.34.1
Signature · 1 line
மணிவண்ணன் சதாசிவம்
Frank Li
maintainerQuoted 46 lines
-----Original Message-----
From: Manivannan Sadhasivam <[email protected]>
Sent: Thursday, November 24, 2022 3:00 AM
To: Frank Li <[email protected]>
Cc: [email protected]; Aisheng Dong <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; linux- [email protected]; dl-linux-imx <[email protected]>; linux- [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Peng Fan <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Subject: [EXT] Re: [PATCH v13 2/2] PCI: endpoint: pci-epf-vntb: using platform MSI as doorbellCaution: EXT Email
Quoted 1 line · Frank Li
┌────────────┐ ┌───────────────
────────────────────┐ ┌───────── ───────┐ > │ │ │ │ │ │ > │ │ │ PCI Endpoint │ │ PCI Host │ > │ │ │ │ │ │ > │ │◄──┤ 1.platform_msi_domain_alloc_irqs()│ │ │ > │ │ │ │ │ │ > │ MSI ├──►│ 2.write_msi_msg() ├──►├─BAR<n> │ > │ Controller │ │ update doorbell register address│ │ │ > │ │ │ for BAR │ │ │ > │ │ │ │ │ 3. Write BAR<n>│ > │ │◄──┼────────────────────── ─────────────┼───┤ │ > │ │ │ │ │ │ > │ ├──►│ 4.Irq Handle │ │ │ > │ │ │ │ │ │ > │ │ │ │ │ │ > └────────────┘ └─────────────── ────────────────────┘ └───────── ───────┘Quoted 1 line
Quoted 2 lines
There are at least couple of BAR regions used in this patch but they were not mentioned in the above diagram.
This patch just affected one BAR regions. Do you like "BAR[DB]"?
Do you want to me draw other BARs, which used by this function?
Quoted 130 lines
The subject should be:
"PCI: endpoint: pci-epf-vntb: Use EP MSI controller to handle DB from host"
Quoted 2 lines
Using platform MSI interrupt controller as endpoint(EP)'s doorbell.
Above line is not needed.
Quoted 1 line
The memory assigned for BAR region by the PCI host is mapped to the
Which BAR? (BAR 1 aka. DB BAR)? There are multiple BAR regions exposed by this function driver.
Quoted 1 line
message address of platform msi interrupt controller in PCI Endpoint.
s/msi/MSI. Also, use either Endpoint or EP, pick one but not both.
Quoted 4 lines
Such that, whenever the PCI host writes to the BAR region, it will trigger an IRQ in the EP.
Basic working follow as
"work flow is"?
Quoted 1 line
1. EP function driver call
platform_msi_domain_alloc_irqs()alloc apci-epf-vntb function driver calls
platform_msi_domain_alloc_irqs()to allocate
MSI's from the platform MSI controller.Quoted 2 lines
MSI irq from MSI controller with call back function
write_msi_msg();
2.write_msg_msgwill config BAR and map to address defined inmsi_msg;The
epf_ntb_write_msi_msg()passed as a callback will write the offset of the MSI controller's MSI address dedicated for each MSI to the doorbell registerdb_offsetand also writes the MSI data todb_dataregister in the CTRL BAR region.Quoted 2 lines
3. Host side trigger an IRQ at Endpoint by write to BAR region.
Finally, the host can trigger doorbell by reading the offset of the doorbell from
db_offsetregister and writing the data read fromdb_dataregister in CTRL
BAR region to the computed address in the DB BAR region.Quoted 7 lines
Add MSI doorbell support for pci-epf-vntb. Query if system has an MSI controller. Set up doorbell address according to struct
msi_msg.So PCI host can write this doorbell address to trigger EP side's IRQ.
If no MSI controller exists, fall back to software polling.
"Add doorbell support to pci-epf-vntb function driver making use of the platform
MSI controller. If the MSI controller is not available, fallback to the polling method."Also, please move this paragraph to the beginning of the description.
Review
Quoted 6 lines
Signed-off-by Frank Li <[email protected]>--- drivers/pci/endpoint/functions/pci-epf-vntb.c | 146 +++++++++++++++--- 1 file changed, 125 insertions(+), 21 deletions(-)(quoted diff) +0
-0
b/drivers/pci/endpoint/functions/pci-epf-vntb.cQuoted 14 lines
index
0d744975f815..f770a068e58c100644drivers/pci/endpoint/functions/pci-epf-vntb.c +1
-0@@ -44,6 +44,7 @@#include <linux/pci-epc.h> #include <linux/pci-epf.h> #include <linux/ntb.h> +#include <linux/msi.h> static struct workqueue_struct *kpcintb_workqueue;@@ -137,11 +138,14 @@ struct epf_ntb {struct epf_ntb_ctrl *reg;drivers/pci/endpoint/functions/pci-epf-vntb.c struct epf_ntb { · line 149
Quoted 10 lines
drivers/pci/endpoint/functions/pci-epf-vntb.c +3
-0u32 *epf_db; +phys_addr_t epf_db_phys;phys_addr_t vpci_mw_phy[MAX_MW ];void __iomem *vpci_mw_addr[MAX_MW ];struct delayed_work cmd_handler; + +int msi_virqbase; };You should add kernel doc comments for this struct in a separate patch. It will
help in understanding the driver better.drivers/pci/endpoint/functions/pci-epf-vntb.c struct epf_ntb { · line 150
Quoted 2 lines
drivers/pci/endpoint/functions/pci-epf-vntb.c +0
-0#define to_epf_ntb (epf_group) container_of((epf_group), struct epf_ntb,group)
drivers/pci/endpoint/functions/pci-epf-vntb.c static void epf_ntb_cmd_handler(struct
Quoted 1 line
drivers/pci/endpoint/functions/pci-epf-vntb.c +0
-0@@ -256,11 +260,13 @@ static void epf_ntb_cmd_handler(struct
work_struct*work)drivers/pci/endpoint/functions/pci-epf-vntb.c static void epf_ntb_cmd_handler(struct work_struct *work) · line 263
Quoted 8 lines
drivers/pci/endpoint/functions/pci-epf-vntb.c +0
-5ntb = container_of(work, struct epf_ntb, cmd_handler.work); -for (i =1 ; i < ntb->db_count; i++) { -if (ntb->epf_db[i]) { - ntb->db |=1 << (i -1 ); -ntb_db_event (&ntb->ntb, i); - ntb->epf_db[i] =0 ;A comment here stating that polling is implemented would be better.
drivers/pci/endpoint/functions/pci-epf-vntb.c static int epf_ntb_configure_interrupt(struct
Quoted 11 lines
drivers/pci/endpoint/functions/pci-epf-vntb.c +7
-0+if (!ntb->epf_db_phys) { +for (i =1 ; i < ntb->db_count; i++) { +if (ntb->epf_db[i]) { + ntb->db |=1 << (i -1 ); +ntb_db_event (&ntb->ntb, i); + ntb->epf_db[i] =0 ; + } } }@@ -518,6 +524,28 @@ static int epf_ntb_configure_interrupt(struct
epf_ntb*ntb)Quoted 3 lines
drivers/pci/endpoint/functions/pci-epf-vntb.c +0
-0return 0 ; }
Quoted 14 lines
drivers/pci/endpoint/functions/pci-epf-vntb.c static int epf_ntb_configure_interrupt(struct epf_ntb *ntb) · line 538
Quoted 12 lines
drivers/pci/endpoint/functions/pci-epf-vntb.c +12
-0+static int epf_ntb_db_size (struct epf_ntb *ntb) +{ +const struct pci_epc_features *epc_features; +size_t size =sizeof (u32) * ntb->db_count; +u32 align; + + epc_features =pci_epc_get_features (ntb->epf->epc, + ntb->epf->func_no, + ntb->epf->vfunc_no); + align = epc_features->align; + +if (size <128 )Shouldn't this be (size > 128)?
This is one coming from pci-epf-ntb.c.
Not sure there are some EP hardware have such limitation.
Quoted 201 lines
Quoted 4 lines
drivers/pci/endpoint/functions/pci-epf-vntb.c +3
-0+ size =128 ; + +if (align)drivers/pci/endpoint/functions/pci-epf-vntb.c static int epf_ntb_configure_interrupt(struct epf_ntb *ntb)
Quoted 10 lines
drivers/pci/endpoint/functions/pci-epf-vntb.c +7
-0+ size =ALIGN (size, align); + else + size =roundup_pow_of_two (size); + +return size; +} +/** * epf_ntb_db_bar_init() - Configure Doorbell window BARs * @ntb: NTB device that facilitates communication between HOST and VHOST
drivers/pci/endpoint/functions/pci-epf-vntb.c static int epf_ntb_db_bar_init(struct epf_ntb
Quoted 1 line
drivers/pci/endpoint/functions/pci-epf-vntb.c +0
-0@@ -539,27 +567,26 @@ static int epf_ntb_db_bar_init(struct epf_ntb*ntb)
drivers/pci/endpoint/functions/pci-epf-vntb.c static int epf_ntb_init_epc_bar(struct epf_ntb
Quoted 42 lines
drivers/pci/endpoint/functions/pci-epf-vntb.c +13
-14ntb->epf->func_no, ntb->epf->vfunc_no); align = epc_features->align; - -if (size <128 ) - size =128 ; - -if (align) - size =ALIGN (size, align); -else - size =roundup_pow_of_two (size); + size =epf_ntb_db_size (ntb); barno = ntb->epf_ntb_bar[BAR_DB ]; + epf_bar = &ntb->epf->bar[barno]; - mw_addr =pci_epf_alloc_space (ntb->epf, size, barno, align,0 ); -if (!mw_addr) { -dev_err (dev,"Failed to allocate OB address\n" ); -return -ENOMEM ; +if (ntb->epf_db_phys) { + mw_addr =NULL ; + epf_bar->phys_addr = ntb->epf_db_phys; + epf_bar->barno = barno; + epf_bar->size = size; + }else { + mw_addr =pci_epf_alloc_space (ntb->epf, size, barno, align,0 ); +if (!mw_addr) { +dev_err (dev,"Failed to allocate doorbell address\n" ); +return -ENOMEM ; + } } ntb->epf_db = mw_addr; - epf_bar = &ntb->epf->bar[barno]; - ret =pci_epc_set_bar (ntb->epf->epc, ntb->epf->func_no, ntb->epf- vfunc_no, epf_bar);if (ret) {dev_err(dev, "Doorbell BAR set failed\n");drivers/pci/endpoint/functions/pci-epf-vntb.c +0
-0@@ -728,6 +755,82 @@ static int epf_ntb_init_epc_bar(struct epf_ntb*ntb)
drivers/pci/endpoint/functions/pci-epf-vntb.c static int epf_ntb_init_epc_bar(struct epf_ntb *ntb) · line 758
Quoted 4 lines
drivers/pci/endpoint/functions/pci-epf-vntb.c +1
-0return 0 ; } +static irqreturn_t epf_ntb_interrupt_handler (int irq,void *data)Shouldn't this function also be guarded?
drivers/pci/endpoint/functions/pci-epf-vntb.c static int epf_ntb_init_epc_bar(struct epf_ntb *ntb) · line 770
Quoted 12 lines
drivers/pci/endpoint/functions/pci-epf-vntb.c +12
-0+{ +struct epf_ntb *ntb = data; +int index; + + index = irq - ntb->msi_virqbase; + ntb->db |=1 << (index -1 ); +ntb_db_event (&ntb->ntb, index); + +return IRQ_HANDLED ; +} + +#ifdef CONFIG_GENERIC_MSI_IRQ_DOMAIN It'd be better to use the relevant commit description as a comment here.
drivers/pci/endpoint/functions/pci-epf-vntb.c static int epf_ntb_init_epc_bar(struct epf_ntb · line 771
Quoted 1 line
drivers/pci/endpoint/functions/pci-epf-vntb.c +1
-0+static void epf_ntb_write_msi_msg(struct msi_desc *desc, struct msi_msg*msg)
drivers/pci/endpoint/functions/pci-epf-vntb.c static int epf_ntb_init_epc_bar(struct epf_ntb *ntb) · line 783
Quoted 12 lines
drivers/pci/endpoint/functions/pci-epf-vntb.c +12
-0+{ +struct epf_ntb *ntb =dev_get_drvdata (desc->dev); +struct epf_ntb_ctrl *reg = ntb->reg; +int size =epf_ntb_db_size (ntb); +u64 addr; + + addr = msg->address_hi; + addr <<=32 ; + addr |= msg->address_lo; + + reg->db_data[desc->msi_index] = msg->data; +A comment stating that the base address to be used as the DB BAR is set here would be useful too.
Quoted 13 lines
drivers/pci/endpoint/functions/pci-epf-vntb.c +13
-0+if (!desc->msi_index) + ntb->epf_db_phys =round_down (addr, size); + + reg->db_offset[desc->msi_index] = addr - ntb->epf_db_phys; +} + +static void epf_ntb_epc_msi_init (struct epf_ntb *ntb) +{ +struct device *dev = &ntb->epf->dev; +struct irq_domain *domain; +int virq; +int ret; +int i;drivers/pci/endpoint/functions/pci-epf-vntb.c static int epf_ntb_init_epc_bar(struct epf_ntb *ntb) · line 806
Quoted 10 lines
drivers/pci/endpoint/functions/pci-epf-vntb.c +10
-0+ + domain =dev_get_msi_domain (ntb->epf->epc->dev.parent); +if (!domain) +return ; + +dev_set_msi_domain (dev, domain); + +if (platform_msi_domain_alloc_irqs (&ntb->epf->dev, + ntb->db_count, + epf_ntb_write_msi_msg)) {Please wrap above two in a single line till 100 column limit.
drivers/pci/endpoint/functions/pci-epf-vntb.c static int epf_ntb_init_epc_bar(struct epf_ntb *ntb) · line 807
Quoted 1 line
drivers/pci/endpoint/functions/pci-epf-vntb.c +1
-0+dev_err (dev,"Can't allocate MSI, falling back to polling mode\n" );This should be
dev_dbg().drivers/pci/endpoint/functions/pci-epf-vntb.c static int epf_ntb_init_epc_bar(struct epf_ntb *ntb)
Quoted 12 lines
drivers/pci/endpoint/functions/pci-epf-vntb.c +12
-0+return ; + } +dev_dbg (dev,"Using MSI as doorbell\n" ); + +for (i =0 ; i < ntb->db_count; i++) { + virq =msi_get_virq (dev, i); + ret =devm_request_irq (dev, virq, + epf_ntb_interrupt_handler,0 , +"pci_epf_vntb" , ntb); + +if (ret) { + dev_err(dev, "Failed to request doorbell IRQ! Falling back topolling mode");
Again,
dev_dbg()drivers/pci/endpoint/functions/pci-epf-vntb.c static int epf_ntb_init_epc_bar(struct epf_ntb *ntb) · line 829
Quoted 10 lines
drivers/pci/endpoint/functions/pci-epf-vntb.c +10
-0+ ntb->epf_db_phys =0 ; +platform_msi_domain_free_irqs (&ntb->epf->dev); + break; + } + +if (!i) + ntb->msi_virqbase = virq;/* msi start virq number */ + } +} +#else Since this is not exposed as an API, just end the ifdef here and...
drivers/pci/endpoint/functions/pci-epf-vntb.c static int epf_ntb_init_epc_bar(struct epf_ntb *ntb)
Quoted 7 lines
drivers/pci/endpoint/functions/pci-epf-vntb.c +4
-0+static void epf_ntb_epc_msi_init (struct epf_ntb *ntb) +{ +} +#endif /* CONFIG_GENERIC_MSI_IRQ_DOMAIN */ /** * epf_ntb_epc_init() - Initialize NTB interface * @ntb: NTB device that facilitates communication between HOST and VHOST
drivers/pci/endpoint/functions/pci-epf-vntb.c static int epf_ntb_bind(struct pci_epf *epf) · line 1443
Quoted 6 lines
drivers/pci/endpoint/functions/pci-epf-vntb.c +2
-0@@ -1336,14 +1439,15 @@ static int epf_ntb_bind(struct pci_epf *epf) goto err_bar_alloc; } +epf_set_drvdata (epf, ntb); +epf_ntb_epc_msi_init (ntb);Guard this function instead:
#ifdef CONFIG_GENERIC_MSI_IRQ_DOMAIN epf_ntb_epc_msi_init(ntb);#endif
Thanks,
ManiQuoted 13 lines
drivers/pci/endpoint/functions/pci-epf-vntb.c +1
-3+ ret =epf_ntb_epc_init (ntb);if (ret) {dev_err (dev,"Failed to initialize EPC\n" ); goto err_bar_alloc; } -epf_set_drvdata (epf, ntb); - pci_space[0 ] = (ntb->vntb_pid <<16 ) | ntb->vntb_vid; pci_vntb_table[0 ].vendor = ntb->vntb_vid; pci_vntb_table[0 ].device = ntb->vntb_pid; --Quoted 2 lines
2.34.1
--
மணிவண்ணன் சதாசிவம்
Quoted 46 lines · Frank Li
Quoted 46 lines
-----Original Message-----
From: Manivannan Sadhasivam <[email protected]>
Sent: Thursday, November 24, 2022 3:00 AM
To: Frank Li <[email protected]>
Cc: [email protected]; Aisheng Dong <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; linux- [email protected]; dl-linux-imx <[email protected]>; linux- [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Peng Fan <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Subject: [EXT] Re: [PATCH v13 2/2] PCI: endpoint: pci-epf-vntb: using platform MSI as doorbellCaution: EXT Email
Quoted 1 line · Frank Li
┌────────────┐ ┌───────────────
────────────────────┐ ┌───────── ───────┐ > │ │ │ │ │ │ > │ │ │ PCI Endpoint │ │ PCI Host │ > │ │ │ │ │ │ > │ │◄──┤ 1.platform_msi_domain_alloc_irqs()│ │ │ > │ │ │ │ │ │ > │ MSI ├──►│ 2.write_msi_msg() ├──►├─BAR<n> │ > │ Controller │ │ update doorbell register address│ │ │ > │ │ │ for BAR │ │ │ > │ │ │ │ │ 3. Write BAR<n>│ > │ │◄──┼────────────────────── ─────────────┼───┤ │ > │ │ │ │ │ │ > │ ├──►│ 4.Irq Handle │ │ │ > │ │ │ │ │ │ > │ │ │ │ │ │ > └────────────┘ └─────────────── ────────────────────┘ └───────── ───────┘Quoted 1 line
Quoted 7 lines
Quoted 2 lines
There are at least couple of BAR regions used in this patch but they were not mentioned in the above diagram.
This patch just affected one BAR regions. Do you like "BAR[DB]"?
Do you want to me draw other BARs, which used by this function?
It'd be good to just mention DB BAR.
Quoted 145 lines
Quoted 130 lines
The subject should be:
"PCI: endpoint: pci-epf-vntb: Use EP MSI controller to handle DB from host"
Quoted 2 lines
Using platform MSI interrupt controller as endpoint(EP)'s doorbell.
Above line is not needed.
Quoted 1 line
The memory assigned for BAR region by the PCI host is mapped to the
Which BAR? (BAR 1 aka. DB BAR)? There are multiple BAR regions exposed by this function driver.
Quoted 1 line
message address of platform msi interrupt controller in PCI Endpoint.
s/msi/MSI. Also, use either Endpoint or EP, pick one but not both.
Quoted 4 lines
Such that, whenever the PCI host writes to the BAR region, it will trigger an IRQ in the EP.
Basic working follow as
"work flow is"?
Quoted 1 line
1. EP function driver call
platform_msi_domain_alloc_irqs()alloc apci-epf-vntb function driver calls
platform_msi_domain_alloc_irqs()to allocate
MSI's from the platform MSI controller.Quoted 2 lines
MSI irq from MSI controller with call back function
write_msi_msg();
2.write_msg_msgwill config BAR and map to address defined inmsi_msg;The
epf_ntb_write_msi_msg()passed as a callback will write the offset of the MSI controller's MSI address dedicated for each MSI to the doorbell registerdb_offsetand also writes the MSI data todb_dataregister in the CTRL BAR region.Quoted 2 lines
3. Host side trigger an IRQ at Endpoint by write to BAR region.
Finally, the host can trigger doorbell by reading the offset of the doorbell from
db_offsetregister and writing the data read fromdb_dataregister in CTRL
BAR region to the computed address in the DB BAR region.Quoted 7 lines
Add MSI doorbell support for pci-epf-vntb. Query if system has an MSI controller. Set up doorbell address according to struct
msi_msg.So PCI host can write this doorbell address to trigger EP side's IRQ.
If no MSI controller exists, fall back to software polling.
"Add doorbell support to pci-epf-vntb function driver making use of the platform
MSI controller. If the MSI controller is not available, fallback to the polling method."Also, please move this paragraph to the beginning of the description.
Review
Quoted 6 lines
Signed-off-by Frank Li <[email protected]>--- drivers/pci/endpoint/functions/pci-epf-vntb.c | 146 +++++++++++++++--- 1 file changed, 125 insertions(+), 21 deletions(-)(quoted diff) +0
-0
b/drivers/pci/endpoint/functions/pci-epf-vntb.cQuoted 14 lines
index
0d744975f815..f770a068e58c100644drivers/pci/endpoint/functions/pci-epf-vntb.c +1
-0@@ -44,6 +44,7 @@#include <linux/pci-epc.h> #include <linux/pci-epf.h> #include <linux/ntb.h> +#include <linux/msi.h> static struct workqueue_struct *kpcintb_workqueue;@@ -137,11 +138,14 @@ struct epf_ntb {struct epf_ntb_ctrl *reg;drivers/pci/endpoint/functions/pci-epf-vntb.c struct epf_ntb { · line 149
Quoted 10 lines
drivers/pci/endpoint/functions/pci-epf-vntb.c +3
-0u32 *epf_db; +phys_addr_t epf_db_phys;phys_addr_t vpci_mw_phy[MAX_MW ];void __iomem *vpci_mw_addr[MAX_MW ];struct delayed_work cmd_handler; + +int msi_virqbase; };You should add kernel doc comments for this struct in a separate patch. It will
help in understanding the driver better.drivers/pci/endpoint/functions/pci-epf-vntb.c struct epf_ntb { · line 150
Quoted 2 lines
drivers/pci/endpoint/functions/pci-epf-vntb.c +0
-0#define to_epf_ntb (epf_group) container_of((epf_group), struct epf_ntb,group)
drivers/pci/endpoint/functions/pci-epf-vntb.c static void epf_ntb_cmd_handler(struct
Quoted 1 line
drivers/pci/endpoint/functions/pci-epf-vntb.c +0
-0@@ -256,11 +260,13 @@ static void epf_ntb_cmd_handler(struct
work_struct*work)drivers/pci/endpoint/functions/pci-epf-vntb.c static void epf_ntb_cmd_handler(struct work_struct *work) · line 263
Quoted 8 lines
drivers/pci/endpoint/functions/pci-epf-vntb.c +0
-5ntb = container_of(work, struct epf_ntb, cmd_handler.work); -for (i =1 ; i < ntb->db_count; i++) { -if (ntb->epf_db[i]) { - ntb->db |=1 << (i -1 ); -ntb_db_event (&ntb->ntb, i); - ntb->epf_db[i] =0 ;A comment here stating that polling is implemented would be better.
drivers/pci/endpoint/functions/pci-epf-vntb.c static int epf_ntb_configure_interrupt(struct
Quoted 11 lines
drivers/pci/endpoint/functions/pci-epf-vntb.c +7
-0+if (!ntb->epf_db_phys) { +for (i =1 ; i < ntb->db_count; i++) { +if (ntb->epf_db[i]) { + ntb->db |=1 << (i -1 ); +ntb_db_event (&ntb->ntb, i); + ntb->epf_db[i] =0 ; + } } }@@ -518,6 +524,28 @@ static int epf_ntb_configure_interrupt(struct
epf_ntb*ntb)Quoted 3 lines
drivers/pci/endpoint/functions/pci-epf-vntb.c +0
-0return 0 ; }Quoted 14 lines
drivers/pci/endpoint/functions/pci-epf-vntb.c static int epf_ntb_configure_interrupt(struct epf_ntb *ntb) · line 538
Quoted 12 lines
drivers/pci/endpoint/functions/pci-epf-vntb.c +12
-0+static int epf_ntb_db_size (struct epf_ntb *ntb) +{ +const struct pci_epc_features *epc_features; +size_t size =sizeof (u32) * ntb->db_count; +u32 align; + + epc_features =pci_epc_get_features (ntb->epf->epc, + ntb->epf->func_no, + ntb->epf->vfunc_no); + align = epc_features->align; + +if (size <128 )Shouldn't this be (size > 128)?
Quoted 3 lines
This is one coming from
pci-epf-ntb.c.
Not sure there are some EP hardware have such limitation.
I'm not sure if that is correct though. drivers/ntb/hw/epf/ntb_hw_epf.c sets the upper limit to 32 (NTB_EPF_MAX_DB_COUNT + 1) DBs, in that case the size cannot go beyond 128.
Thanks,
Mani
Signature · 1 line
மணிவண்ணன் சதாசிவம்
Frank Li
maintainerQuoted 241 lines
-----Original Message-----
From: Manivannan Sadhasivam <[email protected]>
Sent: Thursday, November 24, 2022 12:45 PM
To: Frank Li <[email protected]>
Cc: [email protected]; Aisheng Dong <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; linux- [email protected]; dl-linux-imx <[email protected]>; linux- [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Peng Fan <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [EXT] Re: [PATCH v13 2/2] PCI: endpoint: pci-epf-vntb: using platform MSI as doorbellCaution: EXT Email
Quoted 8 lines · Frank Li
Quoted 8 lines
-----Original Message-----
From: Manivannan Sadhasivam <[email protected]>
Sent: Thursday, November 24, 2022 3:00 AM
To: Frank Li <[email protected]>
Cc: [email protected]; Aisheng Dong <[email protected]>; [email protected]; [email protected];Quoted 3 lines
Quoted 3 lines
[email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; linux- [email protected]; dl-linux-imx <[email protected]>;
linux-
Quoted 5 lines
Quoted 5 lines
[email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Peng Fan <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Subject: [EXT] Re: [PATCH v13 2/2] PCI: endpoint: pci-epf-vntb: usingplatform
Quoted 6 lines
Quoted 6 lines
MSI as doorbell
Caution: EXT Email
Quoted 1 line · Frank Li
┌────────────┐ ┌──────────────
─ > > ────────────────────┐ ┌─────── ── > > ───────┐ > > > │ │ │ │ │ │ > > > │ │ │ PCI Endpoint │ │ PCI Host │ > > > │ │ │ │ │ │ > > > │ │◄──┤ 1.platform_msi_domain_alloc_irqs()│ │ > > │ > > > │ │ │ │ │ │ > > > │ MSI ├──►│ 2.write_msi_msg() ├──►├─BAR<n>
Quoted 5 lines
Quoted 5 lines
│ > │ Controller │ │ update doorbell register address│ │ │ > │ │ │ for BAR │ │ │ > │ │ │ │ │ 3. Write BAR<n>│ > │ │◄──┼────────────────────── > > ─────────────┼───┤ │ > > > │ │ │ │ │ │ > > > │ ├──►│ 4.Irq Handle │ │ │ > > > │ │ │ │ │ │ > > > │ │ │ │ │ │ > > > └────────────┘ └────────────── ─ > > ────────────────────┘ └─────── ── > > ───────┘Quoted 3 lines
Quoted 3 lines
Quoted 1 line
There are at least couple of BAR regions used in this patch but they were
not
Quoted 6 lines
Quoted 1 line
mentioned in the above diagram.
This patch just affected one BAR regions. Do you like "BAR[DB]"?
Do you want to me draw other BARs, which used by this function?
It'd be good to just mention DB BAR.
Quoted 4 lines
Quoted 4 lines
The subject should be:
"PCI: endpoint: pci-epf-vntb: Use EP MSI controller to handle DB from
host"
Quoted 9 lines
Quoted 9 lines
Quoted 3 lines
Using platform MSI interrupt controller as endpoint(EP)'s doorbell.
Above line is not needed.
Quoted 1 line
The memory assigned for BAR region by the PCI host is mapped to the
Which BAR? (BAR 1 aka. DB BAR)? There are multiple BAR regions
exposed by
Quoted 13 lines
Quoted 13 lines
this function driver.
Quoted 1 line
message address of platform msi interrupt controller in PCI Endpoint.
s/msi/MSI. Also, use either Endpoint or EP, pick one but not both.
Quoted 4 lines
Such that, whenever the PCI host writes to the BAR region, it will trigger an IRQ in the EP.
Basic working follow as
"work flow is"?
Quoted 8 lines
Quoted 8 lines
Quoted 1 line
1. EP function driver call
platform_msi_domain_alloc_irqs()alloc apci-epf-vntb function driver calls
platform_msi_domain_alloc_irqs()to allocate
MSI's from the platform MSI controller.Quoted 2 lines
MSI irq from MSI controller with call back function
write_msi_msg();
2.write_msg_msgwill config BAR and map to address defined in
msi_msg;Quoted 2 lines
Quoted 2 lines
The
epf_ntb_write_msi_msg()passed as a callback will write the offset ofthe
Quoted 1 line
Quoted 1 line
MSI controller's MSI address dedicated for each MSI to the doorbell
register
Quoted 1 line
Quoted 1 line
db_offsetand also writes the MSI data todb_dataregister in the CTRLBAR
Quoted 7 lines
Quoted 7 lines
region.
Quoted 2 lines
3. Host side trigger an IRQ at Endpoint by write to BAR region.
Finally, the host can trigger doorbell by reading the offset of the doorbell from
db_offsetregister and writing the data read fromdb_dataregisterin
Quoted 16 lines
Quoted 16 lines
CTRL
BAR region to the computed address in the DB BAR region.Quoted 7 lines
Add MSI doorbell support for pci-epf-vntb. Query if system has an MSI controller. Set up doorbell address according to struct
msi_msg.So PCI host can write this doorbell address to trigger EP side's IRQ.
If no MSI controller exists, fall back to software polling.
"Add doorbell support to pci-epf-vntb function driver making use of the platform
MSI controller. If the MSI controller is not available, fallback to the polling method."Quoted 5 lines
Quoted 5 lines
Also, please move this paragraph to the beginning of the description.
Quoted 3 lines
Signed-off-by Frank Li <[email protected]>--- drivers/pci/endpoint/functions/pci-epf-vntb.c | 146 +++++++++++++++---
Quoted 34 lines
Quoted 34 lines
Review
Quoted 3 lines
1 file changed, 125 insertions(+), 21 deletions(-)(quoted diff) +0
-0
b/drivers/pci/endpoint/functions/pci-epf-vntb.cQuoted 14 lines
index
0d744975f815..f770a068e58c100644drivers/pci/endpoint/functions/pci-epf-vntb.c +1
-0@@ -44,6 +44,7 @@#include <linux/pci-epc.h> #include <linux/pci-epf.h> #include <linux/ntb.h> +#include <linux/msi.h> static struct workqueue_struct *kpcintb_workqueue;@@ -137,11 +138,14 @@ struct epf_ntb {struct epf_ntb_ctrl *reg;drivers/pci/endpoint/functions/pci-epf-vntb.c struct epf_ntb { · line 149
Quoted 10 lines
drivers/pci/endpoint/functions/pci-epf-vntb.c +3
-0u32 *epf_db; +phys_addr_t epf_db_phys;phys_addr_t vpci_mw_phy[MAX_MW ];void __iomem *vpci_mw_addr[MAX_MW ];struct delayed_work cmd_handler; + +int msi_virqbase; };You should add kernel doc comments for this struct in a separate patch. It will
help in understanding the driver better.Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
#define
to_epf_ntb(epf_group)container_of((epf_group), struct
epf_ntb,Quoted 44 lines
Quoted 43 lines
group)
drivers/pci/endpoint/functions/pci-epf-vntb.c static void epf_ntb_cmd_handler(struct
Quoted 1 line
drivers/pci/endpoint/functions/pci-epf-vntb.c +0
-0@@ -256,11 +260,13 @@ static void epf_ntb_cmd_handler(struct
work_struct*work)drivers/pci/endpoint/functions/pci-epf-vntb.c static void epf_ntb_cmd_handler(struct work_struct *work) · line 263
Quoted 8 lines
drivers/pci/endpoint/functions/pci-epf-vntb.c +0
-5ntb = container_of(work, struct epf_ntb, cmd_handler.work); -for (i =1 ; i < ntb->db_count; i++) { -if (ntb->epf_db[i]) { - ntb->db |=1 << (i -1 ); -ntb_db_event (&ntb->ntb, i); - ntb->epf_db[i] =0 ;A comment here stating that polling is implemented would be better.
drivers/pci/endpoint/functions/pci-epf-vntb.c static int epf_ntb_configure_interrupt(struct
Quoted 11 lines
drivers/pci/endpoint/functions/pci-epf-vntb.c +7
-0+if (!ntb->epf_db_phys) { +for (i =1 ; i < ntb->db_count; i++) { +if (ntb->epf_db[i]) { + ntb->db |=1 << (i -1 ); +ntb_db_event (&ntb->ntb, i); + ntb->epf_db[i] =0 ; + } } }@@ -518,6 +524,28 @@ static int epf_ntb_configure_interrupt(struct
epf_ntb*ntb)Quoted 5 lines
drivers/pci/endpoint/functions/pci-epf-vntb.c +2
-0return 0 ; } +static int epf_ntb_db_size (struct epf_ntb *ntb) +{drivers/pci/endpoint/functions/pci-epf-vntb.c static int epf_ntb_configure_interrupt(struct epf_ntb *ntb) · line 538
Quoted 10 lines
drivers/pci/endpoint/functions/pci-epf-vntb.c +10
-0+const struct pci_epc_features *epc_features; +size_t size =sizeof (u32) * ntb->db_count; +u32 align; + + epc_features =pci_epc_get_features (ntb->epf->epc, + ntb->epf->func_no, + ntb->epf->vfunc_no); + align = epc_features->align; + +if (size <128 )Shouldn't this be (size > 128)?
Quoted 8 lines
Quoted 3 lines
This is one coming from
pci-epf-ntb.c.
Not sure there are some EP hardware have such limitation.I'm not sure if that is correct though.
drivers/ntb/hw/epf/ntb_hw_epf.csets the upper limit to 32 (NTB_EPF_MAX_DB_COUNT+ 1) DBs, in that case the size
cannot go beyond 128.
I don’t think so. Please checkdrivers/pci/endpoint/functions/pci-epf-ntb.c
Looks like some EP hardware have mini windows map size requirement.
I think it is not important for this patch.
Quoted 5 lines
Thanks,
Mani
--
மணிவண்ணன் சதாசிவம்
Quoted 17 lines · Frank Li
┌────────────┐ ┌───────────────────────────────────┐ ┌────────────────┐ │ │ │ │ │ │ │ │ │ PCI Endpoint │ │ PCI Host │ │ │ │ │ │ │ │ │◄──┤ 1.platform_msi_domain_alloc_irqs()│ │ │ │ │ │ │ │ │ │ MSI ├──►│ 2.write_msi_msg() ├──►├─BAR<n> │ │ Controller │ │ update doorbell register address│ │ │ │ │ │ for BAR │ │ │ │ │ │ │ │ 3. Write BAR<n>│ │ │◄──┼───────────────────────────────────┼───┤ │ │ │ │ │ │ │ │ ├──►│ 4.Irq Handle │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ └────────────┘ └───────────────────────────────────┘ └────────────────┘
Quoted 1 line
Using platform MSI interrupt controller as endpoint(EP)'s doorbell.
Can you please explain what the MSI controller is in this picture? MSI controller is not a term which is common in the interrupt handling landscape despite the fact that it's pretty wide spread in device tree bindings presumably through intensive copy & pasta cargo cult.
Quoted 5 lines
Basic working follow as
1. EP function driver callplatform_msi_domain_alloc_irqs()alloc a MSI irq from MSI controller with call back functionwrite_msi_msg();
2.write_msg_msgwill config BAR and map to address defined inmsi_msg;
3. Host side trigger an IRQ at Endpoint by write to BAR region.
You're explaining what the code does, but fail to explain the underlying mechanisms.
Platform MSI is definitely the wrong mechanism here. Why?
This is about a PCIe endpoint, which is usually handled by a PCI/MSI interrupt domain. Obviously this usage does not fit into the way how the "global" PCI/MSI domains work.
There is upcoming work and at least the generic parts should show up in 6.2 which addresses exactly the problem you are trying to solve:
https://lore.kernel.org/all/[email protected]
https://lore.kernel.org/all/[email protected]
plus the prove that the platform MSI mess can be replaced by this:
https://lore.kernel.org/all/[email protected]
NTB in it's current form should never have happened, but that's water down the bridge.
What you really want is:
1) Convert your platform to the new MSI parent model
2) Utilize PCI/IMS which is giving you exactly what you need with proper PCI semantics
Thanks,
tglx
Frank Li
maintainerQuoted 46 lines
-----Original Message-----
From: Thomas Gleixner <[email protected]>
Sent: Monday, November 28, 2022 1:15 PM
To: Frank Li <[email protected]>; [email protected]
Cc: Frank Li <[email protected]>; Aisheng Dong <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; linux- [email protected]; dl-linux-imx <[email protected]>; linux- [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Peng Fan <[email protected]>; [email protected]; [email protected]; [email protected]
Subject: [EXT] Re: [PATCH v13 2/2] PCI: endpoint: pci-epf-vntb: using platform MSI as doorbellCaution: EXT Email
Quoted 1 line · Frank Li
┌────────────┐ ┌───────────────
────────────────────┐ ┌───────── ───────┐ > │ │ │ │ │ │ > │ │ │ PCI Endpoint │ │ PCI Host │ > │ │ │ │ │ │ > │ │◄──┤ 1.platform_msi_domain_alloc_irqs()│ │ │ > │ │ │ │ │ │ > │ MSI ├──►│ 2.write_msi_msg() ├──►├─BAR<n> │ > │ Controller │ │ update doorbell register address│ │ │ > │ │ │ for BAR │ │ │ > │ │ │ │ │ 3. Write BAR<n>│ > │ │◄──┼────────────────────── ─────────────┼───┤ │ > │ │ │ │ │ │ > │ ├──►│ 4.Irq Handle │ │ │ > │ │ │ │ │ │ > │ │ │ │ │ │ > └────────────┘ └─────────────── ────────────────────┘ └───────── ───────┘Quoted 1 line
Quoted 6 lines
Quoted 1 line
Using platform MSI interrupt controller as endpoint(EP)'s doorbell.
Can you please explain what the MSI controller is in this picture? MSI controller is not a term which is common in the interrupt handling landscape despite the fact that it's pretty wide spread in device tree bindings presumably through intensive copy & pasta cargo cult.
I use irq-imx-mu-msi to do test. I supposed it should work for all kinds general msi controller. DTS part still not upstream yet because PCI EP enable patches still be under review.
https://lore.kernel.org/all/[email protected]/
Our test platform have not GIC ITS supported yet.
Quoted 11 lines
Quoted 6 lines
Basic working follow as
1. EP function driver callplatform_msi_domain_alloc_irqs()alloc a MSI irq from MSI controller with call back functionwrite_msi_msg();
2.write_msg_msgwill config BAR and map to address defined inmsi_msg;
3. Host side trigger an IRQ at Endpoint by write to BAR region.You're explaining what the code does, but fail to explain the underlying mechanisms.
Platform MSI is definitely the wrong mechanism here. Why?
This patch use Platform MSI. I never said " Platform MSI is definitely the wrong mechanism here".
Base logic is that, get msi controller's message address by irq API. Map this physical address to DB BAR, so PCI host write this DB bar, then EP generate irq.
You can refer v14 version, which should be better description.
Quoted 4 lines
This is about a PCIe endpoint, which is usually handled by a PCI/MSI interrupt domain. Obviously this usage does not fit into the way how the "global" PCI/MSI domains work.
PCI endpoint have not standard PCI configure space to enable/disable MSI irq and MSI address (CAP 05). That's reason why need "platform msi", or you called "global" Now.
Quoted 35 lines
There is upcoming work and at least the generic parts should show up in 6.2 which addresses exactly the problem you are trying to solve:
https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.k
ernel.org%2Fall%2F20221124225331.464480443%40linutronix.de&data =05%7C01%7CFrank.Li%40nxp.com%7C6a07e33e56af45ffc1ff08dad174d02d %7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C6380525969049530 06%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMz IiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Q8jr eVGGLa2M4yhjGO7Njqwdm59XDC0GyLEwkr0k6B0%3D&reserved=0https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.k
ernel.org%2Fall%2F20221124230505.073418677%40linutronix.de&data =05%7C01%7CFrank.Li%40nxp.com%7C6a07e33e56af45ffc1ff08dad174d02d %7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C6380525969049530 06%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMz IiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Tc9p XNJ499ETFgNWQBNLViFk8D5GbvrrwYDlBW%2Bf2qg%3D&reserved=0plus the prove that the platform MSI mess can be replaced by this:
https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.k
ernel.org%2Fall%2F20221121135653.208611233%40linutronix.de&data =05%7C01%7CFrank.Li%40nxp.com%7C6a07e33e56af45ffc1ff08dad174d02d %7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C6380525969049530 06%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMz IiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=R5K NVfcGqxoCam%2FYhY57ihsloWGhGLM3Kh9IkyME4lk%3D&reserved=0NTB in it's current form should never have happened, but that's water down the bridge.
Quoted 6 lines
What you really want is:
1) Convert your platform to the new MSI parent model
2) Utilize PCI/IMS which is giving you exactly what you need with proper PCI semantics
Sorry, I still don't understand yet. This patch is just user of msi controller. Your patches focus on the msi controller itself.
Interface platform_msi_domain_alloc_irqs still exist at your devmsi-v2-part3.
Quoted 4 lines
Thanks,
tglx
Frank!
On Mon, Nov 28 2022 at 21:25, Frank Li wrote:
Can you please fix your mail client to not copy the whole CC list into the reply? It's just pointless noise. A simple:
On Mon, Nov 28 200 at 1:15 PM Thomas Gleixner wrote:
instead of:
Quoted 8 lines
Quoted 8 lines
-----Original Message-----
From: Thomas Gleixner <[email protected]>
Sent: Monday, November 28, 2022 1:15 PM
To: Frank Li <[email protected]>; [email protected]
Cc: Frank Li <[email protected]>; Aisheng Dong <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; linux-
Quoted 8 lines
Quoted 8 lines
[email protected]; dl-linux-imx <[email protected]>; linux- [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Peng Fan <[email protected]>; [email protected]; [email protected]; [email protected]
Subject: [EXT] Re: [PATCH v13 2/2] PCI: endpoint: pci-epf-vntb: using platform MSI as doorbell
is completely sufficient.
Quoted 1 line
Quoted 1 line
Caution: EXT Email
We are neither interested in the oddities of NXP's mail categorization system.
Quoted 11 lines
Quoted 8 lines · Frank Li
Quoted 2 lines
Using platform MSI interrupt controller as endpoint(EP)'s doorbell.
Can you please explain what the MSI controller is in this picture? MSI controller is not a term which is common in the interrupt handling landscape despite the fact that it's pretty wide spread in device tree bindings presumably through intensive copy & pasta cargo cult.
I use irq-imx-mu-msi to do test. I supposed it should work for all kinds general msi controller.
Sure it works by some definition of "works" because obviously that implementation does not care about where a particular message originates from.
But that's still wrong at the conceptual level because it very much matters where a message originates from. PCIe devices and general platform devices have very distinct mechanisms to transport that information.
Just because it "works" does not prove that it is correct.
How are you going to do proper shielding with that approach?
Quoted 1 line
Our test platform have not GIC ITS supported yet.
And therefore the originating device is irrelevant, right? Get to the point where you have ITS and it all falls apart.
Quoted 7 lines
Quoted 4 lines
You're explaining what the code does, but fail to explain the underlying mechanisms.
Platform MSI is definitely the wrong mechanism here. Why?
This patch use Platform MSI. I never said " Platform MSI is definitely the wrong mechanism here".
I did not claim that _you_ said that. _I_ said that this is wrong. See above.
Quoted 3 lines
Base logic is that, get msi controller's message address by irq API. Map this physical address to DB BAR, so PCI host write this DB bar, then EP generate irq.
Again, you are explaining what your implementation is doing, but you are not describing the conceptual level.
Quoted 6 lines
Quoted 3 lines
This is about a PCIe endpoint, which is usually handled by a PCI/MSI interrupt domain. Obviously this usage does not fit into the way how the "global" PCI/MSI domains work.
PCI endpoint have not standard PCI configure space to enable/disable MSI irq and MSI address (CAP 05).
I'm very well aware of the fact that a PCI endpoint does not have the standard MSI configuration space mechanism.
Quoted 1 line
That's reason why need "platform msi", or you called "global"
Again: platform MSI does not convey the PCIe device originator. It might be irrelevant for your actual platform, but that does not make it more correct. Once you have the need to track the origin of a MSI message then the distinction between platform and MSI matters.
Sure you can argue that you don't care, but that does neither make it correct nor future proof and there is no point to rework this whole thing 6 month down the road when you actually have to support GIC-ITS.
Quoted 21 lines
Quoted 21 lines
There is upcoming work and at least the generic parts should show up in 6.2 which addresses exactly the problem you are trying to solve:
https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.k
ernel.org%2Fall%2F20221124225331.464480443%40linutronix.de&data =05%7C01%7CFrank.Li%40nxp.com%7C6a07e33e56af45ffc1ff08dad174d02d %7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C6380525969049530 06%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMz IiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Q8jr eVGGLa2M4yhjGO7Njqwdm59XDC0GyLEwkr0k6B0%3D&reserved=0https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.k
ernel.org%2Fall%2F20221124230505.073418677%40linutronix.de&data =05%7C01%7CFrank.Li%40nxp.com%7C6a07e33e56af45ffc1ff08dad174d02d %7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C6380525969049530 06%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMz IiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Tc9p XNJ499ETFgNWQBNLViFk8D5GbvrrwYDlBW%2Bf2qg%3D&reserved=0plus the prove that the platform MSI mess can be replaced by this:
Quoted 7 lines
Quoted 7 lines
https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.k
ernel.org%2Fall%2F20221121135653.208611233%40linutronix.de&data =05%7C01%7CFrank.Li%40nxp.com%7C6a07e33e56af45ffc1ff08dad174d02d %7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C6380525969049530 06%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMz IiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=R5K NVfcGqxoCam%2FYhY57ihsloWGhGLM3Kh9IkyME4lk%3D&reserved=0
Those outlook artifacts are helpful to our conversation in which way?
Quoted 5 lines
Quoted 5 lines
NTB in it's current form should never have happened, but that's water down the bridge.
What you really want is:
Quoted 7 lines
Quoted 4 lines
1) Convert your platform to the new MSI parent model
2) Utilize PCI/IMS which is giving you exactly what you need with proper PCI semantics
Sorry, I still don't understand yet. This patch is just user of msi controller.
As I explained to you before: The concept of MSI controller does not exist in the kernel. It might exist in your NXP nomenclature, but that's irrelevant here.
Quoted 1 line
Your patches focus on the msi controller itself.
No. They focus on changing the hierarchy model from "global" MSI domains to per device MSI domains so that the weird constructs of platform MSI can be replaced by something which actually matches the hardware and provides a proper abstraction for PCIe/NTB at the right level.
Quoted 1 line
Interface
platform_msi_domain_alloc_irqsstill exist at your devmsi-v2-part3.
Sure, but at:
git://git.kernel.org/pub/scm/linux/kernel/git/tglx/devel.git devmsi-v2-arm
platform_msi_domain_alloc_irqs() does not exist anymore.
You replied to exactly that series which builds on top of devmsi-v2-part3, no?
So what are you trying to tell me?
Thanks,
tglx
Frank Li
maintainerQuoted 41 lines · Frank Li
Quoted 1 line
┌────────────┐ ┌───────────────
────────────────────┐ ┌──────── ────────┐ > │ │ │ │ │ │ > │ │ │ PCI Endpoint │ │ PCI Host │ > │ │ │ │ │ │ > │ │◄──┤ 1.platform_msi_domain_alloc_irqs()│ │ │ > │ │ │ │ │ │ > │ MSI ├──►│ 2.write_msi_msg() ├──►├─BAR<n> │ > │ Controller │ │ update doorbell register address│ │ │ > │ │ │ for BAR │ │ │ > │ │ │ │ │ 3. Write BAR<n>│ > │ │◄──┼────────────────────── ─────────────┼───┤ │ > │ │ │ │ │ │ > │ ├──►│ 4.Irq Handle │ │ │ > │ │ │ │ │ │ > │ │ │ │ │ │ > └────────────┘ └─────────────── ────────────────────┘ └──────── ────────┘Quoted 2 lines
Using platform MSI interrupt controller as endpoint(EP)'s doorbell.
Can you please explain what the MSI controller is in this picture? MSI controller is not a term which is common in the interrupt handling landscape despite the fact that it's pretty wide spread in device tree bindings presumably through intensive copy & pasta cargo cult.
Quoted 5 lines
Basic working follow as
1. EP function driver callplatform_msi_domain_alloc_irqs()alloc a MSI irq from MSI controller with call back functionwrite_msi_msg();
2.write_msg_msgwill config BAR and map to address defined inmsi_msg;
3. Host side trigger an IRQ at Endpoint by write to BAR region.You're explaining what the code does, but fail to explain the underlying mechanisms.
Quoted 9 lines
Platform MSI is definitely the wrong mechanism here. Why?
This is about a PCIe endpoint, which is usually handled by a PCI/MSI interrupt domain. Obviously this usage does not fit into the way how the "global" PCI/MSI domains work.
There is upcoming work and at least the generic parts should show up in 6.2 which addresses exactly the problem you are trying to solve:
@tglx
I finally get a platform, ls1028, which use gic its and also support PCIe EP function. I port my code into linux-next, (next-20230105) and understand your comments.
Its need a device ID to allocate msi irq for each devices.
Epf device is dynamic created, where are field epc, epc.parent point to PCIe EP Controller, which have been probed by device-tree.
I use a workaround to get msi irq number.
dev->of_node= ntb->efp->epc->dev.parent->of_node; Platform_msi_domain_alloc_irqs(dev, ...);
Irq-gic-v3-its-pci-msi use of_node to get a ID.
You said PCI/IMS can resolve this problem. But I still not figure out it yet. Can you give me sample code, which use PCI/IMS or provide a pseudo code?
Frank
Quoted 41 lines
https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.k
ernel.org%2Fall%2F20221124225331.464480443%40linutronix.de&data=0 5%7C01%7CFrank.Li%40nxp.com%7C6a07e33e56af45ffc1ff08dad174d02d%7C 686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C638052596904953006%7C Unknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJB TiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Q8jreVGG La2M4yhjGO7Njqwdm59XDC0GyLEwkr0k6B0%3D&reserved=0https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.k
ernel.org%2Fall%2F20221124230505.073418677%40linutronix.de&data=0 5%7C01%7CFrank.Li%40nxp.com%7C6a07e33e56af45ffc1ff08dad174d02d%7C 686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C638052596904953006%7C Unknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJB TiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Tc9pXNJ49 9ETFgNWQBNLViFk8D5GbvrrwYDlBW%2Bf2qg%3D&reserved=0plus the prove that the platform MSI mess can be replaced by this:
https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.k
ernel.org%2Fall%2F20221121135653.208611233%40linutronix.de&data=0 5%7C01%7CFrank.Li%40nxp.com%7C6a07e33e56af45ffc1ff08dad174d02d%7C 686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C638052596904953006%7C Unknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJB TiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=R5KNVfcG qxoCam%2FYhY57ihsloWGhGLM3Kh9IkyME4lk%3D&reserved=0NTB in it's current form should never have happened, but that's water down the bridge.
What you really want is:
1) Convert your platform to the new MSI parent model
2) Utilize PCI/IMS which is giving you exactly what you need with proper PCI semantics
Thanks,
tglx