PCI: endpoint: set prefetchable bit for 64-bit BARs
Subsystems (6)
v3 cover PCI: endpoint: set prefetchable bit for 64-bit BARs
Shradha Todi wanted to add prefetchable BAR support by adding more things to epc_features:
https://lore.kernel.org/linux-pci/[email protected]/T/#t
The series starts off with some generic cleanups and fixes which was needed to make the implementation of the actual feature easier.
The final patch in the series sets the prefetchable bit for all backing memory that the EPF core allocates for a 64-bit BAR.
Kind regards,
Niklas
Changes since V2:
-Fixed warning when building with W=1 reported by kernel test robot.
Niklas Cassel (9):
PCI: endpoint: pci-epf-test: Fix incorrect loop increment
PCI: endpoint: Allocate a 64-bit BAR if that is the only option
PCI: endpoint: pci-epf-test: Remove superfluous code
PCI: endpoint: pci-epf-test: Simplify pci_epf_test_alloc_space() loop
PCI: endpoint: pci-epf-test: Simplify pci_epf_test_set_bar() loop
PCI: endpoint: pci-epf-test: Clean up pci_epf_test_unbind()
PCI: cadence: Set a 64-bit BAR if requested
PCI: rockchip-ep: Set a 64-bit BAR if requested
PCI: endpoint: Set prefetch when allocating memory for 64-bit BARs
.../pci/controller/cadence/pcie-cadence-ep.c | 5 +- drivers/pci/controller/pcie-rockchip-ep.c | 2 +- drivers/pci/endpoint/functions/pci-epf-test.c | 70 +++++++------------ drivers/pci/endpoint/pci-epf-core.c | 10 ++- 4 files changed, 33 insertions(+), 54 deletions(-)
Signature · 1 line
2.44.0
PATCH v3 1/9 PCI: endpoint: pci-epf-test: Fix incorrect loop increment
pci_epf_test_alloc_space() currently skips the BAR succeeding a 64-bit BAR if the 64-bit flag is set, before calling pci_epf_alloc_space().
However, pci_epf_alloc_space() will set the 64-bit flag if we request an allocation of 4 GB or larger, even if it wasn't set before the allocation.
Thus, we need to check the flag also after pci_epf_alloc_space().
--- drivers/pci/endpoint/functions/pci-epf-test.c | 6 ++++++ 1 file changed, 6 insertions(+)
drivers/pci/endpoint/functions/pci-epf-test.c +6 -0
@@ -865,6 +865,12 @@ static int pci_epf_test_alloc_space(struct pci_epf *epf)dev_err (dev,"Failed to allocate space for BAR%d\n" , bar); epf_test->reg[bar] = base; + +/* +* pci_epf_alloc_space() might have given us a 64-bit BAR, +* if we requested a size larger than 4 GB. +*/ + add = (epf_bar->flags &PCI_BASE_ADDRESS_MEM_TYPE_64 ) ?2 :1 ; }return 0 ;
Signature · 1 line
2.44.0
Re: [PATCH v3 1/9] PCI: endpoint: pci-epf-test: Fix incorrect loop increment
Quoted 6 lines · Niklas Cassel
pci_epf_test_alloc_space()currently skips the BAR succeeding a 64-bit BAR if the 64-bit flag is set, before callingpci_epf_alloc_space().However,
pci_epf_alloc_space()will set the 64-bit flag if we request an allocation of 4 GB or larger, even if it wasn't set before the allocation.
Max BAR size is 1MB currently, so I believe you are adding the check just for the sake of correctness. If so, please mention it explicitly.
- Mani
Quoted 27 lines
Thus, we need to check the flag also after
pci_epf_alloc_space().Signed-off-by Niklas Cassel <[email protected]>--- drivers/pci/endpoint/functions/pci-epf-test.c | 6 ++++++ 1 file changed, 6 insertions(+)drivers/pci/endpoint/functions/pci-epf-test.c +6
-0@@ -865,6 +865,12 @@ static int pci_epf_test_alloc_space(struct pci_epf *epf)dev_err (dev,"Failed to allocate space for BAR%d\n" , bar); epf_test->reg[bar] = base; + +/* +* pci_epf_alloc_space() might have given us a 64-bit BAR, +* if we requested a size larger than 4 GB. +*/ + add = (epf_bar->flags &PCI_BASE_ADDRESS_MEM_TYPE_64 ) ?2 :1 ; }return 0 ;Signature · 1 line
2.44.0
Signature · 1 line
மணிவண்ணன் சதாசிவம்
Re: [PATCH v3 1/9] PCI: endpoint: pci-epf-test: Fix incorrect loop increment
Quoted 10 lines · Manivannan Sadhasivam
Quoted 6 lines · Niklas Cassel
pci_epf_test_alloc_space()currently skips the BAR succeeding a 64-bit BAR if the 64-bit flag is set, before callingpci_epf_alloc_space().However,
pci_epf_alloc_space()will set the 64-bit flag if we request an allocation of 4 GB or larger, even if it wasn't set before the allocation.Max BAR size is 1MB currently, so I believe you are adding the check just for the sake of correctness. If so, please mention it explicitly.
The BAR size defined in:
static size_t bar_size[] = { 512, 512, 1024, 16384, 131072, 1048576 }; will only be used if the BAR is not fixed size.
(A fixed BAR size will override the size in static size_t bar_size[])
So a platform could have a fixed size BAR of e.g. 4GB.
Although, you could argue that if a platform has a fixed BAR size of 4GB, they should have also marked the BAR as "only 64-bit". (Considering that the largest size supported by a 32-bit BAR is 2GB.)
I will drop this patch in V4.
Kind regards,
Niklas
Quoted 33 lines
- Mani
Quoted 27 lines
Thus, we need to check the flag also after
pci_epf_alloc_space().Signed-off-by Niklas Cassel <[email protected]>--- drivers/pci/endpoint/functions/pci-epf-test.c | 6 ++++++ 1 file changed, 6 insertions(+)drivers/pci/endpoint/functions/pci-epf-test.c +6
-0@@ -865,6 +865,12 @@ static int pci_epf_test_alloc_space(struct pci_epf *epf)dev_err (dev,"Failed to allocate space for BAR%d\n" , bar); epf_test->reg[bar] = base; + +/* +* pci_epf_alloc_space() might have given us a 64-bit BAR, +* if we requested a size larger than 4 GB. +*/ + add = (epf_bar->flags &PCI_BASE_ADDRESS_MEM_TYPE_64 ) ?2 :1 ; }return 0 ;Signature · 1 line
2.44.0Signature · 1 line
மணிவண்ணன் சதாசிவம்
PATCH v3 2/9 PCI: endpoint: Allocate a 64-bit BAR if that is the only option
In reply to Niklas Casselpci_epf_alloc_space() already sets the 64-bit flag if the BAR size is larger than 4GB, even if the caller did not explicitly request a 64-bit BAR.
Thus, let pci_epf_alloc_space() also set the 64-bit flag if the hardware description says that the specific BAR can only be 64-bit.
--- drivers/pci/endpoint/functions/pci-epf-test.c | 3 ++- drivers/pci/endpoint/pci-epf-core.c | 7 ++++--- 2 files changed, 6 insertions(+), 4 deletions(-)
drivers/pci/endpoint/functions/pci-epf-test.c +2 -1
@@ -868,7 +868,8 @@ static int pci_epf_test_alloc_space(struct pci_epf *epf)/* * pci_epf_alloc_space() might have given us a 64-bit BAR, -* if we requested a size larger than 4 GB. +* either because the BAR can only be a 64-bit BAR, or if +* we requested a size larger than 4 GB. */ add = (epf_bar->flags &PCI_BASE_ADDRESS_MEM_TYPE_64 ) ?2 :1 ; }
drivers/pci/endpoint/pci-epf-core.c +4 -3
@@ -304,9 +304,10 @@ void *pci_epf_alloc_space(struct pci_epf *epf, size_t size, enum pci_barno bar, epf_bar[bar].addr = space; epf_bar[bar].size = size; epf_bar[bar].barno = bar; - epf_bar[bar].flags |=upper_32_bits (size) ? -PCI_BASE_ADDRESS_MEM_TYPE_64 : -PCI_BASE_ADDRESS_MEM_TYPE_32 ; +if (upper_32_bits (size) || epc_features->bar[bar].only_64bit) + epf_bar[bar].flags |=PCI_BASE_ADDRESS_MEM_TYPE_64 ; +else + epf_bar[bar].flags |=PCI_BASE_ADDRESS_MEM_TYPE_32 ;return space; }
Signature · 1 line
2.44.0
Re: [PATCH v3 2/9] PCI: endpoint: Allocate a 64-bit BAR if that is the only option
Quoted 7 lines · Niklas Cassel
pci_epf_alloc_space()already sets the 64-bit flag if the BAR size is larger than 4GB, even if the caller did not explicitly request a 64-bit BAR.Thus, let
pci_epf_alloc_space()also set the 64-bit flag if the hardware description says that the specific BAR can only be 64-bit.
Could you please update the kdoc of pci_epf_alloc_space() to reflec the same?
Quoted 1 line
Signed-off-by Niklas Cassel <[email protected]>
With that,
- Mani
Quoted 40 lines
--- drivers/pci/endpoint/functions/pci-epf-test.c | 3 ++- drivers/pci/endpoint/pci-epf-core.c | 7 ++++--- 2 files changed, 6 insertions(+), 4 deletions(-)drivers/pci/endpoint/functions/pci-epf-test.c +2
-1@@ -868,7 +868,8 @@ static int pci_epf_test_alloc_space(struct pci_epf *epf)/* * pci_epf_alloc_space() might have given us a 64-bit BAR, -* if we requested a size larger than 4 GB. +* either because the BAR can only be a 64-bit BAR, or if +* we requested a size larger than 4 GB. */ add = (epf_bar->flags &PCI_BASE_ADDRESS_MEM_TYPE_64 ) ?2 :1 ; }drivers/pci/endpoint/pci-epf-core.c +4
-3@@ -304,9 +304,10 @@ void *pci_epf_alloc_space(struct pci_epf *epf, size_t size, enum pci_barno bar, epf_bar[bar].addr = space; epf_bar[bar].size = size; epf_bar[bar].barno = bar; - epf_bar[bar].flags |=upper_32_bits (size) ? -PCI_BASE_ADDRESS_MEM_TYPE_64 : -PCI_BASE_ADDRESS_MEM_TYPE_32 ; +if (upper_32_bits (size) || epc_features->bar[bar].only_64bit) + epf_bar[bar].flags |=PCI_BASE_ADDRESS_MEM_TYPE_64 ; +else + epf_bar[bar].flags |=PCI_BASE_ADDRESS_MEM_TYPE_32 ;return space; }Signature · 1 line
2.44.0
Signature · 1 line
மணிவண்ணன் சதாசிவம்
PATCH v3 3/9 PCI: endpoint: pci-epf-test: Remove superfluous code
In reply to Niklas CasselThe pci-epf-test does no special configuration at all, it simply requests a 64-bit BAR if the hardware requires it. However, this flag is now automatically set when allocating a BAR that can only be a 64-bit BAR, so we can drop pci_epf_configure_bar() completely.
--- drivers/pci/endpoint/functions/pci-epf-test.c | 14 -------------- 1 file changed, 14 deletions(-)
drivers/pci/endpoint/functions/pci-epf-test.c +0 -14
@@ -877,19 +877,6 @@ static int pci_epf_test_alloc_space(struct pci_epf *epf)return 0 ; } -static void pci_epf_configure_bar (struct pci_epf *epf, -const struct pci_epc_features *epc_features) -{ -struct pci_epf_bar *epf_bar; -int i; - -for (i =0 ; i <PCI_STD_NUM_BARS ; i++) { - epf_bar = &epf->bar[i]; -if (epc_features->bar[i].only_64bit) - epf_bar->flags |=PCI_BASE_ADDRESS_MEM_TYPE_64 ; - } -} -static int pci_epf_test_bind (struct pci_epf *epf) {int ret;
@@ -914,7 +901,6 @@ static int pci_epf_test_bind(struct pci_epf *epf) test_reg_bar =pci_epc_get_first_free_bar (epc_features);if (test_reg_bar <0 )return -EINVAL ; -pci_epf_configure_bar (epf, epc_features); epf_test->test_reg_bar = test_reg_bar; epf_test->epc_features = epc_features;
Signature · 1 line
2.44.0
Re: [PATCH v3 3/9] PCI: endpoint: pci-epf-test: Remove superfluous code
Quoted 6 lines · Niklas Cassel
The pci-epf-test does no special configuration at all, it simply requests a 64-bit BAR if the hardware requires it. However, this flag is now automatically set when allocating a BAR that can only be a 64-bit BAR, so we can drop
pci_epf_configure_bar()completely.Signed-off-by Niklas Cassel <[email protected]>
- Mani
Quoted 39 lines
--- drivers/pci/endpoint/functions/pci-epf-test.c | 14 -------------- 1 file changed, 14 deletions(-)drivers/pci/endpoint/functions/pci-epf-test.c +0
-14@@ -877,19 +877,6 @@ static int pci_epf_test_alloc_space(struct pci_epf *epf)return 0 ; } -static void pci_epf_configure_bar (struct pci_epf *epf, -const struct pci_epc_features *epc_features) -{ -struct pci_epf_bar *epf_bar; -int i; - -for (i =0 ; i <PCI_STD_NUM_BARS ; i++) { - epf_bar = &epf->bar[i]; -if (epc_features->bar[i].only_64bit) - epf_bar->flags |=PCI_BASE_ADDRESS_MEM_TYPE_64 ; - } -} -static int pci_epf_test_bind (struct pci_epf *epf) {int ret;@@ -914,7 +901,6 @@ static int pci_epf_test_bind(struct pci_epf *epf) test_reg_bar =pci_epc_get_first_free_bar (epc_features);if (test_reg_bar <0 )return -EINVAL ; -pci_epf_configure_bar (epf, epc_features); epf_test->test_reg_bar = test_reg_bar; epf_test->epc_features = epc_features;Signature · 1 line
2.44.0
Signature · 1 line
மணிவண்ணன் சதாசிவம்
PATCH v3 4/9 PCI: endpoint: pci-epf-test: Simplify pci_epf_test_alloc_space() loop
In reply to Niklas CasselMake pci-epf-test use pci_epc_get_next_free_bar() just like pci-epf-ntb.c and pci-epf-vntb.c.
Using pci_epc_get_next_free_bar() also makes it more obvious that pci-epf-test does no special configuration at all.
This way, the code is more consistent between EPF drivers, and pci-epf-test does not need to explicitly check if the BAR is reserved, or if the index belongs to a BAR succeeding a 64-bit only BAR.
--- drivers/pci/endpoint/functions/pci-epf-test.c | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-)
drivers/pci/endpoint/functions/pci-epf-test.c +8 -8
@@ -823,8 +823,8 @@ static int pci_epf_test_alloc_space(struct pci_epf *epf)size_t pba_size =0 ;bool msix_capable;void *base; -int bar, add;enum pci_barno test_reg_bar = epf_test->test_reg_bar; +enum pci_barno bar;const struct pci_epc_features *epc_features;size_t test_reg_size;
@@ -849,16 +849,14 @@ static int pci_epf_test_alloc_space(struct pci_epf *epf) } epf_test->reg[test_reg_bar] = base; -for (bar =0 ; bar <PCI_STD_NUM_BARS ; bar += add) { - epf_bar = &epf->bar[bar]; - add = (epf_bar->flags &PCI_BASE_ADDRESS_MEM_TYPE_64 ) ?2 :1 ; +for (bar =BAR_0 ; bar <PCI_STD_NUM_BARS ; bar++) { + bar =pci_epc_get_next_free_bar (epc_features, bar); +if (bar ==NO_BAR ) +break ;if (bar == test_reg_bar)continue ; -if (epc_features->bar[bar].type ==BAR_RESERVED ) -continue ; - base =pci_epf_alloc_space (epf, bar_size[bar], bar, epc_features,PRIMARY_INTERFACE );if (!base)
@@ -871,7 +869,9 @@ static int pci_epf_test_alloc_space(struct pci_epf *epf)* either because the BAR can only be a 64-bit BAR, or if * we requested a size larger than 4 GB. */ - add = (epf_bar->flags &PCI_BASE_ADDRESS_MEM_TYPE_64 ) ?2 :1 ; + epf_bar = &epf->bar[bar]; +if (epf_bar->flags &PCI_BASE_ADDRESS_MEM_TYPE_64 ) + bar++; }return 0 ;
Signature · 1 line
2.44.0
Re: [PATCH v3 4/9] PCI: endpoint: pci-epf-test: Simplify pci_epf_test_alloc_space() loop
Quoted 11 lines · Niklas Cassel
Make pci-epf-test use
pci_epc_get_next_free_bar()just likepci-epf-ntb.candpci-epf-vntb.c.Using
pci_epc_get_next_free_bar()also makes it more obvious that pci-epf-test does no special configuration at all.This way, the code is more consistent between EPF drivers, and pci-epf-test does not need to explicitly check if the BAR is reserved, or if the index belongs to a BAR succeeding a 64-bit only BAR.
Signed-off-by Niklas Cassel <[email protected]>
- Mani
Quoted 53 lines
--- drivers/pci/endpoint/functions/pci-epf-test.c | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-)drivers/pci/endpoint/functions/pci-epf-test.c +8
-8@@ -823,8 +823,8 @@ static int pci_epf_test_alloc_space(struct pci_epf *epf)size_t pba_size =0 ;bool msix_capable;void *base; -int bar, add;enum pci_barno test_reg_bar = epf_test->test_reg_bar; +enum pci_barno bar;const struct pci_epc_features *epc_features;size_t test_reg_size;@@ -849,16 +849,14 @@ static int pci_epf_test_alloc_space(struct pci_epf *epf) } epf_test->reg[test_reg_bar] = base; -for (bar =0 ; bar <PCI_STD_NUM_BARS ; bar += add) { - epf_bar = &epf->bar[bar]; - add = (epf_bar->flags &PCI_BASE_ADDRESS_MEM_TYPE_64 ) ?2 :1 ; +for (bar =BAR_0 ; bar <PCI_STD_NUM_BARS ; bar++) { + bar =pci_epc_get_next_free_bar (epc_features, bar); +if (bar ==NO_BAR ) +break ;if (bar == test_reg_bar)continue ; -if (epc_features->bar[bar].type ==BAR_RESERVED ) -continue ; - base =pci_epf_alloc_space (epf, bar_size[bar], bar, epc_features,PRIMARY_INTERFACE );if (!base)@@ -871,7 +869,9 @@ static int pci_epf_test_alloc_space(struct pci_epf *epf)* either because the BAR can only be a 64-bit BAR, or if * we requested a size larger than 4 GB. */ - add = (epf_bar->flags &PCI_BASE_ADDRESS_MEM_TYPE_64 ) ?2 :1 ; + epf_bar = &epf->bar[bar]; +if (epf_bar->flags &PCI_BASE_ADDRESS_MEM_TYPE_64 ) + bar++; }return 0 ;Signature · 1 line
2.44.0
Signature · 1 line
மணிவண்ணன் சதாசிவம்
PATCH v3 5/9 PCI: endpoint: pci-epf-test: Simplify pci_epf_test_set_bar() loop
In reply to Niklas CasselSimplify the loop in pci_epf_test_set_bar().
If we allocated memory for the BAR, we need to call set_bar() for that BAR, if we did not allocated memory for that BAR, we need to skip. It is as simple as that. This also matches the logic in pci_epf_test_unbind().
A 64-bit BAR will still only be one allocation, with the BAR succeeding the 64-bit BAR being null.
While at it, remove the misleading comment.
A EPC .set_bar() callback should never change the epf_bar->flags. (E.g. to set a 64-bit BAR if we requested a 32-bit BAR.)
A .set_bar() callback should do what we request it to do. If it can't satisfy the request, it should return an error.
If platform has a specific requirement, e.g. that a certain BAR has to be a 64-bit BAR, then it should specify that by setting the .only_64bit flag for that specific BAR in epc_features->bar[], such that pci_epf_alloc_space() will return a epf_bar with the 64-bit flag set. (Such that .set_bar() will receive a request to set a 64-bit BAR.)
--- drivers/pci/endpoint/functions/pci-epf-test.c | 21 ++++--------------- 1 file changed, 4 insertions(+), 17 deletions(-)
drivers/pci/endpoint/functions/pci-epf-test.c +4 -17
@@ -709,31 +709,18 @@ static void pci_epf_test_unbind(struct pci_epf *epf)static int pci_epf_test_set_bar (struct pci_epf *epf) { -int bar, add; -int ret; -struct pci_epf_bar *epf_bar; +int bar, ret;struct pci_epc *epc = epf->epc;struct device *dev = &epf->dev;struct pci_epf_test *epf_test =epf_get_drvdata (epf);enum pci_barno test_reg_bar = epf_test->test_reg_bar; -const struct pci_epc_features *epc_features; - - epc_features = epf_test->epc_features; -for (bar =0 ; bar <PCI_STD_NUM_BARS ; bar += add) { - epf_bar = &epf->bar[bar]; -/* -* pci_epc_set_bar() sets PCI_BASE_ADDRESS_MEM_TYPE_64 -* if the specific implementation required a 64-bit BAR, -* even if we only requested a 32-bit BAR. -*/ - add = (epf_bar->flags &PCI_BASE_ADDRESS_MEM_TYPE_64 ) ?2 :1 ; - -if (epc_features->bar[bar].type ==BAR_RESERVED ) +for (bar =0 ; bar <PCI_STD_NUM_BARS ; bar++) { +if (!epf_test->reg[bar])continue ; ret =pci_epc_set_bar (epc, epf->func_no, epf->vfunc_no, - epf_bar); + &epf->bar[bar]);if (ret) {pci_epf_free_space (epf, epf_test->reg[bar], bar,PRIMARY_INTERFACE );
Signature · 1 line
2.44.0
Re: [PATCH v3 5/9] PCI: endpoint: pci-epf-test: Simplify pci_epf_test_set_bar() loop
Quoted 9 lines · Niklas Cassel
Simplify the loop in
pci_epf_test_set_bar().
If we allocated memory for the BAR, we need to callset_bar()for that BAR, if we did not allocated memory for that BAR, we need to skip. It is as simple as that. This also matches the logic inpci_epf_test_unbind().A 64-bit BAR will still only be one allocation, with the BAR succeeding the 64-bit BAR being null.
Quoted 7 lines
While at it, remove the misleading comment.
A EPC .set_bar() callback should never change theepf_bar->flags. (E.g. to set a 64-bit BAR if we requested a 32-bit BAR.)A .set_bar() callback should do what we request it to do. If it can't satisfy the request, it should return an error.
That's a valid assumption. But...
Quoted 6 lines
If platform has a specific requirement, e.g. that a certain BAR has to be a 64-bit BAR, then it should specify that by setting the .only_64bit flag for that specific BAR in epc_features->bar[], such that
pci_epf_alloc_space()will return aepf_barwith the 64-bit flag set. (Such that .set_bar() will receive a request to set a 64-bit BAR.)
Looks like pcie-cadence-ep is setting the PCI_BASE_ADDRESS_MEM_TYPE_64 flag if the requested size is >2GB (I don't know why 2GB instead of 4GB in the first place).
- Mani
Quoted 48 lines
Signed-off-by Niklas Cassel <[email protected]>--- drivers/pci/endpoint/functions/pci-epf-test.c | 21 ++++--------------- 1 file changed, 4 insertions(+), 17 deletions(-)drivers/pci/endpoint/functions/pci-epf-test.c +4
-17@@ -709,31 +709,18 @@ static void pci_epf_test_unbind(struct pci_epf *epf)static int pci_epf_test_set_bar (struct pci_epf *epf) { -int bar, add; -int ret; -struct pci_epf_bar *epf_bar; +int bar, ret;struct pci_epc *epc = epf->epc;struct device *dev = &epf->dev;struct pci_epf_test *epf_test =epf_get_drvdata (epf);enum pci_barno test_reg_bar = epf_test->test_reg_bar; -const struct pci_epc_features *epc_features; - - epc_features = epf_test->epc_features; -for (bar =0 ; bar <PCI_STD_NUM_BARS ; bar += add) { - epf_bar = &epf->bar[bar]; -/* -* pci_epc_set_bar() sets PCI_BASE_ADDRESS_MEM_TYPE_64 -* if the specific implementation required a 64-bit BAR, -* even if we only requested a 32-bit BAR. -*/ - add = (epf_bar->flags &PCI_BASE_ADDRESS_MEM_TYPE_64 ) ?2 :1 ; - -if (epc_features->bar[bar].type ==BAR_RESERVED ) +for (bar =0 ; bar <PCI_STD_NUM_BARS ; bar++) { +if (!epf_test->reg[bar])continue ; ret =pci_epc_set_bar (epc, epf->func_no, epf->vfunc_no, - epf_bar); + &epf->bar[bar]);if (ret) {pci_epf_free_space (epf, epf_test->reg[bar], bar,PRIMARY_INTERFACE );Signature · 1 line
2.44.0
Signature · 1 line
மணிவண்ணன் சதாசிவம்
Re: [PATCH v3 5/9] PCI: endpoint: pci-epf-test: Simplify pci_epf_test_set_bar() loop
Quoted 27 lines · Manivannan Sadhasivam
Quoted 9 lines · Niklas Cassel
Simplify the loop in
pci_epf_test_set_bar().
If we allocated memory for the BAR, we need to callset_bar()for that BAR, if we did not allocated memory for that BAR, we need to skip. It is as simple as that. This also matches the logic inpci_epf_test_unbind().A 64-bit BAR will still only be one allocation, with the BAR succeeding the 64-bit BAR being null.
Quoted 7 lines
While at it, remove the misleading comment.
A EPC .set_bar() callback should never change theepf_bar->flags. (E.g. to set a 64-bit BAR if we requested a 32-bit BAR.)A .set_bar() callback should do what we request it to do. If it can't satisfy the request, it should return an error.
That's a valid assumption. But...
Quoted 6 lines
If platform has a specific requirement, e.g. that a certain BAR has to be a 64-bit BAR, then it should specify that by setting the .only_64bit flag for that specific BAR in epc_features->bar[], such that
pci_epf_alloc_space()will return aepf_barwith the 64-bit flag set. (Such that .set_bar() will receive a request to set a 64-bit BAR.)
Quoted 3 lines
Looks like pcie-cadence-ep is setting the
PCI_BASE_ADDRESS_MEM_TYPE_64flag if the requested size is >2GB (I don't know why 2GB instead of 4GB in the first place).
That is dead code that will never be able to execute.
Ever since commit f25b5fae29d4 ("PCI: endpoint: Setting a BAR size > 4 GB is invalid if 64-bit flag is not set") it has been impossible to get the .set_bar() callback with a BAR size > 2 GB.
Yes, 2GB!
The author of f25b5fae29d4 is an idiot (yes, it is me). Anyway, the code itself in that commit is doing the right thing...
2GB is the maximum size of a 32-bit BAR.
So, since the the code in pcie-cadence-ep is dead code, I don't see any problem with this commit.
Kind regards,
Niklas
Quoted 54 lines
- Mani
Quoted 48 lines
Signed-off-by Niklas Cassel <[email protected]>--- drivers/pci/endpoint/functions/pci-epf-test.c | 21 ++++--------------- 1 file changed, 4 insertions(+), 17 deletions(-)drivers/pci/endpoint/functions/pci-epf-test.c +4
-17@@ -709,31 +709,18 @@ static void pci_epf_test_unbind(struct pci_epf *epf)static int pci_epf_test_set_bar (struct pci_epf *epf) { -int bar, add; -int ret; -struct pci_epf_bar *epf_bar; +int bar, ret;struct pci_epc *epc = epf->epc;struct device *dev = &epf->dev;struct pci_epf_test *epf_test =epf_get_drvdata (epf);enum pci_barno test_reg_bar = epf_test->test_reg_bar; -const struct pci_epc_features *epc_features; - - epc_features = epf_test->epc_features; -for (bar =0 ; bar <PCI_STD_NUM_BARS ; bar += add) { - epf_bar = &epf->bar[bar]; -/* -* pci_epc_set_bar() sets PCI_BASE_ADDRESS_MEM_TYPE_64 -* if the specific implementation required a 64-bit BAR, -* even if we only requested a 32-bit BAR. -*/ - add = (epf_bar->flags &PCI_BASE_ADDRESS_MEM_TYPE_64 ) ?2 :1 ; - -if (epc_features->bar[bar].type ==BAR_RESERVED ) +for (bar =0 ; bar <PCI_STD_NUM_BARS ; bar++) { +if (!epf_test->reg[bar])continue ; ret =pci_epc_set_bar (epc, epf->func_no, epf->vfunc_no, - epf_bar); + &epf->bar[bar]);if (ret) {pci_epf_free_space (epf, epf_test->reg[bar], bar,PRIMARY_INTERFACE );Signature · 1 line
2.44.0Signature · 1 line
மணிவண்ணன் சதாசிவம்
PATCH v3 6/9 PCI: endpoint: pci-epf-test: Clean up pci_epf_test_unbind()
In reply to Niklas CasselClean up pci_epf_test_unbind() by using a continue if we did not allocate memory for the BAR index. This reduces the indentation level by one.
This makes pci_epf_test_unbind() more similar to pci_epf_test_set_bar().
--- drivers/pci/endpoint/functions/pci-epf-test.c | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-)
drivers/pci/endpoint/functions/pci-epf-test.c +6 -8
@@ -690,20 +690,18 @@ static void pci_epf_test_unbind(struct pci_epf *epf) {struct pci_epf_test *epf_test =epf_get_drvdata (epf);struct pci_epc *epc = epf->epc; -struct pci_epf_bar *epf_bar;int bar;cancel_delayed_work (&epf_test->cmd_handler);pci_epf_test_clean_dma_chan (epf_test);for (bar =0 ; bar <PCI_STD_NUM_BARS ; bar++) { - epf_bar = &epf->bar[bar]; +if (!epf_test->reg[bar]) +continue ; -if (epf_test->reg[bar]) { -pci_epc_clear_bar (epc, epf->func_no, epf->vfunc_no, - epf_bar); -pci_epf_free_space (epf, epf_test->reg[bar], bar, -PRIMARY_INTERFACE ); - } +pci_epc_clear_bar (epc, epf->func_no, epf->vfunc_no, + &epf->bar[bar]); +pci_epf_free_space (epf, epf_test->reg[bar], bar, +PRIMARY_INTERFACE ); } }
Signature · 1 line
2.44.0
Re: [PATCH v3 6/9] PCI: endpoint: pci-epf-test: Clean up pci_epf_test_unbind()
Quoted 5 lines · Niklas Cassel
Clean up
pci_epf_test_unbind()by using a continue if we did not allocate memory for the BAR index. This reduces the indentation level by one.This makes
pci_epf_test_unbind()more similar topci_epf_test_set_bar().
I've proposed to move the clear_bar and free_space code to separate helper functions in my series [1]. If this series gets merged first (it really makes sense), then this patch can be dropped now.
- Mani
Quoted 39 lines
Signed-off-by Niklas Cassel <[email protected]>--- drivers/pci/endpoint/functions/pci-epf-test.c | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-)drivers/pci/endpoint/functions/pci-epf-test.c +6
-8@@ -690,20 +690,18 @@ static void pci_epf_test_unbind(struct pci_epf *epf) {struct pci_epf_test *epf_test =epf_get_drvdata (epf);struct pci_epc *epc = epf->epc; -struct pci_epf_bar *epf_bar;int bar;cancel_delayed_work (&epf_test->cmd_handler);pci_epf_test_clean_dma_chan (epf_test);for (bar =0 ; bar <PCI_STD_NUM_BARS ; bar++) { - epf_bar = &epf->bar[bar]; +if (!epf_test->reg[bar]) +continue ; -if (epf_test->reg[bar]) { -pci_epc_clear_bar (epc, epf->func_no, epf->vfunc_no, - epf_bar); -pci_epf_free_space (epf, epf_test->reg[bar], bar, -PRIMARY_INTERFACE ); - } +pci_epc_clear_bar (epc, epf->func_no, epf->vfunc_no, + &epf->bar[bar]); +pci_epf_free_space (epf, epf_test->reg[bar], bar, +PRIMARY_INTERFACE ); } }Signature · 1 line
2.44.0
Signature · 1 line
மணிவண்ணன் சதாசிவம்
Re: [PATCH v3 6/9] PCI: endpoint: pci-epf-test: Clean up pci_epf_test_unbind()
Quoted 10 lines · Manivannan Sadhasivam
Quoted 5 lines · Niklas Cassel
Clean up
pci_epf_test_unbind()by using a continue if we did not allocate memory for the BAR index. This reduces the indentation level by one.This makes
pci_epf_test_unbind()more similar topci_epf_test_set_bar().I've proposed to move the
clear_barandfree_spacecode to separate helper functions in my series [1]. If this series gets merged first (it really makes sense), then this patch can be dropped now.
I've been a bit busy this week, hopefully I will have time to review your two outstanding series before the end of the week.
I do think that this series is smaller and less complex than your two series, so if you ask me, I think it would make sense if this series (the respin of this series) goes first.
Kind regards,
Niklas
Quoted 45 lines
- Mani
Quoted 39 lines
Signed-off-by Niklas Cassel <[email protected]>--- drivers/pci/endpoint/functions/pci-epf-test.c | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-)drivers/pci/endpoint/functions/pci-epf-test.c +6
-8@@ -690,20 +690,18 @@ static void pci_epf_test_unbind(struct pci_epf *epf) {struct pci_epf_test *epf_test =epf_get_drvdata (epf);struct pci_epc *epc = epf->epc; -struct pci_epf_bar *epf_bar;int bar;cancel_delayed_work (&epf_test->cmd_handler);pci_epf_test_clean_dma_chan (epf_test);for (bar =0 ; bar <PCI_STD_NUM_BARS ; bar++) { - epf_bar = &epf->bar[bar]; +if (!epf_test->reg[bar]) +continue ; -if (epf_test->reg[bar]) { -pci_epc_clear_bar (epc, epf->func_no, epf->vfunc_no, - epf_bar); -pci_epf_free_space (epf, epf_test->reg[bar], bar, -PRIMARY_INTERFACE ); - } +pci_epc_clear_bar (epc, epf->func_no, epf->vfunc_no, + &epf->bar[bar]); +pci_epf_free_space (epf, epf_test->reg[bar], bar, +PRIMARY_INTERFACE ); } }Signature · 1 line
2.44.0Signature · 1 line
மணிவண்ணன் சதாசிவம்
PATCH v3 7/9 PCI: cadence: Set a 64-bit BAR if requested
In reply to Niklas CasselEver since commit f25b5fae29d4 ("PCI: endpoint: Setting a BAR size > 4 GB is invalid if 64-bit flag is not set") it has been impossible to get the .set_bar() callback with a BAR size > 4 GB, if the BAR was also not requested to be configured as a 64-bit BAR.
Thus, forcing setting the 64-bit flag for BARs larger than 4 GB in the lower level driver is dead code and can be removed.
It is however possible that an EPF driver configures a BAR as 64-bit, even if the requested size is < 4 GB.
Respect the requested BAR configuration, just like how it is already repected with regards to the prefetchable bit.
--- drivers/pci/controller/cadence/pcie-cadence-ep.c | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-)
drivers/pci/controller/cadence/pcie-cadence-ep.c +1 -4
@@ -99,14 +99,11 @@ static int cdns_pcie_ep_set_bar(struct pci_epc *epc, u8 fn, u8 vfn, ctrl =CDNS_PCIE_LM_BAR_CFG_CTRL_IO_32BITS ; }else {bool is_prefetch = !!(flags &PCI_BASE_ADDRESS_MEM_PREFETCH ); -bool is_64bits = sz >SZ_2G ; +bool is_64bits = !!(flags &PCI_BASE_ADDRESS_MEM_TYPE_64 );if (is_64bits && (bar &1 ))return -EINVAL ; -if (is_64bits && !(flags &PCI_BASE_ADDRESS_MEM_TYPE_64 )) - epf_bar->flags |=PCI_BASE_ADDRESS_MEM_TYPE_64 ; -if (is_64bits && is_prefetch) ctrl =CDNS_PCIE_LM_BAR_CFG_CTRL_PREFETCH_MEM_64BITS ;else if (is_prefetch)
Signature · 1 line
2.44.0
Re: [PATCH v3 7/9] PCI: cadence: Set a 64-bit BAR if requested
Quoted 6 lines · Niklas Cassel
Ever since commit
f25b5fae29d4("PCI: endpoint: Setting a BAR size > 4 GB is invalid if 64-bit flag is not set") it has been impossible to get the .set_bar() callback with a BAR size > 4 GB, if the BAR was also not requested to be configured as a 64-bit BAR.Thus, forcing setting the 64-bit flag for BARs larger than 4 GB in the
2 GB
Quoted 9 lines
lower level driver is dead code and can be removed.
It is however possible that an EPF driver configures a BAR as 64-bit, even if the requested size is < 4 GB.
Respect the requested BAR configuration, just like how it is already repected with regards to the prefetchable bit.
Signed-off-by Niklas Cassel <[email protected]>
Okay, here you are fixing this driver. But this should be moved before patch 5/9. With that,
- Mani
Quoted 27 lines
--- drivers/pci/controller/cadence/pcie-cadence-ep.c | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-)drivers/pci/controller/cadence/pcie-cadence-ep.c +1
-4@@ -99,14 +99,11 @@ static int cdns_pcie_ep_set_bar(struct pci_epc *epc, u8 fn, u8 vfn, ctrl =CDNS_PCIE_LM_BAR_CFG_CTRL_IO_32BITS ; }else {bool is_prefetch = !!(flags &PCI_BASE_ADDRESS_MEM_PREFETCH ); -bool is_64bits = sz >SZ_2G ; +bool is_64bits = !!(flags &PCI_BASE_ADDRESS_MEM_TYPE_64 );if (is_64bits && (bar &1 ))return -EINVAL ; -if (is_64bits && !(flags &PCI_BASE_ADDRESS_MEM_TYPE_64 )) - epf_bar->flags |=PCI_BASE_ADDRESS_MEM_TYPE_64 ; -if (is_64bits && is_prefetch) ctrl =CDNS_PCIE_LM_BAR_CFG_CTRL_PREFETCH_MEM_64BITS ;else if (is_prefetch)Signature · 1 line
2.44.0
Signature · 1 line
மணிவண்ணன் சதாசிவம்
Re: [PATCH v3 7/9] PCI: cadence: Set a 64-bit BAR if requested
Quoted 9 lines · Manivannan Sadhasivam
Quoted 6 lines · Niklas Cassel
Ever since commit
f25b5fae29d4("PCI: endpoint: Setting a BAR size > 4 GB is invalid if 64-bit flag is not set") it has been impossible to get the .set_bar() callback with a BAR size > 4 GB, if the BAR was also not requested to be configured as a 64-bit BAR.Thus, forcing setting the 64-bit flag for BARs larger than 4 GB in the
2 GB
Will fix in V4.
Quoted 6 lines
Quoted 6 lines
lower level driver is dead code and can be removed.
It is however possible that an EPF driver configures a BAR as 64-bit, even if the requested size is < 4 GB.
Quoted 7 lines
Quoted 4 lines
Respect the requested BAR configuration, just like how it is already repected with regards to the prefetchable bit.
Signed-off-by Niklas Cassel <[email protected]>Okay, here you are fixing this driver. But this should be moved before patch 5/9. With that,
As I wrote in my reply to patch 5/9, I don't think we need to move them around.
The code that sets flag PCI_BASE_ADDRESS_MEM_TYPE_64 in this driver is dead code, it couldn't happen before this commit. So unless you insist, I think it is better to keep the current ordering, such that all pci-epf-test patches are after each other.
Kind regards,
Niklas
Quoted 35 lines
Reviewed-by Manivannan Sadhasivam <[email protected]>- Mani
Quoted 27 lines
--- drivers/pci/controller/cadence/pcie-cadence-ep.c | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-)drivers/pci/controller/cadence/pcie-cadence-ep.c +1
-4@@ -99,14 +99,11 @@ static int cdns_pcie_ep_set_bar(struct pci_epc *epc, u8 fn, u8 vfn, ctrl =CDNS_PCIE_LM_BAR_CFG_CTRL_IO_32BITS ; }else {bool is_prefetch = !!(flags &PCI_BASE_ADDRESS_MEM_PREFETCH ); -bool is_64bits = sz >SZ_2G ; +bool is_64bits = !!(flags &PCI_BASE_ADDRESS_MEM_TYPE_64 );if (is_64bits && (bar &1 ))return -EINVAL ; -if (is_64bits && !(flags &PCI_BASE_ADDRESS_MEM_TYPE_64 )) - epf_bar->flags |=PCI_BASE_ADDRESS_MEM_TYPE_64 ; -if (is_64bits && is_prefetch) ctrl =CDNS_PCIE_LM_BAR_CFG_CTRL_PREFETCH_MEM_64BITS ;else if (is_prefetch)Signature · 1 line
2.44.0Signature · 1 line
மணிவண்ணன் சதாசிவம்
PATCH v3 8/9 PCI: rockchip-ep: Set a 64-bit BAR if requested
In reply to Niklas CasselEver since commit f25b5fae29d4 ("PCI: endpoint: Setting a BAR size > 4 GB is invalid if 64-bit flag is not set") it has been impossible to get the .set_bar() callback with a BAR size > 4 GB, if the BAR was also not requested to be configured as a 64-bit BAR.
It is however possible that an EPF driver configures a BAR as 64-bit, even if the requested size is < 4 GB.
Respect the requested BAR configuration, just like how it is already repected with regards to the prefetchable bit.
--- drivers/pci/controller/pcie-rockchip-ep.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/pci/controller/pcie-rockchip-ep.c +1 -1
@@ -153,7 +153,7 @@ static int rockchip_pcie_ep_set_bar(struct pci_epc *epc, u8 fn, u8 vfn, ctrl =ROCKCHIP_PCIE_CORE_BAR_CFG_CTRL_IO_32BITS ; }else {bool is_prefetch = !!(flags &PCI_BASE_ADDRESS_MEM_PREFETCH ); -bool is_64bits = sz >SZ_2G ; +bool is_64bits = !!(flags &PCI_BASE_ADDRESS_MEM_TYPE_64 );if (is_64bits && (bar &1 ))return -EINVAL ;
Signature · 1 line
2.44.0
Re: [PATCH v3 8/9] PCI: rockchip-ep: Set a 64-bit BAR if requested
Quoted 7 lines · Niklas Cassel
Ever since commit
f25b5fae29d4("PCI: endpoint: Setting a BAR size > 4 GB is invalid if 64-bit flag is not set") it has been impossible to get the .set_bar() callback with a BAR size > 4 GB, if the BAR was also not requested to be configured as a 64-bit BAR.It is however possible that an EPF driver configures a BAR as 64-bit, even if the requested size is < 4 GB.
2 GB
Quoted 5 lines
Respect the requested BAR configuration, just like how it is already repected with regards to the prefetchable bit.
Signed-off-by Niklas Cassel <[email protected]>
Ah, I missed this one. But the same comment as previous one applies to this patch also.
- Mani
Quoted 20 lines
--- drivers/pci/controller/pcie-rockchip-ep.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/pci/controller/pcie-rockchip-ep.c +1
-1@@ -153,7 +153,7 @@ static int rockchip_pcie_ep_set_bar(struct pci_epc *epc, u8 fn, u8 vfn, ctrl =ROCKCHIP_PCIE_CORE_BAR_CFG_CTRL_IO_32BITS ; }else {bool is_prefetch = !!(flags &PCI_BASE_ADDRESS_MEM_PREFETCH ); -bool is_64bits = sz >SZ_2G ; +bool is_64bits = !!(flags &PCI_BASE_ADDRESS_MEM_TYPE_64 );if (is_64bits && (bar &1 ))return -EINVAL ;Signature · 1 line
2.44.0
Signature · 1 line
மணிவண்ணன் சதாசிவம்
Re: [PATCH v3 8/9] PCI: rockchip-ep: Set a 64-bit BAR if requested
In reply to Niklas CasselQuoted 10 lines · Niklas Cassel
Ever since commit
f25b5fae29d4("PCI: endpoint: Setting a BAR size > 4 GB is invalid if 64-bit flag is not set") it has been impossible to get the .set_bar() callback with a BAR size > 4 GB, if the BAR was also not requested to be configured as a 64-bit BAR.It is however possible that an EPF driver configures a BAR as 64-bit, even if the requested size is < 4 GB.
Respect the requested BAR configuration, just like how it is already repected with regards to the prefetchable bit.
Does this (and the similar cadence patch) need a Fixes: tag for f25b5fae29d4?
Quoted 21 lines
Signed-off-by Niklas Cassel <[email protected]>--- drivers/pci/controller/pcie-rockchip-ep.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/pci/controller/pcie-rockchip-ep.c +1
-1@@ -153,7 +153,7 @@ static int rockchip_pcie_ep_set_bar(struct pci_epc *epc, u8 fn, u8 vfn, ctrl =ROCKCHIP_PCIE_CORE_BAR_CFG_CTRL_IO_32BITS ; }else {bool is_prefetch = !!(flags &PCI_BASE_ADDRESS_MEM_PREFETCH ); -bool is_64bits = sz >SZ_2G ; +bool is_64bits = !!(flags &PCI_BASE_ADDRESS_MEM_TYPE_64 );if (is_64bits && (bar &1 ))return -EINVAL ;Signature · 1 line
2.44.0
Re: [PATCH v3 8/9] PCI: rockchip-ep: Set a 64-bit BAR if requested
Quoted 6 lines · Bjorn Helgaas
Quoted 5 lines · Niklas Cassel
Ever since commit
f25b5fae29d4("PCI: endpoint: Setting a BAR size > 4 GB is invalid if 64-bit flag is not set") it has been impossible to get the .set_bar() callback with a BAR size > 4 GB, if the BAR was also not requested to be configured as a 64-bit BAR.
Quoted 8 lines
Quoted 5 lines
It is however possible that an EPF driver configures a BAR as 64-bit, even if the requested size is < 4 GB.
Respect the requested BAR configuration, just like how it is already repected with regards to the prefetchable bit.
Does this (and the similar cadence patch) need a Fixes: tag for
f25b5fae29d4?
I don't think so.
Both patches are about respecting the configuration requested by an EPF driver.
So if an EPF driver requests a 64-bit BAR, the EPC driver should configure that. (Regardless of the size that the EPF driver requests for the BAR.)
If we really want a Fixes-tag, I would imagine that it will be the respective initial commits that added these drivers (pcie-cadence-ep.c and pcie-rockchip-ep.c), as it has been this way since then.
If you look at the EPF drivers we currently have, they will currently only request a 64-bit BAR if any of the BARs can only be configured as a 64-bit BAR because of hardware limitiations.
$ git grep only_64bit
Neither of these two drivers have any such hardware limitiations, so these commits are currently a bit pointless.
However, the drivers should of course do the right thing, because other EPC drivers might look at them and copy their code.
And who knows, maybe sometime in the future there will be an EPF driver that will explicitly request a 64-bit BAR, regardless of size.
TL;DR: I don't think these two commits are worth backporting.
Quoted 22 lines
Quoted 22 lines
Signed-off-by Niklas Cassel <[email protected]>--- drivers/pci/controller/pcie-rockchip-ep.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/pci/controller/pcie-rockchip-ep.c +1
-1@@ -153,7 +153,7 @@ static int rockchip_pcie_ep_set_bar(struct pci_epc *epc, u8 fn, u8 vfn, ctrl =ROCKCHIP_PCIE_CORE_BAR_CFG_CTRL_IO_32BITS ; }else {bool is_prefetch = !!(flags &PCI_BASE_ADDRESS_MEM_PREFETCH ); -bool is_64bits = sz >SZ_2G ; +bool is_64bits = !!(flags &PCI_BASE_ADDRESS_MEM_TYPE_64 );if (is_64bits && (bar &1 ))return -EINVAL ;Signature · 1 line
2.44.0
Re: [PATCH v3 8/9] PCI: rockchip-ep: Set a 64-bit BAR if requested
Quoted 37 lines · Niklas Cassel
Quoted 6 lines · Bjorn Helgaas
Quoted 5 lines · Niklas Cassel
Ever since commit
f25b5fae29d4("PCI: endpoint: Setting a BAR size > 4 GB is invalid if 64-bit flag is not set") it has been impossible to get the .set_bar() callback with a BAR size > 4 GB, if the BAR was also not requested to be configured as a 64-bit BAR.Quoted 8 lines
Quoted 5 lines
It is however possible that an EPF driver configures a BAR as 64-bit, even if the requested size is < 4 GB.
Respect the requested BAR configuration, just like how it is already repected with regards to the prefetchable bit.
Does this (and the similar cadence patch) need a Fixes: tag for
f25b5fae29d4?I don't think so.
Both patches are about respecting the configuration requested by an EPF driver.
So if an EPF driver requests a 64-bit BAR, the EPC driver should configure that. (Regardless of the size that the EPF driver requests for the BAR.)
If we really want a Fixes-tag, I would imagine that it will be the respective initial commits that added these drivers (
pcie-cadence-ep.candpcie-rockchip-ep.c), as it has been this way since then.If you look at the EPF drivers we currently have, they will currently only request a 64-bit BAR if any of the BARs can only be configured as a 64-bit BAR because of hardware limitiations.
$ git grep only_64bitNeither of these two drivers have any such hardware limitiations, so these commits are currently a bit pointless.
Quoted 7 lines
However, the drivers should of course do the right thing, because other EPC drivers might look at them and copy their code.
And who knows, maybe sometime in the future there will be an EPF driver that will explicitly request a 64-bit BAR, regardless of size.
TL;DR: I don't think these two commits are worth backporting.
OK, thanks!
Quoted 26 lines
Quoted 16 lines
Quoted 16 lines
Signed-off-by Niklas Cassel <[email protected]>--- drivers/pci/controller/pcie-rockchip-ep.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/pci/controller/pcie-rockchip-ep.c +1
-1@@ -153,7 +153,7 @@ static int rockchip_pcie_ep_set_bar(struct pci_epc *epc, u8 fn, u8 vfn, ctrl =ROCKCHIP_PCIE_CORE_BAR_CFG_CTRL_IO_32BITS ; }else {bool is_prefetch = !!(flags &PCI_BASE_ADDRESS_MEM_PREFETCH ); -bool is_64bits = sz >SZ_2G ; +bool is_64bits = !!(flags &PCI_BASE_ADDRESS_MEM_TYPE_64 );Quoted 5 lines
Quoted 5 lines
drivers/pci/controller/pcie-rockchip-ep.c +0
-0if (is_64bits && (bar &1 ))return -EINVAL ;Signature · 1 line
2.44.0_______________________________________________
Linux-rockchip mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-rockchip
Re: [PATCH v3 8/9] PCI: rockchip-ep: Set a 64-bit BAR if requested
In reply to Niklas Casseldrivers/pci/controller/pcie-rockchip-ep.c static int rockchip_pcie_ep_set_bar(struct pci_epc *epc, u8 fn, u8 vfn, · line 159
Quoted 13 lines · Niklas Cassel
...
drivers/pci/controller/pcie-rockchip-ep.c +1
-1@@ -153,7 +153,7 @@ static int rockchip_pcie_ep_set_bar(struct pci_epc *epc, u8 fn, u8 vfn, ctrl =ROCKCHIP_PCIE_CORE_BAR_CFG_CTRL_IO_32BITS ; }else {bool is_prefetch = !!(flags &PCI_BASE_ADDRESS_MEM_PREFETCH ); -bool is_64bits = sz >SZ_2G ; +bool is_64bits = !!(flags &PCI_BASE_ADDRESS_MEM_TYPE_64 );if (is_64bits && (bar &1 ))return -EINVAL ;
Completely unrelated to *these* patches, but the BAR_CFG_CTRL definitions in both cadence and rockchip lead to some awkward case analysis:
#define ROCKCHIP_PCIE_CORE_BAR_CFG_CTRL_MEM_32BITS 0x4 #define ROCKCHIP_PCIE_CORE_BAR_CFG_CTRL_PREFETCH_MEM_32BITS 0x5 #define ROCKCHIP_PCIE_CORE_BAR_CFG_CTRL_MEM_64BITS 0x6 #define ROCKCHIP_PCIE_CORE_BAR_CFG_CTRL_PREFETCH_MEM_64BITS 0x7
if (is_64bits && is_prefetch)
ctrl = ROCKCHIP_PCIE_CORE_BAR_CFG_CTRL_PREFETCH_MEM_64BITS;
else if (is_prefetch)
ctrl = ROCKCHIP_PCIE_CORE_BAR_CFG_CTRL_PREFETCH_MEM_32BITS;
else if (is_64bits)
ctrl = ROCKCHIP_PCIE_CORE_BAR_CFG_CTRL_MEM_64BITS;
else
ctrl = ROCKCHIP_PCIE_CORE_BAR_CFG_CTRL_MEM_32BITS;that *could* be just something like this:
#define ROCKCHIP_PCIE_CORE_BAR_CFG_CTRL_MEM 0x4 #define ROCKCHIP_PCIE_CORE_BAR_CFG_CTRL_64BITS 0x2 #define ROCKCHIP_PCIE_CORE_BAR_CFG_CTRL_PREFETCH 0x1
ctrl = ROCKCHIP_PCIE_CORE_BAR_CFG_CTRL_MEM;
if (is_64bits)
ctrl |= ROCKCHIP_PCIE_CORE_BAR_CFG_CTRL_64BITS;
if (is_prefetch)
ctrl |= ROCKCHIP_PCIE_CORE_BAR_CFG_CTRL_PREFETCH;Re: [PATCH v3 8/9] PCI: rockchip-ep: Set a 64-bit BAR if requested
Quoted 39 lines · Bjorn Helgaas
drivers/pci/controller/pcie-rockchip-ep.c static int rockchip_pcie_ep_set_bar(struct pci_epc *epc, u8 fn, u8 vfn, · line 159
Quoted 13 lines · Niklas Cassel
...
drivers/pci/controller/pcie-rockchip-ep.c +1
-1@@ -153,7 +153,7 @@ static int rockchip_pcie_ep_set_bar(struct pci_epc *epc, u8 fn, u8 vfn, ctrl =ROCKCHIP_PCIE_CORE_BAR_CFG_CTRL_IO_32BITS ; }else {bool is_prefetch = !!(flags &PCI_BASE_ADDRESS_MEM_PREFETCH ); -bool is_64bits = sz >SZ_2G ; +bool is_64bits = !!(flags &PCI_BASE_ADDRESS_MEM_TYPE_64 );if (is_64bits && (bar &1 ))return -EINVAL ;Completely unrelated to *these* patches, but the
BAR_CFG_CTRLdefinitions in both cadence and rockchip lead to some awkward case analysis:#define ROCKCHIP_PCIE_CORE_BAR_CFG_CTRL_MEM_32BITS 0x4 #define ROCKCHIP_PCIE_CORE_BAR_CFG_CTRL_PREFETCH_MEM_32BITS 0x5 #define ROCKCHIP_PCIE_CORE_BAR_CFG_CTRL_MEM_64BITS 0x6 #define ROCKCHIP_PCIE_CORE_BAR_CFG_CTRL_PREFETCH_MEM_64BITS 0x7if (is_64bits && is_prefetch) ctrl = ROCKCHIP_PCIE_CORE_BAR_CFG_CTRL_PREFETCH_MEM_64BITS; else if (is_prefetch) ctrl = ROCKCHIP_PCIE_CORE_BAR_CFG_CTRL_PREFETCH_MEM_32BITS; else if (is_64bits) ctrl = ROCKCHIP_PCIE_CORE_BAR_CFG_CTRL_MEM_64BITS; else ctrl = ROCKCHIP_PCIE_CORE_BAR_CFG_CTRL_MEM_32BITS;that *could* be just something like this:
#define ROCKCHIP_PCIE_CORE_BAR_CFG_CTRL_MEM 0x4 #define ROCKCHIP_PCIE_CORE_BAR_CFG_CTRL_64BITS 0x2 #define ROCKCHIP_PCIE_CORE_BAR_CFG_CTRL_PREFETCH 0x1
Quoted 5 lines
ctrl = ROCKCHIP_PCIE_CORE_BAR_CFG_CTRL_MEM;
if (is_64bits)
ctrl |= ROCKCHIP_PCIE_CORE_BAR_CFG_CTRL_64BITS;
if (is_prefetch)
ctrl |= ROCKCHIP_PCIE_CORE_BAR_CFG_CTRL_PREFETCH;If you send the cleanup patches, I will send the Reviewed-by tags ;)
Kind regards,
Niklas
Re: [PATCH v3 8/9] PCI: rockchip-ep: Set a 64-bit BAR if requested
Quoted 40 lines · Niklas Cassel
Quoted 39 lines · Bjorn Helgaas
drivers/pci/controller/pcie-rockchip-ep.c static int rockchip_pcie_ep_set_bar(struct pci_epc *epc, u8 fn, u8 vfn, · line 159
Quoted 13 lines · Niklas Cassel
...
drivers/pci/controller/pcie-rockchip-ep.c +1
-1@@ -153,7 +153,7 @@ static int rockchip_pcie_ep_set_bar(struct pci_epc *epc, u8 fn, u8 vfn, ctrl =ROCKCHIP_PCIE_CORE_BAR_CFG_CTRL_IO_32BITS ; }else {bool is_prefetch = !!(flags &PCI_BASE_ADDRESS_MEM_PREFETCH ); -bool is_64bits = sz >SZ_2G ; +bool is_64bits = !!(flags &PCI_BASE_ADDRESS_MEM_TYPE_64 );if (is_64bits && (bar &1 ))return -EINVAL ;Completely unrelated to *these* patches, but the
BAR_CFG_CTRLdefinitions in both cadence and rockchip lead to some awkward case analysis:#define ROCKCHIP_PCIE_CORE_BAR_CFG_CTRL_MEM_32BITS 0x4 #define ROCKCHIP_PCIE_CORE_BAR_CFG_CTRL_PREFETCH_MEM_32BITS 0x5 #define ROCKCHIP_PCIE_CORE_BAR_CFG_CTRL_MEM_64BITS 0x6 #define ROCKCHIP_PCIE_CORE_BAR_CFG_CTRL_PREFETCH_MEM_64BITS 0x7if (is_64bits && is_prefetch) ctrl = ROCKCHIP_PCIE_CORE_BAR_CFG_CTRL_PREFETCH_MEM_64BITS; else if (is_prefetch) ctrl = ROCKCHIP_PCIE_CORE_BAR_CFG_CTRL_PREFETCH_MEM_32BITS; else if (is_64bits) ctrl = ROCKCHIP_PCIE_CORE_BAR_CFG_CTRL_MEM_64BITS; else ctrl = ROCKCHIP_PCIE_CORE_BAR_CFG_CTRL_MEM_32BITS;that *could* be just something like this:
#define ROCKCHIP_PCIE_CORE_BAR_CFG_CTRL_MEM 0x4 #define ROCKCHIP_PCIE_CORE_BAR_CFG_CTRL_64BITS 0x2 #define ROCKCHIP_PCIE_CORE_BAR_CFG_CTRL_PREFETCH 0x1
Quoted 7 lines
Quoted 5 lines
ctrl = ROCKCHIP_PCIE_CORE_BAR_CFG_CTRL_MEM; if (is_64bits) ctrl |= ROCKCHIP_PCIE_CORE_BAR_CFG_CTRL_64BITS; if (is_prefetch) ctrl |= ROCKCHIP_PCIE_CORE_BAR_CFG_CTRL_PREFETCH;If you send the cleanup patches, I will send the Reviewed-by tags ;)
Fair enough :)
PATCH v3 9/9 PCI: endpoint: Set prefetch when allocating memory for 64-bit BARs
In reply to Niklas CasselFrom the PCIe 6.0 base spec:
"Generally only 64-bit BARs are good candidates, since only Legacy Endpoints are permitted to set the Prefetchable bit in 32-bit BARs, and most scalable platforms map all 32-bit Memory BARs into non-prefetchable Memory Space regardless of the Prefetchable bit value."
"For a PCI Express Endpoint, 64-bit addressing must be supported for all BARs that have the Prefetchable bit Set. 32-bit addressing is permitted for all BARs that do not have the Prefetchable bit Set."
"Any device that has a range that behaves like normal memory should mark the range as prefetchable. A linear frame buffer in a graphics device is an example of a range that should be marked prefetchable."
The PCIe spec tells us that we should have the prefetchable bit set for 64-bit BARs backed by "normal memory". The backing memory that we allocate for a 64-bit BAR using pci_epf_alloc_space() (which calls dma_alloc_coherent()) is obviously "normal memory".
Thus, set the prefetchable bit when allocating backing memory for a 64-bit BAR.
--- drivers/pci/endpoint/pci-epf-core.c | 3 +++ 1 file changed, 3 insertions(+)
drivers/pci/endpoint/pci-epf-core.c +3 -0
@@ -309,6 +309,9 @@ void *pci_epf_alloc_space(struct pci_epf *epf, size_t size, enum pci_barno bar,else epf_bar[bar].flags |=PCI_BASE_ADDRESS_MEM_TYPE_32 ; +if (epf_bar[bar].flags &PCI_BASE_ADDRESS_MEM_TYPE_64 ) + epf_bar[bar].flags |=PCI_BASE_ADDRESS_MEM_PREFETCH ; +return space; }EXPORT_SYMBOL_GPL (pci_epf_alloc_space);
Signature · 1 line
2.44.0
Re: [PATCH v3 9/9] PCI: endpoint: Set prefetch when allocating memory for 64-bit BARs
+ Arnd
Quoted 1 line · Niklas Cassel
From the PCIe 6.0 base spec:
It'd be good to mention the section also.
Quoted 9 lines
"Generally only 64-bit BARs are good candidates, since only Legacy Endpoints are permitted to set the Prefetchable bit in 32-bit BARs, and most scalable platforms map all 32-bit Memory BARs into non-prefetchable Memory Space regardless of the Prefetchable bit value."
"For a PCI Express Endpoint, 64-bit addressing must be supported for all BARs that have the Prefetchable bit Set. 32-bit addressing is permitted for all BARs that do not have the Prefetchable bit Set."
Quoted 9 lines
"Any device that has a range that behaves like normal memory should mark the range as prefetchable. A linear frame buffer in a graphics device is an example of a range that should be marked prefetchable."
The PCIe spec tells us that we should have the prefetchable bit set for 64-bit BARs backed by "normal memory". The backing memory that we allocate for a 64-bit BAR using
pci_epf_alloc_space()(which callsdma_alloc_coherent()) is obviously "normal memory".
I'm not sure this is correct. Memory returned by 'dma_alloc_coherent' is not the 'normal memory' but rather 'consistent/coherent memory'. Here the question is, can the memory returned by dma_alloc_coherent() be prefetched or write-combined on all architectures.
I hope Arnd can answer this question.
- Mani
Quoted 25 lines
Thus, set the prefetchable bit when allocating backing memory for a 64-bit BAR.
Signed-off-by Niklas Cassel <[email protected]>--- drivers/pci/endpoint/pci-epf-core.c | 3 +++ 1 file changed, 3 insertions(+)drivers/pci/endpoint/pci-epf-core.c +3
-0@@ -309,6 +309,9 @@ void *pci_epf_alloc_space(struct pci_epf *epf, size_t size, enum pci_barno bar,else epf_bar[bar].flags |=PCI_BASE_ADDRESS_MEM_TYPE_32 ; +if (epf_bar[bar].flags &PCI_BASE_ADDRESS_MEM_TYPE_64 ) + epf_bar[bar].flags |=PCI_BASE_ADDRESS_MEM_PREFETCH ; +return space; }EXPORT_SYMBOL_GPL (pci_epf_alloc_space);Signature · 1 line
2.44.0
Signature · 1 line
மணிவண்ணன் சதாசிவம்
Re: [PATCH v3 9/9] PCI: endpoint: Set prefetch when allocating memory for 64-bit BARs
Quoted 20 lines · Manivannan Sadhasivam
Quoted 9 lines · Niklas Cassel
"Generally only 64-bit BARs are good candidates, since only Legacy Endpoints are permitted to set the Prefetchable bit in 32-bit BARs, and most scalable platforms map all 32-bit Memory BARs into non-prefetchable Memory Space regardless of the Prefetchable bit value."
"For a PCI Express Endpoint, 64-bit addressing must be supported for all BARs that have the Prefetchable bit Set. 32-bit addressing is permitted for all BARs that do not have the Prefetchable bit Set."
Quoted 9 lines
"Any device that has a range that behaves like normal memory should mark the range as prefetchable. A linear frame buffer in a graphics device is an example of a range that should be marked prefetchable."
The PCIe spec tells us that we should have the prefetchable bit set for 64-bit BARs backed by "normal memory". The backing memory that we allocate for a 64-bit BAR using
pci_epf_alloc_space()(which callsdma_alloc_coherent()) is obviously "normal memory".
Quoted 6 lines
I'm not sure this is correct. Memory returned by '
dma_alloc_coherent' is not the 'normal memory' but rather 'consistent/coherent memory'. Here the question is, can the memory returned bydma_alloc_coherent()be prefetched or write-combined on all architectures.I hope Arnd can answer this question.
I think there are three separate questions here when talking about a scenario where a PCI master accesses memory behind a PCI endpoint:
- The CPU on the host side ususally uses ioremap() for mapping the PCI BAR of the device. If the BAR is marked as prefetchable, we usually allow mapping it using ioremap_wc() for write-combining or ioremap_wt() for a write-through mappings that allow both write-combining and prefetching. On some architectures, these all fall back to normal register mappings which do none of these. If it uses write-combining or prefetching, the host side driver will need to manually serialize against concurrent access from the endpoint side.
- The endpoint device accessing a buffer in memory is controlled by the endpoint driver and may decide to prefetch data into a local cache independent of the other two. I don't know if any of the suppored endpoint devices actually do that. A prefetch from the PCI host side would appear as a normal transaction here.
- The local CPU on the endpoint side may access the same buffer as the endpoint device. On low-end SoCs the DMA from the PCI endpoint is not coherent with the CPU caches, so the CPU may need to map it as uncacheable to allow data consistency with a the CPU on the PCI host side. On higher-end SoCs (e.g. most non-ARM ones) DMA is coherent with the caches, so the CPU on the endpoint side may map the buffer as cached and still be coherent with a CPU on the PCI host side that has mapped it with ioremap().
Arnd
Re: [PATCH v3 9/9] PCI: endpoint: Set prefetch when allocating memory for 64-bit BARs
Hello all,
Quoted 26 lines · Arnd Bergmann
Quoted 25 lines · Manivannan Sadhasivam
Quoted 9 lines · Niklas Cassel
"Generally only 64-bit BARs are good candidates, since only Legacy Endpoints are permitted to set the Prefetchable bit in 32-bit BARs, and most scalable platforms map all 32-bit Memory BARs into non-prefetchable Memory Space regardless of the Prefetchable bit value."
"For a PCI Express Endpoint, 64-bit addressing must be supported for all BARs that have the Prefetchable bit Set. 32-bit addressing is permitted for all BARs that do not have the Prefetchable bit Set."
Quoted 9 lines
"Any device that has a range that behaves like normal memory should mark the range as prefetchable. A linear frame buffer in a graphics device is an example of a range that should be marked prefetchable."
The PCIe spec tells us that we should have the prefetchable bit set for 64-bit BARs backed by "normal memory". The backing memory that we allocate for a 64-bit BAR using
pci_epf_alloc_space()(which callsdma_alloc_coherent()) is obviously "normal memory".I'm not sure this is correct. Memory returned by '
dma_alloc_coherent' is not the 'normal memory' but rather 'consistent/coherent memory'. Here the question is, can the memory returned bydma_alloc_coherent()be prefetched or write-combined on all architectures.
Quoted 4 lines
Quoted 1 line
I hope Arnd can answer this question.
I think there are three separate questions here when talking about a scenario where a PCI master accesses memory behind a PCI endpoint:
I think the question is if the PCI epf-core, which runs on the endpoint side, and which calls dma_alloc_coherent() to allocate backing memory for a BAR, can set/mark the Prefetchable bit for the BAR (if we also set/mark the BAR as a 64-bit BAR).
The PCIe 6.0 spec, 7.5.1.2.1 Base Address Registers (Offset 10h - 24h), states:
"Any device that has a range that behaves like normal memory should mark the range as prefetchable. A linear frame buffer in a graphics device is an example of a range that should be marked prefetchable."
Does not backing memory allocated for a specific BAR using dma_alloc_coherent() on the EP side behave like normal memory from the host's point of view?
On the host side, this will mean that the host driver sees the Prefetchable bit, and as according to:
https://docs.kernel.org/driver-api/device-io.html
The host might map the BAR using ioremap_wc().
Looking specifically at drivers/misc/pci_endpoint_test.c, it maps the BARs using pci_ioremap_bar():
https://elixir.bootlin.com/linux/v6.8/source/drivers/pci/pci.c#L252
which will not map it using ioremap_wc().
(But the code we have in the PCI epf-core must of course work with host side drivers other than pci_endpoint_test.c as well.)
Quoted 17 lines
- The CPU on the host side ususally uses
ioremap()for mapping the PCI BAR of the device. If the BAR is marked as prefetchable, we usually allow mapping it usingioremap_wc()for write-combining orioremap_wt()for a write-through mappings that allow both write-combining and prefetching. On some architectures, these all fall back to normal register mappings which do none of these. If it uses write-combining or prefetching, the host side driver will need to manually serialize against concurrent access from the endpoint side.- The endpoint device accessing a buffer in memory is controlled by the endpoint driver and may decide to prefetch data into a local cache independent of the other two. I don't know if any of the suppored endpoint devices actually do that. A prefetch from the PCI host side would appear as a normal transaction here.
Quoted 3 lines
- The local CPU on the endpoint side may access the same buffer as the endpoint device. On low-end SoCs the DMA from the PCI endpoint is not coherent with the CPU caches, so the CPU may
I don't follow. When doing DMA *from* the endpoint, then the DMA HW on the EP side will read or write data to a buffer allocated on the host side (most likely using dma_alloc_coherent()), but what does that got to do with how the EP configures the BARs that it exposes?
Quoted 6 lines
need to map it as uncacheable to allow data consistency with a the CPU on the PCI host side. On higher-end SoCs (e.g. most non-ARM ones) DMA is coherent with the caches, so the CPU on the endpoint side may map the buffer as cached and still be coherent with a CPU on the PCI host side that has mapped it with
ioremap().
Kind regards,
Niklas
Re: [PATCH v3 9/9] PCI: endpoint: Set prefetch when allocating memory for 64-bit BARs
Quoted 56 lines · Niklas Cassel
Hello all,
Quoted 26 lines · Arnd Bergmann
Quoted 25 lines · Manivannan Sadhasivam
Quoted 9 lines · Niklas Cassel
"Generally only 64-bit BARs are good candidates, since only Legacy Endpoints are permitted to set the Prefetchable bit in 32-bit BARs, and most scalable platforms map all 32-bit Memory BARs into non-prefetchable Memory Space regardless of the Prefetchable bit value."
"For a PCI Express Endpoint, 64-bit addressing must be supported for all BARs that have the Prefetchable bit Set. 32-bit addressing is permitted for all BARs that do not have the Prefetchable bit Set."
Quoted 9 lines
"Any device that has a range that behaves like normal memory should mark the range as prefetchable. A linear frame buffer in a graphics device is an example of a range that should be marked prefetchable."
The PCIe spec tells us that we should have the prefetchable bit set for 64-bit BARs backed by "normal memory". The backing memory that we allocate for a 64-bit BAR using
pci_epf_alloc_space()(which callsdma_alloc_coherent()) is obviously "normal memory".I'm not sure this is correct. Memory returned by '
dma_alloc_coherent' is not the 'normal memory' but rather 'consistent/coherent memory'. Here the question is, can the memory returned bydma_alloc_coherent()be prefetched or write-combined on all architectures.Quoted 4 lines
Quoted 1 line
I hope Arnd can answer this question.
I think there are three separate questions here when talking about a scenario where a PCI master accesses memory behind a PCI endpoint:
I think the question is if the PCI epf-core, which runs on the endpoint side, and which calls
dma_alloc_coherent()to allocate backing memory for a BAR, can set/mark the Prefetchable bit for the BAR (if we also set/mark the BAR as a 64-bit BAR).The PCIe 6.0 spec, 7.5.1.2.1 Base Address Registers (Offset 10h - 24h), states:
"Any device that has a range that behaves like normal memory should mark the range as prefetchable. A linear frame buffer in a graphics device is an example of a range that should be marked prefetchable."Does not backing memory allocated for a specific BAR using
dma_alloc_coherent()on the EP side behave like normal memory from the host's point of view?On the host side, this will mean that the host driver sees the Prefetchable bit, and as according to:
https://docs.kernel.org/driver-api/device-io.html
The host might map the BAR usingioremap_wc().
Quoted 8 lines
Looking specifically at
drivers/misc/pci_endpoint_test.c, it maps the BARs usingpci_ioremap_bar():
https://elixir.bootlin.com/linux/v6.8/source/drivers/pci/pci.c#L252
which will not map it usingioremap_wc().
(But the code we have in the PCI epf-core must of course work with host side drivers other thanpci_endpoint_test.cas well.)
Right. I don't see any problem with the host side assumption. But my question is, is it OK to advertise the coherent memory allocated on the endpoint as prefetchable to the host.
As you quoted the spec,
"Any device that has a range that behaves like normal memory should mark the range as prefetchable."
Here, the coherent memory allocated by the device(endpoint) won't behave as a normal memory on the _endpoint_. But I'm certainly not sure if there are any implications in exposing this memory as a 'normal memory' to the host.
- Mani
Quoted 37 lines
Quoted 17 lines
- The CPU on the host side ususally uses
ioremap()for mapping the PCI BAR of the device. If the BAR is marked as prefetchable, we usually allow mapping it usingioremap_wc()for write-combining orioremap_wt()for a write-through mappings that allow both write-combining and prefetching. On some architectures, these all fall back to normal register mappings which do none of these. If it uses write-combining or prefetching, the host side driver will need to manually serialize against concurrent access from the endpoint side.- The endpoint device accessing a buffer in memory is controlled by the endpoint driver and may decide to prefetch data into a local cache independent of the other two. I don't know if any of the suppored endpoint devices actually do that. A prefetch from the PCI host side would appear as a normal transaction here.
Quoted 3 lines
- The local CPU on the endpoint side may access the same buffer as the endpoint device. On low-end SoCs the DMA from the PCI endpoint is not coherent with the CPU caches, so the CPU may
I don't follow. When doing DMA *from* the endpoint, then the DMA HW on the EP side will read or write data to a buffer allocated on the host side (most likely using
dma_alloc_coherent()), but what does that got to do with how the EP configures the BARs that it exposes?Quoted 6 lines
need to map it as uncacheable to allow data consistency with a the CPU on the PCI host side. On higher-end SoCs (e.g. most non-ARM ones) DMA is coherent with the caches, so the CPU on the endpoint side may map the buffer as cached and still be coherent with a CPU on the PCI host side that has mapped it with
ioremap().Kind regards,
Niklas
Signature · 1 line
மணிவண்ணன் சதாசிவம்
Re: [PATCH v3 9/9] PCI: endpoint: Set prefetch when allocating memory for 64-bit BARs
In reply to Niklas CasselQuoted 17 lines · Niklas Cassel
Quoted 4 lines · Arnd Bergmann
On Fri, Mar 15, 2024, at 07:44, Manivannan Sadhasivam wrote:
I think there are three separate questions here when talking about a scenario where a PCI master accesses memory behind a PCI endpoint:
I think the question is if the PCI epf-core, which runs on the endpoint side, and which calls
dma_alloc_coherent()to allocate backing memory for a BAR, can set/mark the Prefetchable bit for the BAR (if we also set/mark the BAR as a 64-bit BAR).The PCIe 6.0 spec, 7.5.1.2.1 Base Address Registers (Offset 10h - 24h), states:
"Any device that has a range that behaves like normal memory should mark the range as prefetchable. A linear frame buffer in a graphics device is an example of a range that should be marked prefetchable."
Quoted 3 lines
Does not backing memory allocated for a specific BAR using
dma_alloc_coherent()on the EP side behave like normal memory from the host's point of view?
I'm not sure I follow this logic: If the device wants the buffer to act like "normal memory", then it can be marked as prefetchable and mapped into the host as write-combining, but I think in this case you *don't* want it to be coherent on the endpoint side either but use a streaming mapping with explicit cache management instead.
Conversely, if the endpoint side requires a coherent mapping, then I think you will want a strictly ordered (non-wc, non-frefetchable) mapping on the host side as well.
It would be helpful to have actual endpoint function drivers in the kernel rather than just the test drivers to see what type of serialization you actually want for best performance on both sides.
Can you give a specific example of an endpoint that you are actually interested in, maybe just one that we have a host-side device driver for in tree?
Quoted 11 lines
On the host side, this will mean that the host driver sees the Prefetchable bit, and as according to:
https://docs.kernel.org/driver-api/device-io.html
The host might map the BAR usingioremap_wc().Looking specifically at
drivers/misc/pci_endpoint_test.c, it maps the BARs usingpci_ioremap_bar():
https://elixir.bootlin.com/linux/v6.8/source/drivers/pci/pci.c#L252
which will not map it usingioremap_wc().
(But the code we have in the PCI epf-core must of course work with host side drivers other thanpci_endpoint_test.cas well.)
It is to some degree architecture specific here. On powerpc and i386 with MTTRs, any prefetchable BAR will be mapped as write-combining IIRC, but on arm and arm64 it only depends on whether the host side driver uses ioremap() or ioremap_wc().
Quoted 8 lines
Quoted 3 lines
- The local CPU on the endpoint side may access the same buffer as the endpoint device. On low-end SoCs the DMA from the PCI endpoint is not coherent with the CPU caches, so the CPU may
I don't follow. When doing DMA *from* the endpoint, then the DMA HW on the EP side will read or write data to a buffer allocated on the host side (most likely using
dma_alloc_coherent()), but what does that got to do with how the EP configures the BARs that it exposes?
I meant doing DMA to the memory of the endpoint side, not the host side. DMA to the host side memory is completely separate from this question.
Arnd
Re: [PATCH v3 9/9] PCI: endpoint: Set prefetch when allocating memory for 64-bit BARs
Hello Arnd,
Quoted 8 lines · Arnd Bergmann
I'm not sure I follow this logic: If the device wants the buffer to act like "normal memory", then it can be marked as prefetchable and mapped into the host as write-combining, but I think in this case you *don't* want it to be coherent on the endpoint side either but use a streaming mapping with explicit cache management instead.
Quoted 8 lines
Conversely, if the endpoint side requires a coherent mapping, then I think you will want a strictly ordered (non-wc, non-frefetchable) mapping on the host side as well.
It would be helpful to have actual endpoint function drivers in the kernel rather than just the test drivers to see what type of serialization you actually want for best performance on both sides.
Yes, that would be nice.
This specific API, pci_epf_alloc_space(), is only used by the following drivers:drivers/pci/endpoint/functions/pci-epf-test.c drivers/pci/endpoint/functions/pci-epf-ntb.c drivers/pci/endpoint/functions/pci-epf-vntb.c
pci_epf_alloc_space() is only used to allocate backing memory for the BARs.
Quoted 4 lines
Can you give a specific example of an endpoint that you are actually interested in, maybe just one that we have a host-side device driver for in tree?
I personally just care about pci-epf-test, but obviously I don't want to regress any other user of pci_epf_alloc_space().
Looking at the endpoint side driver:drivers/pci/endpoint/functions/pci-epf-test.c
and the host side driver:drivers/misc/pci_endpoint_test.c
On the RC side, allocating buffers that the EP will DMA to is done using: kzalloc() + dma_map_single().
On EP side:drivers/pci/endpoint/functions/pci-epf-test.c
uses dma_map_single() when using DMA, and signals completion using MSI.
On EP side:
When reading/writing to the BARs, it simply does:
READ_ONCE()/WRITE_ONCE():
https://github.com/torvalds/linux/blob/v6.8/drivers/pci/endpoint/functions/pci-epf-test.c#L643-L648
There is no dma_sync(), so the pci-test-epf driver currently seems to depend on the backing memory being allocated by dma_alloc_coherent().
Quoted 5 lines
If you don't care about ordering on that level, I would use
dma_map_sg()on the endpoint side and prefetchable mapping on the host side, with the endpoint using dma_sync_*() to pass buffer ownership between the two sides, as controlled by some other communication method (non-prefetchable BAR, MSI, ...).
I don't think that there is no big reason why pci-epf-test is implemented using dma_alloc_coherent() rather than dma_sync() for the memory backing the BARs, but that is the way it is.
Since I don't feel like totally rewriting pci-epf-test, and since you say that we shouldn't use dma_alloc_coherent() for the memory backing the BARs together with exporting the BAR as prefetchable, I will drop this patch from the series in the next revision.
Kind regards,
Niklas
Re: [PATCH v3 9/9] PCI: endpoint: Set prefetch when allocating memory for 64-bit BARs
Quoted 17 lines · Niklas Cassel
On Mon, Mar 18, 2024 at 08:25:36AM +0100, Arnd Bergmann wrote:
I personally just care about pci-epf-test, but obviously I don't want to regress any other user of
pci_epf_alloc_space().Looking at the endpoint side driver:
drivers/pci/endpoint/functions/pci-epf-test.c
and the host side driver:drivers/misc/pci_endpoint_test.cOn the RC side, allocating buffers that the EP will DMA to is done using:
kzalloc()+dma_map_single().On EP side:
drivers/pci/endpoint/functions/pci-epf-test.c
usesdma_map_single()when using DMA, and signals completion using MSI.
Quoted 7 lines
On EP side:
When reading/writing to the BARs, it simply does:
READ_ONCE()/WRITE_ONCE():
https://github.com/torvalds/linux/blob/v6.8/drivers/pci/endpoint/functions/pci-epf-test.c#L643-L648There is no
dma_sync(), so the pci-test-epf driver currently seems to depend on the backing memory being allocated bydma_alloc_coherent().
From my reading of that function, this is really some kind of command buffer that implements individual structured registers and can be accessed from both sides at the same time, so it would not actually make sense with the streaming interface and wc/prefetchable access in place of explicit READ_ONCE/WRITE_ONCE and readl/writel accesses.
Quoted 6 lines
Quoted 5 lines
If you don't care about ordering on that level, I would use
dma_map_sg()on the endpoint side and prefetchable mapping on the host side, with the endpoint using dma_sync_*() to pass buffer ownership between the two sides, as controlled by some other communication method (non-prefetchable BAR, MSI, ...).
Quoted 8 lines
I don't think that there is no big reason why pci-epf-test is implemented using
dma_alloc_coherent()rather thandma_sync()for the memory backing the BARs, but that is the way it is.Since I don't feel like totally rewriting pci-epf-test, and since you say that we shouldn't use
dma_alloc_coherent()for the memory backing the BARs together with exporting the BAR as prefetchable, I will drop this patch from the series in the next revision.
Ok. It might still be useful to extend the driver to also allow transferring streaming data through a BAR on the endpoint side. From what I can tell, it currently supports using either slave DMA or a RC side buffer that ioremapped into the endpoint, but that uses a regular ioremap() as well. Mapping the RC side buffer as WC should make it possible to transfer data from EP to RC more efficiently, but for the RC to EP transfers you really want the buffer to be allocated on the EP, so you can ioremap_wc() it to the RC for a memcpy_toio, or cacheable read from the EP.
Arnd
Re: [PATCH v3 9/9] PCI: endpoint: Set prefetch when allocating memory for 64-bit BARs
Quoted 26 lines · Arnd Bergmann
Quoted 17 lines · Niklas Cassel
On Mon, Mar 18, 2024 at 08:25:36AM +0100, Arnd Bergmann wrote:
I personally just care about pci-epf-test, but obviously I don't want to regress any other user of
pci_epf_alloc_space().Looking at the endpoint side driver:
drivers/pci/endpoint/functions/pci-epf-test.c
and the host side driver:drivers/misc/pci_endpoint_test.cOn the RC side, allocating buffers that the EP will DMA to is done using:
kzalloc()+dma_map_single().On EP side:
drivers/pci/endpoint/functions/pci-epf-test.c
usesdma_map_single()when using DMA, and signals completion using MSI.Quoted 7 lines
On EP side:
When reading/writing to the BARs, it simply does:
READ_ONCE()/WRITE_ONCE():
https://github.com/torvalds/linux/blob/v6.8/drivers/pci/endpoint/functions/pci-epf-test.c#L643-L648There is no
dma_sync(), so the pci-test-epf driver currently seems to depend on the backing memory being allocated bydma_alloc_coherent().
Quoted 7 lines
From my reading of that function, this is really some kind of command buffer that implements individual structured registers and can be accessed from both sides at the same time, so it would not actually make sense with the streaming interface and wc/prefetchable access in place of explicit READ_ONCE/WRITE_ONCE and readl/writel accesses.
Right. We should stick to the current implementation for now until a function driver with streaming DMA usecase comes in.
- Mani
Quoted 27 lines
Quoted 6 lines
Quoted 5 lines
If you don't care about ordering on that level, I would use
dma_map_sg()on the endpoint side and prefetchable mapping on the host side, with the endpoint using dma_sync_*() to pass buffer ownership between the two sides, as controlled by some other communication method (non-prefetchable BAR, MSI, ...).Quoted 8 lines
I don't think that there is no big reason why pci-epf-test is implemented using
dma_alloc_coherent()rather thandma_sync()for the memory backing the BARs, but that is the way it is.Since I don't feel like totally rewriting pci-epf-test, and since you say that we shouldn't use
dma_alloc_coherent()for the memory backing the BARs together with exporting the BAR as prefetchable, I will drop this patch from the series in the next revision.Ok. It might still be useful to extend the driver to also allow transferring streaming data through a BAR on the endpoint side. From what I can tell, it currently supports using either slave DMA or a RC side buffer that ioremapped into the endpoint, but that uses a regular
ioremap()as well. Mapping the RC side buffer as WC should make it possible to transfer data from EP to RC more efficiently, but for the RC to EP transfers you really want the buffer to be allocated on the EP, so you canioremap_wc()it to the RC for amemcpy_toio, or cacheable read from the EP.Arnd
Signature · 1 line
மணிவண்ணன் சதாசிவம்
Re: [PATCH v3 9/9] PCI: endpoint: Set prefetch when allocating memory for 64-bit BARs
In reply to Arnd BergmannQuoted 47 lines · Arnd Bergmann
Quoted 20 lines · Manivannan Sadhasivam
Quoted 9 lines · Niklas Cassel
"Generally only 64-bit BARs are good candidates, since only Legacy Endpoints are permitted to set the Prefetchable bit in 32-bit BARs, and most scalable platforms map all 32-bit Memory BARs into non-prefetchable Memory Space regardless of the Prefetchable bit value."
"For a PCI Express Endpoint, 64-bit addressing must be supported for all BARs that have the Prefetchable bit Set. 32-bit addressing is permitted for all BARs that do not have the Prefetchable bit Set."
Quoted 9 lines
"Any device that has a range that behaves like normal memory should mark the range as prefetchable. A linear frame buffer in a graphics device is an example of a range that should be marked prefetchable."
The PCIe spec tells us that we should have the prefetchable bit set for 64-bit BARs backed by "normal memory". The backing memory that we allocate for a 64-bit BAR using
pci_epf_alloc_space()(which callsdma_alloc_coherent()) is obviously "normal memory".Quoted 6 lines
I'm not sure this is correct. Memory returned by '
dma_alloc_coherent' is not the 'normal memory' but rather 'consistent/coherent memory'. Here the question is, can the memory returned bydma_alloc_coherent()be prefetched or write-combined on all architectures.I hope Arnd can answer this question.
I think there are three separate questions here when talking about a scenario where a PCI master accesses memory behind a PCI endpoint:
- The CPU on the host side ususally uses
ioremap()for mapping the PCI BAR of the device. If the BAR is marked as prefetchable, we usually allow mapping it usingioremap_wc()for write-combining orioremap_wt()for a write-through mappings that allow both write-combining and prefetching. On some architectures, these all fall back to normal register mappings which do none of these. If it uses write-combining or prefetching, the host side driver will need to manually serialize against concurrent access from the endpoint side.- The endpoint device accessing a buffer in memory is controlled by the endpoint driver and may decide to prefetch data into a local cache independent of the other two. I don't know if any of the suppored endpoint devices actually do that. A prefetch from the PCI host side would appear as a normal transaction here.
Quoted 10 lines
- The local CPU on the endpoint side may access the same buffer as the endpoint device. On low-end SoCs the DMA from the PCI endpoint is not coherent with the CPU caches, so the CPU may need to map it as uncacheable to allow data consistency with a the CPU on the PCI host side. On higher-end SoCs (e.g. most non-ARM ones) DMA is coherent with the caches, so the CPU on the endpoint side may map the buffer as cached and still be coherent with a CPU on the PCI host side that has mapped it with
ioremap().
Thanks Arnd for the reply.
But I'm not sure I got the answer I was looking for. So let me rephrase my question a bit.
For BAR memory, PCIe spec states that,
'A PCI Express Function requesting Memory Space through a BAR must set the BAR's Prefetchable bit unless the range contains locations with read side effects or locations in which the Function does not tolerate write merging'
So here, spec refers the backing memory allocated on the endpoint side as the 'range' i.e, the BAR memory allocated on the host that gets mapped on the endpoint.
Currently on the endpoint side, we use dma_alloc_coherent() to allocate the memory for each BAR and map it using iATU.
So I want to know if the memory range allocated in the endpoint through dma_alloc_coherent() satisfies the above two conditions in PCIe spec on all architectures:
1. No Read side effects
2. Tolerates write merging
I believe the reason why we are allocating the coherent memory on the endpoint first up is not all PCIe controllers are DMA coherent as you said above.
- Mani
Signature · 1 line
மணிவண்ணன் சதாசிவம்
Re: [PATCH v3 9/9] PCI: endpoint: Set prefetch when allocating memory for 64-bit BARs
Quoted 24 lines · Manivannan Sadhasivam
Quoted 2 lines · Arnd Bergmann
Quoted 1 line · Manivannan Sadhasivam
On Wed, Mar 13, 2024 at 11:58:01AM +0100, Niklas Cassel wrote:
But I'm not sure I got the answer I was looking for. So let me rephrase my question a bit.
For BAR memory, PCIe spec states that,
'A PCI Express Function requesting Memory Space through a BAR must set the BAR's Prefetchable bit unless the range contains locations with read side effects or locations in which the Function does not tolerate write merging'
So here, spec refers the backing memory allocated on the endpoint side as the 'range' i.e, the BAR memory allocated on the host that gets mapped on the endpoint.
Currently on the endpoint side, we use
dma_alloc_coherent()to allocate the memory for each BAR and map it using iATU.So I want to know if the memory range allocated in the endpoint through
dma_alloc_coherent()satisfies the above two conditions in PCIe spec on all architectures:
Quoted 5 lines
1. No Read side effects
2. Tolerates write mergingI believe the reason why we are allocating the coherent memory on the endpoint first up is not all PCIe controllers are DMA coherent as you said above.
As far as I can tell, we never have read side effects for memory backed BARs, but the write merging is something that depends on how the memory is used:
If you have anything in that memory that relies on ordering, you probably want to map it as coherent on the endpoint side, and non-prefetchable on the host controller side, and then use the normal rmb()/wmb() barriers on both ends between serialized accesses. An example of this would be having blocks of data separate from metadata that says whether the data is valid.
If you don't care about ordering on that level, I would use dma_map_sg() on the endpoint side and prefetchable mapping on the host side, with the endpoint using dma_sync_*() to pass buffer ownership between the two sides, as controlled by some other communication method (non-prefetchable BAR, MSI, ...).
Arnd
Re: [PATCH v3 9/9] PCI: endpoint: Set prefetch when allocating memory for 64-bit BARs
Quoted 43 lines · Arnd Bergmann
Quoted 24 lines · Manivannan Sadhasivam
Quoted 2 lines · Arnd Bergmann
Quoted 1 line · Manivannan Sadhasivam
On Wed, Mar 13, 2024 at 11:58:01AM +0100, Niklas Cassel wrote:
But I'm not sure I got the answer I was looking for. So let me rephrase my question a bit.
For BAR memory, PCIe spec states that,
'A PCI Express Function requesting Memory Space through a BAR must set the BAR's Prefetchable bit unless the range contains locations with read side effects or locations in which the Function does not tolerate write merging'
So here, spec refers the backing memory allocated on the endpoint side as the 'range' i.e, the BAR memory allocated on the host that gets mapped on the endpoint.
Currently on the endpoint side, we use
dma_alloc_coherent()to allocate the memory for each BAR and map it using iATU.So I want to know if the memory range allocated in the endpoint through
dma_alloc_coherent()satisfies the above two conditions in PCIe spec on all architectures:Quoted 5 lines
1. No Read side effects
2. Tolerates write mergingI believe the reason why we are allocating the coherent memory on the endpoint first up is not all PCIe controllers are DMA coherent as you said above.
As far as I can tell, we never have read side effects for memory backed BARs, but the write merging is something that depends on how the memory is used:
If you have anything in that memory that relies on ordering, you probably want to map it as coherent on the endpoint side, and non-prefetchable on the host controller side, and then use the normal rmb()/wmb() barriers on both ends between serialized accesses. An example of this would be having blocks of data separate from metadata that says whether the data is valid.
Quoted 6 lines
If you don't care about ordering on that level, I would use
dma_map_sg()on the endpoint side and prefetchable mapping on the host side, with the endpoint using dma_sync_*() to pass buffer ownership between the two sides, as controlled by some other communication method (non-prefetchable BAR, MSI, ...).
Right now, there are only Test and a couple of NTB drivers making use of the pci_epf_alloc_space() API and they do not need streaming DMA.
So to conclude, we should just live with coherent allocation/non-prefetch for now and extend it to streaming DMA/prefetch once we have a function driver that needs it.
Thanks a lot for your inputs!
- Mani
Signature · 1 line
மணிவண்ணன் சதாசிவம்