Implement NTB Controller using multiple PCI EP
Subsystems (7)
v2 cover Implement NTB Controller using multiple PCI EP
This series is about implementing SW defined NTB using multiple endpoint instances. This series has been tested using 2 endpoint instances in J7 connected to two DRA7 boards. However there is nothing platform specific for the NTB functionality.
This was presented in Linux Plumbers Conference. The presentation can be found @ [1]
RFC patch series can be found @ [2]
v1 patch series can be found @ [3]
This series has been validated after applying [4]
Changes from v1:
1) As per Rob's comment, removed support for creating NTB function device from DT
2) Add support to create NTB EPF device using configfs (added support in configfs to associate primary and secondary EPC with EPF.
Changes from RFC:
1) Converted the DT binding patches to YAML schema and merged the DT binding patches together
2) NTB documentation is converted to .rst
3) One HOST can now interrupt the other HOST using MSI-X interrupts
4) Added support for teardown of memory window and doorbell configuration
5) Add support to provide support 64-bit memory window size from DT
[1] -> https://www.linuxplumbersconf.org/event/4/contributions/395/attachments/284/481/Implementing_NTB_Controller_Using_PCIe_Endpoint_-_final.pdf
[2] -> http://lore.kernel.org/r/[email protected]
[3] -> http://lore.kernel.org/r/[email protected]
[4] -> http://lore.kernel.org/r/[email protected]
Kishon Vijay Abraham I (14):
Documentation: PCI: Add specification for the *PCI NTB* function
device
PCI: endpoint: Make *_get_first_free_bar() take into account 64 bit
BAR
PCI: endpoint: Add helper API to get the 'next' unreserved BAR
PCI: endpoint: Make *_free_bar() to return error codes on failure
PCI: endpoint: Remove unused pci_epf_match_device()
PCI: endpoint: Add support to associate secondary EPC with EPF
PCI: endpoint: Add support in configfs to associate two EPCs with EPF
PCI: endpoint: Add pci_epc_ops to map MSI irq
PCI: cadence: Implement ->msi_map_irq() ops
PCI: endpoint: Add EP function driver to provide NTB functionality
PCI: Add TI J721E device to pci ids
NTB: Add support for EPF PCI-Express Non-Transparent Bridge
NTB: tool: Enable the NTB/PCIe link on the local or remote side of
bridge
Documentation: PCI: Add userguide for PCI endpoint NTB function
Documentation/PCI/endpoint/index.rst | 2 + .../PCI/endpoint/pci-endpoint-cfs.rst | 10 + .../PCI/endpoint/pci-ntb-function.rst | 344 +++ Documentation/PCI/endpoint/pci-ntb-howto.rst | 141 ++ drivers/misc/pci_endpoint_test.c | 1 - drivers/ntb/hw/Kconfig | 1 + drivers/ntb/hw/Makefile | 1 + drivers/ntb/hw/epf/Kconfig | 6 + drivers/ntb/hw/epf/Makefile | 1 + drivers/ntb/hw/epf/ntb_hw_epf.c | 750 ++++++ drivers/ntb/test/ntb_tool.c | 1 + .../pci/controller/cadence/pcie-cadence-ep.c | 50 + drivers/pci/endpoint/functions/Kconfig | 12 + drivers/pci/endpoint/functions/Makefile | 1 + drivers/pci/endpoint/functions/pci-epf-ntb.c | 2151 +++++++++++++++++ drivers/pci/endpoint/functions/pci-epf-test.c | 13 +- drivers/pci/endpoint/pci-ep-cfs.c | 154 +- drivers/pci/endpoint/pci-epc-core.c | 131 +- drivers/pci/endpoint/pci-epf-core.c | 73 +- include/linux/pci-epc.h | 38 +- include/linux/pci-epf.h | 23 +- include/linux/pci_ids.h | 1 + 22 files changed, 3838 insertions(+), 67 deletions(-) create mode 100644 Documentation/PCI/endpoint/pci-ntb-function.rst create mode 100644 Documentation/PCI/endpoint/pci-ntb-howto.rst create mode 100644 drivers/ntb/hw/epf/Kconfig create mode 100644 drivers/ntb/hw/epf/Makefile create mode 100644 drivers/ntb/hw/epf/ntb_hw_epf.c create mode 100644 drivers/pci/endpoint/functions/pci-epf-ntb.c
Signature · 1 line
2.17.1
PATCH v2 1/14 Documentation: PCI: Add specification for the *PCI NTB* function device
Add specification for the *PCI NTB* function device. The endpoint function driver and the host PCI driver should be created based on this specification.
--- Documentation/PCI/endpoint/index.rst | 1 + .../PCI/endpoint/pci-ntb-function.rst | 344 ++++++++++++++++++ 2 files changed, 345 insertions(+) create mode 100644 Documentation/PCI/endpoint/pci-ntb-function.rst
Documentation/PCI/endpoint/index.rst +1 -0
@@ -11,3 +11,4 @@ PCI Endpoint Framework pci-endpoint-cfs pci-test-function pci-test-howto + pci-ntb-function
Documentation/PCI/endpoint/pci-ntb-function.rst new file +344 -0
344 lines not on this page · raw message
Signature · 1 line
2.17.1
Re: [PATCH v2 01/14] Documentation: PCI: Add specification for the *PCI NTB* function device
Documentation/PCI/endpoint/pci-ntb-function.rst line 11
Quoted 13 lines · Kishon Vijay Abraham I
+++
b/Documentation/PCI/endpoint/pci-ntb-function.rstDocumentation/PCI/endpoint/pci-ntb-function.rst new file +11
-0@@ -0,0 +1,344 @@ +.. SPDX-License-Identifier: GPL-2.0 + +================= +PCI NTB Function +================= + +:Author: Kishon Vijay Abraham I <[email protected]> + +PCI NTB Function allows two different systems (or hosts) to communicate +with each other by configurig the endpoint instances in such a way that +transactions from one system is routed to the other system.
At no point in this document do you expand "NTB" into Non-Transparent Bridge. The above paragraph probably also needs to say something like "By making each host appear as a device to the other host". Although maybe that's not entirely accurate? It's been a few years since I last played with NTBs.
So how about the following opening paragraph:
PCI Non Transparent Bridges (NTB) allow two host systems to communicate with each other by exposing each host as a device to the other host. NTBs typically support the ability to generate interrupts on the remote machine, expose memory ranges as BARs and perform DMA. They also support scratchpads which are areas of memory within the NTB that are accessible from both machines.
... feel free to fix that up if my memory is out of date or corrupted.
Re: [PATCH v2 01/14] Documentation: PCI: Add specification for the *PCI NTB* function device
Hi Matthew,
Quoted 23 lines · Matthew Wilcox
Documentation/PCI/endpoint/pci-ntb-function.rst line 11
Quoted 13 lines · Kishon Vijay Abraham I
+++
b/Documentation/PCI/endpoint/pci-ntb-function.rstDocumentation/PCI/endpoint/pci-ntb-function.rst new file +11
-0@@ -0,0 +1,344 @@ +.. SPDX-License-Identifier: GPL-2.0 + +================= +PCI NTB Function +================= + +:Author: Kishon Vijay Abraham I <[email protected]> + +PCI NTB Function allows two different systems (or hosts) to communicate +with each other by configurig the endpoint instances in such a way that +transactions from one system is routed to the other system.At no point in this document do you expand "NTB" into Non-Transparent Bridge. The above paragraph probably also needs to say something like "By making each host appear as a device to the other host". Although maybe that's not entirely accurate? It's been a few years since I last played with NTBs.
So how about the following opening paragraph:
Quoted 8 lines
PCI Non Transparent Bridges (NTB) allow two host systems to communicate with each other by exposing each host as a device to the other host. NTBs typically support the ability to generate interrupts on the remote machine, expose memory ranges as BARs and perform DMA. They also support scratchpads which are areas of memory within the NTB that are accessible from both machines.
... feel free to fix that up if my memory is out of date or corrupted.
I think that's accurate. I'll wait for review comments on the rest of the series and I'll fix this one in my next revision.
Thanks
Kishon
PATCH v2 2/14 PCI: endpoint: Make *_get_first_free_bar() take into account 64 bit BAR
In reply to Kishon Vijay Abraham Ipci_epc_get_first_free_bar() uses only "reserved_bar" member in epc_features to get the first unreserved BAR. However if the reserved BAR is also a 64-bit BAR, then the next BAR shouldn't be returned (since 64-bit BAR uses two BARs).
Make pci_epc_get_first_free_bar() take into account 64 bit BAR while returning the first free unreserved BAR.
--- drivers/pci/endpoint/pci-epc-core.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-)
drivers/pci/endpoint/pci-epc-core.c +10 -2
@@ -93,12 +93,20 @@ EXPORT_SYMBOL_GPL(pci_epc_get);unsigned int pci_epc_get_first_free_bar (const struct pci_epc_features *epc_features) { -int free_bar; +unsigned long free_bar;if (!epc_features)return 0 ; - free_bar =ffz (epc_features->reserved_bar); +/* Find if the reserved BAR is also a 64-bit BAR */ + free_bar = epc_features->reserved_bar & epc_features->bar_fixed_64bit; + +/* Set the adjacent bit if the reserved BAR is also a 64-bit BAR */ + free_bar <<=1 ; + free_bar |= epc_features->reserved_bar; + +/* Now find the free BAR */ + free_bar =ffz (free_bar);if (free_bar >5 )return 0 ;
Signature · 1 line
2.17.1
PATCH v2 3/14 PCI: endpoint: Add helper API to get the 'next' unreserved BAR
In reply to Kishon Vijay Abraham IAdd an API to get the next unreserved BAR starting from a given BAR number that can be used by the endpoint function.
--- drivers/pci/endpoint/pci-epc-core.c | 26 ++++++++++++++++++++++---- include/linux/pci-epc.h | 2 ++ 2 files changed, 24 insertions(+), 4 deletions(-)
drivers/pci/endpoint/pci-epc-core.c +22 -4
@@ -87,17 +87,36 @@ EXPORT_SYMBOL_GPL(pci_epc_get);* pci_epc_get_first_free_bar() - helper to get first unreserved BAR * @epc_features: pci_epc_features structure that holds the reserved bar bitmap * -* Invoke to get the first unreserved BAR that can be used for endpoint +* Invoke to get the first unreserved BAR that can be used by the endpoint * function. For any incorrect value in reserved_bar return '0'. */ unsigned int pci_epc_get_first_free_bar (const struct pci_epc_features *epc_features) +{ +return pci_epc_get_next_free_bar (epc_features,BAR_0 ); +} +EXPORT_SYMBOL_GPL (pci_epc_get_first_free_bar); + +/** +* pci_epc_get_next_free_bar() - helper to get unreserved BAR starting from @bar +* @epc_features: pci_epc_features structure that holds the reserved bar bitmap +* @bar: the starting BAR number from where unreserved BAR should be searched +* +* Invoke to get the next unreserved BAR starting from @bar that can be used +* for endpoint function. For any incorrect value in reserved_bar return '0'. +*/ +unsigned int pci_epc_get_next_free_bar (const struct pci_epc_features + *epc_features,enum pci_barno bar) {unsigned long free_bar;if (!epc_features)return 0 ; +/* If 'bar - 1' is a 64-bit BAR, move to the next BAR */ +if ((epc_features->bar_fixed_64bit <<1 ) &1 << bar) + bar++; +/* Find if the reserved BAR is also a 64-bit BAR */ free_bar = epc_features->reserved_bar & epc_features->bar_fixed_64bit;
@@ -105,14 +124,13 @@ unsigned int pci_epc_get_first_free_bar(const struct pci_epc_features free_bar <<=1 ; free_bar |= epc_features->reserved_bar; -/* Now find the free BAR */ - free_bar =ffz (free_bar); + free_bar =find_next_zero_bit (&free_bar,6 , bar);if (free_bar >5 )return 0 ;return free_bar; } -EXPORT_SYMBOL_GPL (pci_epc_get_first_free_bar); +EXPORT_SYMBOL_GPL (pci_epc_get_next_free_bar);/** * pci_epc_get_features() - get the features supported by EPC
include/linux/pci-epc.h +2 -0
@@ -203,6 +203,8 @@ const struct pci_epc_features *pci_epc_get_features(struct pci_epc *epc, u8 func_no);unsigned int pci_epc_get_first_free_bar (const struct pci_epc_features *epc_features); +unsigned int pci_epc_get_next_free_bar (const struct pci_epc_features + *epc_features,enum pci_barno bar);struct pci_epc *pci_epc_get (const char *epc_name);void pci_epc_put (struct pci_epc *epc);
Signature · 1 line
2.17.1
PATCH v2 4/14 PCI: endpoint: Make *_free_bar() to return error codes on failure
In reply to Kishon Vijay Abraham IModify pci_epc_get_next_free_bar() and pci_epc_get_first_free_bar() to return error values if there are no free BARs available.
--- drivers/pci/endpoint/functions/pci-epf-test.c | 2 ++ drivers/pci/endpoint/pci-epc-core.c | 12 ++++++------ include/linux/pci-epc.h | 8 ++++---- include/linux/pci-epf.h | 1 + 4 files changed, 13 insertions(+), 10 deletions(-)
drivers/pci/endpoint/functions/pci-epf-test.c +2 -0
@@ -834,6 +834,8 @@ static int pci_epf_test_bind(struct pci_epf *epf) linkup_notifier = epc_features->linkup_notifier; core_init_notifier = epc_features->core_init_notifier; test_reg_bar =pci_epc_get_first_free_bar (epc_features); +if (test_reg_bar <0 ) +return -EINVAL ;pci_epf_configure_bar (epf, epc_features); }
drivers/pci/endpoint/pci-epc-core.c +6 -6
@@ -90,8 +90,8 @@ EXPORT_SYMBOL_GPL(pci_epc_get);* Invoke to get the first unreserved BAR that can be used by the endpoint * function. For any incorrect value in reserved_bar return '0'. */ -unsigned int pci_epc_get_first_free_bar (const struct pci_epc_features - *epc_features) +enum pci_barno +pci_epc_get_first_free_bar (const struct pci_epc_features *epc_features) {return pci_epc_get_next_free_bar (epc_features,BAR_0 ); }
@@ -105,13 +105,13 @@ EXPORT_SYMBOL_GPL(pci_epc_get_first_free_bar);* Invoke to get the next unreserved BAR starting from @bar that can be used * for endpoint function. For any incorrect value in reserved_bar return '0'. */ -unsigned int pci_epc_get_next_free_bar (const struct pci_epc_features - *epc_features,enum pci_barno bar) +enum pci_barno pci_epc_get_next_free_bar (const struct pci_epc_features + *epc_features,enum pci_barno bar) {unsigned long free_bar;if (!epc_features) -return 0 ; +return BAR_0 ;/* If 'bar - 1' is a 64-bit BAR, move to the next BAR */ if ((epc_features->bar_fixed_64bit <<1 ) &1 << bar)
@@ -126,7 +126,7 @@ unsigned int pci_epc_get_next_free_bar(const struct pci_epc_features free_bar =find_next_zero_bit (&free_bar,6 , bar);if (free_bar >5 ) -return 0 ; +return NO_BAR ;return free_bar; }
include/linux/pci-epc.h +4 -4
@@ -201,10 +201,10 @@ int pci_epc_start(struct pci_epc *epc);void pci_epc_stop (struct pci_epc *epc);const struct pci_epc_features *pci_epc_get_features (struct pci_epc *epc,u8 func_no); -unsigned int pci_epc_get_first_free_bar (const struct pci_epc_features - *epc_features); -unsigned int pci_epc_get_next_free_bar (const struct pci_epc_features - *epc_features,enum pci_barno bar); +enum pci_barno +pci_epc_get_first_free_bar (const struct pci_epc_features *epc_features); +enum pci_barno pci_epc_get_next_free_bar (const struct pci_epc_features + *epc_features,enum pci_barno bar);struct pci_epc *pci_epc_get (const char *epc_name);void pci_epc_put (struct pci_epc *epc);
include/linux/pci-epf.h +1 -0
@@ -21,6 +21,7 @@ enum pci_notify_event { };enum pci_barno { +NO_BAR =-1 ,BAR_0 ,BAR_1 ,BAR_2 ,
Signature · 1 line
2.17.1
PATCH v2 5/14 PCI: endpoint: Remove unused pci_epf_match_device()
In reply to Kishon Vijay Abraham IRemove unused pci_epf_match_device() function added in pci-epf-core.c
--- drivers/pci/endpoint/pci-epf-core.c | 16 ---------------- include/linux/pci-epf.h | 2 -- 2 files changed, 18 deletions(-)
drivers/pci/endpoint/pci-epf-core.c +0 -16
@@ -280,22 +280,6 @@ struct pci_epf *pci_epf_create(const char *name) }EXPORT_SYMBOL_GPL (pci_epf_create); -const struct pci_epf_device_id * -pci_epf_match_device (const struct pci_epf_device_id *id,struct pci_epf *epf) -{ -if (!id || !epf) -return NULL ; - -while (*id->name) { -if (strcmp (epf->name, id->name) ==0 ) -return id; - id++; - } - -return NULL ; -} -EXPORT_SYMBOL_GPL (pci_epf_match_device); -static void pci_epf_dev_release (struct device *dev) {struct pci_epf *epf =to_pci_epf (dev);
include/linux/pci-epf.h +0 -2
@@ -165,8 +165,6 @@ static inline void *epf_get_drvdata(struct pci_epf *epf)return dev_get_drvdata (&epf->dev); } -const struct pci_epf_device_id * -pci_epf_match_device (const struct pci_epf_device_id *id,struct pci_epf *epf);struct pci_epf *pci_epf_create (const char *name);void pci_epf_destroy (struct pci_epf *epf); int __pci_epf_register_driver(struct pci_epf_driver *driver,
Signature · 1 line
2.17.1
PATCH v2 6/14 PCI: endpoint: Add support to associate secondary EPC with EPF
In reply to Kishon Vijay Abraham IIn the case of standard endpoint functions, only one endpoint controller (EPC) will be associated with an endpoint function (EPF). However for providing NTB (non transparent bridge) functionality, two EPCs should be associated with a single EPF. Add support to associate secondary EPC with EPF. This is in preparation for adding NTB endpoint function driver.
--- drivers/pci/endpoint/functions/pci-epf-test.c | 11 ++-- drivers/pci/endpoint/pci-ep-cfs.c | 6 +- drivers/pci/endpoint/pci-epc-core.c | 50 ++++++++++++---- drivers/pci/endpoint/pci-epf-core.c | 57 +++++++++++++------ include/linux/pci-epc.h | 25 +++++++- include/linux/pci-epf.h | 17 +++++- 6 files changed, 128 insertions(+), 38 deletions(-)
drivers/pci/endpoint/functions/pci-epf-test.c +7 -4
@@ -619,7 +619,8 @@ static void pci_epf_test_unbind(struct pci_epf *epf)if (epf_test->reg[bar]) {pci_epc_clear_bar (epc, epf->func_no, epf_bar); -pci_epf_free_space (epf, epf_test->reg[bar], bar); +pci_epf_free_space (epf, epf_test->reg[bar], bar, +PRIMARY_INTERFACE ); } } }
@@ -651,7 +652,8 @@ static int pci_epf_test_set_bar(struct pci_epf *epf) ret =pci_epc_set_bar (epc, epf->func_no, epf_bar);if (ret) { -pci_epf_free_space (epf, epf_test->reg[bar], bar); +pci_epf_free_space (epf, epf_test->reg[bar], bar, +PRIMARY_INTERFACE );dev_err (dev,"Failed to set BAR%d\n" , bar);if (bar == test_reg_bar)return ret;
@@ -771,7 +773,7 @@ static int pci_epf_test_alloc_space(struct pci_epf *epf) } base =pci_epf_alloc_space (epf, test_reg_size, test_reg_bar, - epc_features->align); + epc_features->align,PRIMARY_INTERFACE );if (!base) {dev_err (dev,"Failed to allocated register space\n" );return -ENOMEM ;
@@ -789,7 +791,8 @@ static int pci_epf_test_alloc_space(struct pci_epf *epf)continue ; base =pci_epf_alloc_space (epf, bar_size[bar], bar, - epc_features->align); + epc_features->align, +PRIMARY_INTERFACE );if (!base)dev_err (dev,"Failed to allocate space for BAR%d\n" , bar);
drivers/pci/endpoint/pci-ep-cfs.c +3 -3
@@ -94,13 +94,13 @@ static int pci_epc_epf_link(struct config_item *epc_item,struct pci_epc *epc = epc_group->epc;struct pci_epf *epf = epf_group->epf; - ret =pci_epc_add_epf (epc, epf); + ret =pci_epc_add_epf (epc, epf,PRIMARY_INTERFACE );if (ret)return ret; ret =pci_epf_bind (epf);if (ret) { -pci_epc_remove_epf (epc, epf); +pci_epc_remove_epf (epc, epf,PRIMARY_INTERFACE );return ret; }
@@ -120,7 +120,7 @@ static void pci_epc_epf_unlink(struct config_item *epc_item, epc = epc_group->epc; epf = epf_group->epf;pci_epf_unbind (epf); -pci_epc_remove_epf (epc, epf); +pci_epc_remove_epf (epc, epf,PRIMARY_INTERFACE ); }static struct configfs_item_operations pci_epc_item_ops = {
drivers/pci/endpoint/pci-epc-core.c +39 -11
86 lines not on this page · raw message
drivers/pci/endpoint/pci-epf-core.c +41 -16
97 lines not on this page · raw message
include/linux/pci-epc.h +23 -2
@@ -13,6 +13,12 @@struct pci_epc ; +enum pci_epc_interface_type { +UNKNOWN_INTERFACE =-1 , +PRIMARY_INTERFACE , +SECONDARY_INTERFACE , +}; +enum pci_epc_irq_type {PCI_EPC_IRQ_UNKNOWN ,PCI_EPC_IRQ_LEGACY ,
@@ -20,6 +26,19 @@ enum pci_epc_irq_type { PCI_EPC_IRQ_MSIX, }; +static inline const char * +pci_epc_interface_string (enum pci_epc_interface_type type) +{ +switch (type) { +case PRIMARY_INTERFACE : +return "primary" ; +case SECONDARY_INTERFACE : +return "secondary" ; +default : +return "UNKNOWN interface" ; + } +} +/** * struct pci_epc_ops - set of function pointers for performing EPC operations * @write_header: ops to populate configuration space header
@@ -175,10 +194,12 @@ __pci_epc_create(struct device *dev, const struct pci_epc_ops *ops, struct module *owner);void devm_pci_epc_destroy (struct device *dev,struct pci_epc *epc);void pci_epc_destroy (struct pci_epc *epc); -int pci_epc_add_epf (struct pci_epc *epc,struct pci_epf *epf); +int pci_epc_add_epf (struct pci_epc *epc,struct pci_epf *epf, +enum pci_epc_interface_type type);void pci_epc_linkup (struct pci_epc *epc);void pci_epc_init_notify (struct pci_epc *epc); -void pci_epc_remove_epf (struct pci_epc *epc,struct pci_epf *epf); +void pci_epc_remove_epf (struct pci_epc *epc,struct pci_epf *epf, +enum pci_epc_interface_type type);int pci_epc_write_header (struct pci_epc *epc,u8 func_no,struct pci_epf_header *hdr); int pci_epc_set_bar(struct pci_epc *epc, u8 func_no,
include/linux/pci-epf.h +15 -2
@@ -14,6 +14,7 @@#include <linux/pci.h> struct pci_epf ; +enum pci_epc_interface_type ;enum pci_notify_event {CORE_INIT ,
@@ -119,6 +120,11 @@ struct pci_epf_bar {* @list: to add pci_epf as a list of PCI endpoint functions to pci_epc * @nb: notifier block to notify EPF of any EPC events (like linkup) * @lock: mutex to protect pci_epf_ops +* @sec_epc: the secondary EPC device to which this EPF device is bound +* @sec_epc_list: to add pci_epf as list of PCI endpoint functions to secondary +* EPC device +* @sec_epc_bar: represents the BAR of EPF device associated with secondary EPC +* @sec_epc_func_no: unique (physical) function number within the secondary EPC */ struct pci_epf {struct device dev;
@@ -135,6 +141,12 @@ struct pci_epf {struct notifier_block nb;/* mutex to protect against concurrent access of pci_epf_ops */ struct mutex lock; + +/* Below members are to attach secondary EPC to an endpoint function */ +struct pci_epc *sec_epc; +struct list_head sec_epc_list; +struct pci_epf_bar sec_epc_bar[6 ]; +u8 sec_epc_func_no; };/**
@@ -171,8 +183,9 @@ int __pci_epf_register_driver(struct pci_epf_driver *driver, struct module *owner);void pci_epf_unregister_driver (struct pci_epf_driver *driver);void *pci_epf_alloc_space (struct pci_epf *epf,size_t size,enum pci_barno bar, -size_t align); -void pci_epf_free_space (struct pci_epf *epf,void *addr,enum pci_barno bar); +size_t align,enum pci_epc_interface_type type); +void pci_epf_free_space (struct pci_epf *epf,void *addr,enum pci_barno bar, +enum pci_epc_interface_type type);int pci_epf_bind (struct pci_epf *epf);void pci_epf_unbind (struct pci_epf *epf);#endif /* __LINUX_PCI_EPF_H */
Signature · 1 line
2.17.1
PATCH v2 7/14 PCI: endpoint: Add support in configfs to associate two EPCs with EPF
In reply to Kishon Vijay Abraham INow that PCI endpoint core supports to add secondary endpoint controller (EPC) with endpoint function (EPF), Add support in configfs to associate two EPCs with EPF. This creates "primary" and "secondary" directory inside the directory created by users for EPF device. Users have to add a symlink of endpoint controller (pci_ep/controllers/) to "primary" or "secondary" directory to bind EPF to primary and secondary EPF interfaces respectively. Existing method of linking directory representing EPF device to directory representing EPC device to associate a single EPC device with a EPF device will continue to work.
--- .../PCI/endpoint/pci-endpoint-cfs.rst | 10 ++ drivers/pci/endpoint/pci-ep-cfs.c | 148 ++++++++++++++++++ 2 files changed, 158 insertions(+)
Documentation/PCI/endpoint/pci-endpoint-cfs.rst +10 -0
@@ -68,6 +68,16 @@ created) ... subsys_vendor_id ... subsys_id ... interrupt_pin + ... primary/ + ... <Symlink EPC Device1>/ + ... secondary/ + ... <Symlink EPC Device2>/ + +If an EPF device has to be associated with 2 EPCs (like in the case of +Non-transparent bridge), symlink of endpoint controller connected to primary +interface should be added in 'primary' directory and symlink of endpoint +controller connected to secondary interface should be added in 'secondary' +directory. EPC Device ==========
drivers/pci/endpoint/pci-ep-cfs.c +148 -0
176 lines not on this page · raw message
Signature · 1 line
2.17.1
PATCH v2 8/14 PCI: endpoint: Add pci_epc_ops to map MSI irq
In reply to Kishon Vijay Abraham IAdd pci_epc_ops to map physical address to MSI address and return MSI data. The physical address is an address in the outbound region. This is required to implement doorbell functionality of NTB (non transparent bridge) wherein EPC on either side of the interface (primary and secondary) can directly write to the physical address (in outbound region) of the other interface to ring doorbell using MSI.
--- drivers/pci/endpoint/pci-epc-core.c | 40 +++++++++++++++++++++++++++++ include/linux/pci-epc.h | 7 +++++ 2 files changed, 47 insertions(+)
drivers/pci/endpoint/pci-epc-core.c +40 -0
@@ -230,6 +230,46 @@ int pci_epc_raise_irq(struct pci_epc *epc, u8 func_no, }EXPORT_SYMBOL_GPL (pci_epc_raise_irq); +/** +* pci_epc_map_msi_irq() - Map physical address to MSI address and return +* MSI data +* @epc: the EPC device which has the MSI capability +* @func_no: the physical endpoint function number in the EPC device +* @vfunc_no: the virtual endpoint function number in the physical function +* @phys_addr: the physical address of the outbound region +* @interrupt_num: the MSI interrupt number +* @entry_size: Size of Outbound address region for each interrupt +* @msi_data: the data that should be written in order to raise MSI interrupt +* with interrupt number as 'interrupt num' +* +* Invoke to map physical address to MSI address and return MSI data. The +* physical address should be an address in the outbound region. This is +* required to implement doorbell functionality of NTB wherein EPC on either +* side of the interface (primary and secondary) can directly write to the +* physical address (in outbound region) of the other interface to ring +* doorbell. +*/ +int pci_epc_map_msi_irq (struct pci_epc *epc,u8 func_no,u8 vfunc_no, +phys_addr_t phys_addr,u8 interrupt_num,u32 entry_size, +u32 *msi_data) +{ +int ret; + +if (IS_ERR_OR_NULL (epc)) +return -EINVAL ; + +if (!epc->ops->map_msi_irq) +return -EINVAL ; + +mutex_lock (&epc->lock); + ret = epc->ops->map_msi_irq (epc, func_no, vfunc_no, phys_addr, + interrupt_num, entry_size, msi_data); +mutex_unlock (&epc->lock); + +return ret; +} +EXPORT_SYMBOL_GPL (pci_epc_map_msi_irq); +/** * pci_epc_get_msi() - get the number of MSI interrupt numbers allocated * @epc: the EPC device to which MSI interrupts was requested
include/linux/pci-epc.h +7 -0
@@ -55,6 +55,7 @@ pci_epc_interface_string(enum pci_epc_interface_type type)* @get_msix: ops to get the number of MSI-X interrupts allocated by the RC * from the MSI-X capability register * @raise_irq: ops to raise a legacy, MSI or MSI-X interrupt +* @map_msi_irq: ops to map physical address to MSI address and return MSI data * @start: ops to start the PCI link * @stop: ops to stop the PCI link * @owner: the module owner containing the ops
@@ -77,6 +78,9 @@ struct pci_epc_ops {int (*get_msix)(struct pci_epc *epc,u8 func_no);int (*raise_irq)(struct pci_epc *epc,u8 func_no,enum pci_epc_irq_type type,u16 interrupt_num); +int (*map_msi_irq)(struct pci_epc *epc,u8 func_no,u8 vfunc_no, +phys_addr_t phys_addr,u8 interrupt_num, +u32 entry_size,u32 *msi_data);int (*start)(struct pci_epc *epc);void (*stop)(struct pci_epc *epc); const struct pci_epc_features* (*get_features)(struct pci_epc *epc,
@@ -216,6 +220,9 @@ int pci_epc_get_msi(struct pci_epc *epc, u8 func_no);int pci_epc_set_msix (struct pci_epc *epc,u8 func_no,u16 interrupts,enum pci_barno ,u32 offset);int pci_epc_get_msix (struct pci_epc *epc,u8 func_no); +int pci_epc_map_msi_irq (struct pci_epc *epc,u8 func_no, +phys_addr_t phys_addr,u8 interrupt_num, +u32 entry_size,u32 *msi_data);int pci_epc_raise_irq (struct pci_epc *epc,u8 func_no,enum pci_epc_irq_type type,u16 interrupt_num);int pci_epc_start (struct pci_epc *epc);
Signature · 1 line
2.17.1
PATCH v2 9/14 PCI: cadence: Implement ->msi_map_irq() ops
In reply to Kishon Vijay Abraham IImplement ->msi_map_irq() ops in order to map physical address to MSI address and return MSI data.
--- .../pci/controller/cadence/pcie-cadence-ep.c | 50 +++++++++++++++++++ drivers/pci/endpoint/pci-epc-core.c | 7 ++- include/linux/pci-epc.h | 2 +- 3 files changed, 54 insertions(+), 5 deletions(-)
drivers/pci/controller/cadence/pcie-cadence-ep.c +50 -0
@@ -380,6 +380,54 @@ static int cdns_pcie_ep_send_msi_irq(struct cdns_pcie_ep *ep, u8 fn,return 0 ; } +static int cdns_pcie_ep_map_msi_irq (struct pci_epc *epc,u8 fn, +phys_addr_t addr,u8 interrupt_num, +u32 entry_size,u32 *msi_data) +{ +struct cdns_pcie_ep *ep =epc_get_drvdata (epc); +u32 cap =CDNS_PCIE_EP_FUNC_MSI_CAP_OFFSET ; +struct cdns_pcie *pcie = &ep->pcie; +u16 flags, mme, data, data_mask; +u8 msi_count; +u64 pci_addr; +int ret; +int i; + +/* Check whether the MSI feature has been enabled by the PCI host. */ + flags =cdns_pcie_ep_fn_readw (pcie, fn, cap +PCI_MSI_FLAGS ); +if (!(flags &PCI_MSI_FLAGS_ENABLE )) +return -EINVAL ; + +/* Get the number of enabled MSIs */ + mme = (flags &PCI_MSI_FLAGS_QSIZE ) >>4 ; + msi_count =1 << mme; +if (!interrupt_num || interrupt_num > msi_count) +return -EINVAL ; + +/* Compute the data value to be written. */ + data_mask = msi_count -1 ; + data =cdns_pcie_ep_fn_readw (pcie, fn, cap +PCI_MSI_DATA_64 ); + data = data & ~data_mask; + +/* Get the PCI address where to write the data into. */ + pci_addr =cdns_pcie_ep_fn_readl (pcie, fn, cap +PCI_MSI_ADDRESS_HI ); + pci_addr <<=32 ; + pci_addr |=cdns_pcie_ep_fn_readl (pcie, fn, cap +PCI_MSI_ADDRESS_LO ); + pci_addr &=GENMASK_ULL (63 ,2 ); + +for (i =0 ; i < interrupt_num; i++) { + ret =cdns_pcie_ep_map_addr (epc, fn, addr, pci_addr, + entry_size); +if (ret) +return ret; + addr = addr + entry_size; + } + + *msi_data = data; + +return 0 ; +} +static int cdns_pcie_ep_send_msix_irq (struct cdns_pcie_ep *ep,u8 fn,u16 interrupt_num) {
@@ -480,6 +528,7 @@ static const struct pci_epc_features cdns_pcie_epc_features = { .linkup_notifier = false, .msi_capable = true, .msix_capable = true, + .align = 256, }; static const struct pci_epc_features*
@@ -499,6 +548,7 @@ static const struct pci_epc_ops cdns_pcie_epc_ops = { .set_msix = cdns_pcie_ep_set_msix, .get_msix = cdns_pcie_ep_get_msix, .raise_irq = cdns_pcie_ep_raise_irq, + .map_msi_irq = cdns_pcie_ep_map_msi_irq, .start = cdns_pcie_ep_start, .get_features = cdns_pcie_ep_get_features, };
drivers/pci/endpoint/pci-epc-core.c +3 -4
@@ -235,7 +235,6 @@ EXPORT_SYMBOL_GPL(pci_epc_raise_irq);* MSI data * @epc: the EPC device which has the MSI capability * @func_no: the physical endpoint function number in the EPC device -* @vfunc_no: the virtual endpoint function number in the physical function * @phys_addr: the physical address of the outbound region * @interrupt_num: the MSI interrupt number * @entry_size: Size of Outbound address region for each interrupt
@@ -249,7 +248,7 @@ EXPORT_SYMBOL_GPL(pci_epc_raise_irq);* physical address (in outbound region) of the other interface to ring * doorbell. */ -int pci_epc_map_msi_irq (struct pci_epc *epc,u8 func_no,u8 vfunc_no, +int pci_epc_map_msi_irq (struct pci_epc *epc,u8 func_no,phys_addr_t phys_addr,u8 interrupt_num,u32 entry_size,u32 *msi_data) {
@@ -262,8 +261,8 @@ int pci_epc_map_msi_irq(struct pci_epc *epc, u8 func_no, u8 vfunc_no,return -EINVAL ;mutex_lock (&epc->lock); - ret = epc->ops->map_msi_irq (epc, func_no, vfunc_no, phys_addr, - interrupt_num, entry_size, msi_data); + ret = epc->ops->map_msi_irq (epc, func_no, phys_addr, interrupt_num, + entry_size, msi_data);mutex_unlock (&epc->lock);return ret;
include/linux/pci-epc.h +1 -1
@@ -78,7 +78,7 @@ struct pci_epc_ops {int (*get_msix)(struct pci_epc *epc,u8 func_no);int (*raise_irq)(struct pci_epc *epc,u8 func_no,enum pci_epc_irq_type type,u16 interrupt_num); -int (*map_msi_irq)(struct pci_epc *epc,u8 func_no,u8 vfunc_no, +int (*map_msi_irq)(struct pci_epc *epc,u8 func_no,phys_addr_t phys_addr,u8 interrupt_num,u32 entry_size,u32 *msi_data);int (*start)(struct pci_epc *epc);
Signature · 1 line
2.17.1
PATCH v2 10/14 PCI: endpoint: Add EP function driver to provide NTB functionality
In reply to Kishon Vijay Abraham IAdd a new endpoint function driver to provide NTB functionality using multiple PCIe endpoint instances.
--- drivers/pci/endpoint/functions/Kconfig | 12 + drivers/pci/endpoint/functions/Makefile | 1 + drivers/pci/endpoint/functions/pci-epf-ntb.c | 2151 ++++++++++++++++++ include/linux/pci-epf.h | 3 + 4 files changed, 2167 insertions(+) create mode 100644 drivers/pci/endpoint/functions/pci-epf-ntb.c
drivers/pci/endpoint/functions/Kconfig +12 -0
@@ -12,3 +12,15 @@ config PCI_EPF_TEST for PCI Endpoint. If in doubt, say "N" to disable Endpoint test driver. + +config PCI_EPF_NTB + tristate "PCI Endpoint NTB driver" + depends on PCI_ENDPOINT + help + Select this configuration option to enable the NTB driver + for PCI Endpoint. NTB driver implements NTB controller + functionality using multiple PCIe endpoint instances. It + can support NTB endpoint function devices created using + device tree. + + If in doubt, say "N" to disable Endpoint NTB driver.
drivers/pci/endpoint/functions/Makefile +1 -0
@@ -4,3 +4,4 @@ # obj-$(CONFIG_PCI_EPF_TEST) += pci-epf-test.o +obj-$(CONFIG_PCI_EPF_NTB) += pci-epf-ntb.o
drivers/pci/endpoint/functions/pci-epf-ntb.c new file +2151 -0
2,151 lines not on this page · raw message
include/linux/pci-epf.h +3 -0
@@ -9,6 +9,7 @@#ifndef __LINUX_PCI_EPF_H#define __LINUX_PCI_EPF_H +#include <linux/configfs.h> #include <linux/device.h> #include <linux/mod_devicetable.h> #include <linux/pci.h>
@@ -125,6 +126,7 @@ struct pci_epf_bar {* EPC device * @sec_epc_bar: represents the BAR of EPF device associated with secondary EPC * @sec_epc_func_no: unique (physical) function number within the secondary EPC +* @group: configfs group associated with the EPF device */ struct pci_epf {struct device dev;
@@ -147,6 +149,7 @@ struct pci_epf {struct list_head sec_epc_list;struct pci_epf_bar sec_epc_bar[6 ];u8 sec_epc_func_no; +struct config_group *group; };/**
Signature · 1 line
2.17.1
PATCH v2 11/14 PCI: Add TI J721E device to pci ids
In reply to Kishon Vijay Abraham IAdd TI J721E device to the pci id database. Since this device has a configurable PCIe endpoint, it could be used with different drivers.
--- drivers/misc/pci_endpoint_test.c | 1 - include/linux/pci_ids.h | 1 + 2 files changed, 1 insertion(+), 1 deletion(-)
drivers/misc/pci_endpoint_test.c +0 -1
@@ -68,7 +68,6 @@#define PCI_ENDPOINT_TEST_FLAGS 0x2c#define FLAG_USE_DMA BIT(0) -#define PCI_DEVICE_ID_TI_J721E 0xb00d#define PCI_DEVICE_ID_TI_AM654 0xb00c#define is_am654_pci_dev (pdev) \
include/linux/pci_ids.h +1 -0
@@ -879,6 +879,7 @@#define PCI_DEVICE_ID_TI_X620 0xac8d#define PCI_DEVICE_ID_TI_X420 0xac8e#define PCI_DEVICE_ID_TI_XX20_FM 0xac8f +#define PCI_DEVICE_ID_TI_J721E 0xb00d#define PCI_DEVICE_ID_TI_DRA74x 0xb500#define PCI_DEVICE_ID_TI_DRA72x 0xb501
Signature · 1 line
2.17.1
PATCH v2 12/14 NTB: Add support for EPF PCI-Express Non-Transparent Bridge
In reply to Kishon Vijay Abraham IAdd support for EPF PCI-Express Non-Transparent Bridge (NTB) device. This driver is platform independent and could be used by any platform which have multiple PCIe endpoint instances configured using the pci-epf-ntb driver. The driver connnects to the standard NTB sub-system interface. The EPF NTB device has configurable number of memory windows (Max 4), configurable number of doorbell (Max 32), and configurable number of scratch-pad registers.
--- drivers/ntb/hw/Kconfig | 1 + drivers/ntb/hw/Makefile | 1 + drivers/ntb/hw/epf/Kconfig | 6 + drivers/ntb/hw/epf/Makefile | 1 + drivers/ntb/hw/epf/ntb_hw_epf.c | 750 ++++++++++++++++++++++++++++++++ 5 files changed, 759 insertions(+) create mode 100644 drivers/ntb/hw/epf/Kconfig create mode 100644 drivers/ntb/hw/epf/Makefile create mode 100644 drivers/ntb/hw/epf/ntb_hw_epf.c
drivers/ntb/hw/Kconfig +1 -0
@@ -2,4 +2,5 @@ source "drivers/ntb/hw/amd/Kconfig" source "drivers/ntb/hw/idt/Kconfig" source "drivers/ntb/hw/intel/Kconfig" +source "drivers/ntb/hw/epf/Kconfig" source "drivers/ntb/hw/mscc/Kconfig"
drivers/ntb/hw/Makefile +1 -0
@@ -2,4 +2,5 @@ obj-$(CONFIG_NTB_AMD) += amd/ obj-$(CONFIG_NTB_IDT) += idt/ obj-$(CONFIG_NTB_INTEL) += intel/ +obj-$(CONFIG_NTB_EPF) += epf/ obj-$(CONFIG_NTB_SWITCHTEC) += mscc/
drivers/ntb/hw/epf/Kconfig new file +6 -0
@@ -0,0 +1,6 @@ +config NTB_EPF + tristate "Generic EPF Non-Transparent Bridge support" + depends on m + help + This driver supports EPF NTB on configurable endpoint. + If unsure, say N.
drivers/ntb/hw/epf/Makefile new file +1 -0
@@ -0,0 +1 @@ +obj-$(CONFIG_NTB_EPF) += ntb_hw_epf.o
drivers/ntb/hw/epf/ntb_hw_epf.c new file +750 -0
750 lines not on this page · raw message
Signature · 1 line
2.17.1
PATCH v2 13/14 NTB: tool: Enable the NTB/PCIe link on the local or remote side of bridge
In reply to Kishon Vijay Abraham IInvoke ntb_link_enable() to enable the NTB/PCIe link on the local or remote side of the bridge.
--- drivers/ntb/test/ntb_tool.c | 1 + 1 file changed, 1 insertion(+)
drivers/ntb/test/ntb_tool.c +1 -0
@@ -1638,6 +1638,7 @@ static int tool_probe(struct ntb_client *self, struct ntb_dev *ntb)tool_setup_dbgfs (tc); +ntb_link_enable (ntb,NTB_SPEED_AUTO ,NTB_WIDTH_AUTO );return 0 ; err_clear_mws:
Signature · 1 line
2.17.1
PATCH v2 14/14 Documentation: PCI: Add userguide for PCI endpoint NTB function
In reply to Kishon Vijay Abraham IAdd documentation to help users use pci-epf-ntb function driver and existing host side NTB infrastructure for NTB functionality.
--- Documentation/PCI/endpoint/index.rst | 1 + Documentation/PCI/endpoint/pci-ntb-howto.rst | 141 +++++++++++++++++++ 2 files changed, 142 insertions(+) create mode 100644 Documentation/PCI/endpoint/pci-ntb-howto.rst
Documentation/PCI/endpoint/index.rst +1 -0
@@ -12,3 +12,4 @@ PCI Endpoint Framework pci-test-function pci-test-howto pci-ntb-function + pci-ntb-howto
Documentation/PCI/endpoint/pci-ntb-howto.rst new file +141 -0
141 lines not on this page · raw message
Signature · 1 line
2.17.1