PCI: Linux kernel SR-IOV support
Subsystems (7)
v7 cover PCI: Linux kernel SR-IOV support
Greetings,
Following patches are intended to support SR-IOV capability in the Linux kernel. With these patches, people can turn a PCI device with the capability into multiple ones from software perspective, which will benefit KVM and achieve other purposes such as QoS, security, and etc.
The Physical Function and Virtual Function drivers using the SR-IOV APIs will come soon!
Major changes from v6 to v7:
1, remove boot-time resource rebalancing support. (Greg KH) 2, emit uevent upon the PF driver is loaded. (Greg KH)
3, put SR-IOV callback function into the 'pci_driver'. (Matthew Wilcox) 4, register SR-IOV service at the PF loading stage.
5, remove unnecessary APIs (pci_iov_enable/disable).
---
[PATCH 1/13 v7] PCI: enhance pci_ari_enabled()
[PATCH 2/13 v7] PCI: remove unnecessary arg of pci_update_resource() [PATCH 3/13 v7] PCI: define PCI resource names in an 'enum' [PATCH 4/13 v7] PCI: remove unnecessary condition check in pci_restore_bars() [PATCH 5/13 v7] PCI: export __pci_read_base()
[PATCH 6/13 v7] PCI: make pci_alloc_child_bus() be able to handle NULL bridge [PATCH 7/13 v7] PCI: add a new function to map BAR offset
[PATCH 8/13 v7] PCI: cleanup pci_bus_add_devices()
[PATCH 9/13 v7] PCI: split a new function from pci_bus_add_devices() [PATCH 10/13 v7] PCI: support the SR-IOV capability
[PATCH 11/13 v7] PCI: reserve bus range for SR-IOV device
[PATCH 12/13 v7] PCI: document the SR-IOV sysfs entries
[PATCH 13/13 v7] PCI: document for SR-IOV user and developer
---
Single Root I/O Virtualization (SR-IOV) capability defined by PCI-SIG is intended to enable multiple system software to share PCI hardware resources. PCI device that supports this capability can be extended to one Physical Functions plus multiple Virtual Functions. Physical Function, which could be considered as the "real" PCI device, reflects the hardware instance and manages all physical resources. Virtual Functions are associated with a Physical Function and shares physical resources with the Physical Function.Software can control allocation of Virtual Functions via registers encapsulated in the capability structure.
SR-IOV specification can be found at
http://www.pcisig.com/members/downloads/specifications/iov/sr-iov1.0_11Sep07.pdf
Devices that support SR-IOV are available from following vendors:
http://download.intel.com/design/network/ProdBrf/320025.pdf
http://www.netxen.com/products/chipsolutions/NX3031.html
http://www.neterion.com/products/x3100.html
PATCH v7 1/13 PCI: enhance pci_ari_enabled()
Change parameter of pci_ari_enabled() from 'pci_dev' to 'pci_bus'.
ARI forwarding on the bridge mostly concerns the subordinate devices rather than the bridge itself. So this change will make the function easier to use.
--- drivers/pci/pci.h | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-)
drivers/pci/pci.h +3 -3
@@ -162,13 +162,13 @@ struct pci_slot_attribute {extern void pci_enable_ari (struct pci_dev *dev);/** * pci_ari_enabled - query ARI forwarding status -* @dev: the PCI device +* @bus: the PCI bus * * Returns 1 if ARI forwarding is enabled, or 0 if not enabled; */ -static inline int pci_ari_enabled (struct pci_dev *dev) +static inline int pci_ari_enabled (struct pci_bus *bus) { -return dev->ari_enabled; +return bus->self && bus->self->ari_enabled; }#endif /* DRIVERS_PCI_H */
Signature · 1 line
1.5.6.4
PATCH v7 1/13 PCI: enhance pci_ari_enabled()
In reply to Yu ZhaoChange parameter of pci_ari_enabled() from 'pci_dev' to 'pci_bus'.
ARI forwarding on the bridge mostly concerns the subordinate devices rather than the bridge itself. So this change will make the function easier to use.
--- drivers/pci/pci.h | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-)
drivers/pci/pci.h +3 -3
@@ -162,13 +162,13 @@ struct pci_slot_attribute {extern void pci_enable_ari (struct pci_dev *dev);/** * pci_ari_enabled - query ARI forwarding status -* @dev: the PCI device +* @bus: the PCI bus * * Returns 1 if ARI forwarding is enabled, or 0 if not enabled; */ -static inline int pci_ari_enabled (struct pci_dev *dev) +static inline int pci_ari_enabled (struct pci_bus *bus) { -return dev->ari_enabled; +return bus->self && bus->self->ari_enabled; }#endif /* DRIVERS_PCI_H */
Signature · 1 line
1.5.6.4
PATCH v7 2/13 PCI: remove unnecessary arg of pci_update_resource()
In reply to Yu ZhaoThis cleanup removes unnecessary argument 'struct resource *res' in pci_update_resource(), so it takes same arguments as other companion functions (pci_assign_resource(), etc.).
--- drivers/pci/pci.c | 4 ++-- drivers/pci/setup-res.c | 7 ++++--- include/linux/pci.h | 2 +- 3 files changed, 7 insertions(+), 6 deletions(-)
drivers/pci/pci.c +2 -2
@@ -377,8 +377,8 @@ pci_restore_bars(struct pci_dev *dev)return ; } -for (i =0 ; i < numres; i ++) -pci_update_resource (dev, &dev->resource[i], i); +for (i =0 ; i < numres; i++) +pci_update_resource (dev, i); }static struct pci_platform_pm_ops *pci_platform_pm;
drivers/pci/setup-res.c +4 -3
@@ -26,11 +26,12 @@#include "pci.h" -void pci_update_resource (struct pci_dev *dev,struct resource *res,int resno) +void pci_update_resource (struct pci_dev *dev,int resno) {struct pci_bus_region region;u32 new, check, mask;int reg; +struct resource *res = dev->resource + resno;/* * Ignore resources for unimplemented BARs and unused resource slots
@@ -162,7 +163,7 @@ int pci_assign_resource(struct pci_dev *dev, int resno) }else { res->flags &= ~IORESOURCE_STARTALIGN ;if (resno <PCI_BRIDGE_RESOURCES ) -pci_update_resource (dev, res, resno); +pci_update_resource (dev, resno); }return ret;
@@ -197,7 +198,7 @@ int pci_assign_resource_fixed(struct pci_dev *dev, int resno)dev_err (&dev->dev,"BAR %d: can't allocate %s resource %pR\n" , resno, res->flags &IORESOURCE_IO ?"I/O" :"mem" , res); }else if (resno <PCI_BRIDGE_RESOURCES ) { -pci_update_resource (dev, res, resno); +pci_update_resource (dev, resno); }return ret;
include/linux/pci.h +1 -1
@@ -642,7 +642,7 @@ int pcie_get_readrq(struct pci_dev *dev);int pcie_set_readrq (struct pci_dev *dev,int rq);int pci_reset_function (struct pci_dev *dev);int pci_execute_reset_function (struct pci_dev *dev); -void pci_update_resource (struct pci_dev *dev,struct resource *res,int resno); +void pci_update_resource (struct pci_dev *dev,int resno);int __must_check pci_assign_resource (struct pci_dev *dev,int i);int pci_select_bars (struct pci_dev *dev,unsigned long flags);
Signature · 1 line
1.5.6.4
PATCH v7 2/13 PCI: remove unnecessary arg of pci_update_resource()
In reply to Yu ZhaoThis cleanup removes unnecessary argument 'struct resource *res' in pci_update_resource(), so it takes same arguments as other companion functions (pci_assign_resource(), etc.).
--- drivers/pci/pci.c | 4 ++-- drivers/pci/setup-res.c | 7 ++++--- include/linux/pci.h | 2 +- 3 files changed, 7 insertions(+), 6 deletions(-)
drivers/pci/pci.c +2 -2
@@ -377,8 +377,8 @@ pci_restore_bars(struct pci_dev *dev)return ; } -for (i =0 ; i < numres; i ++) -pci_update_resource (dev, &dev->resource[i], i); +for (i =0 ; i < numres; i++) +pci_update_resource (dev, i); }static struct pci_platform_pm_ops *pci_platform_pm;
drivers/pci/setup-res.c +4 -3
@@ -26,11 +26,12 @@#include "pci.h" -void pci_update_resource (struct pci_dev *dev,struct resource *res,int resno) +void pci_update_resource (struct pci_dev *dev,int resno) {struct pci_bus_region region;u32 new, check, mask;int reg; +struct resource *res = dev->resource + resno;/* * Ignore resources for unimplemented BARs and unused resource slots
@@ -162,7 +163,7 @@ int pci_assign_resource(struct pci_dev *dev, int resno) }else { res->flags &= ~IORESOURCE_STARTALIGN ;if (resno <PCI_BRIDGE_RESOURCES ) -pci_update_resource (dev, res, resno); +pci_update_resource (dev, resno); }return ret;
@@ -197,7 +198,7 @@ int pci_assign_resource_fixed(struct pci_dev *dev, int resno)dev_err (&dev->dev,"BAR %d: can't allocate %s resource %pR\n" , resno, res->flags &IORESOURCE_IO ?"I/O" :"mem" , res); }else if (resno <PCI_BRIDGE_RESOURCES ) { -pci_update_resource (dev, res, resno); +pci_update_resource (dev, resno); }return ret;
include/linux/pci.h +1 -1
@@ -642,7 +642,7 @@ int pcie_get_readrq(struct pci_dev *dev);int pcie_set_readrq (struct pci_dev *dev,int rq);int pci_reset_function (struct pci_dev *dev);int pci_execute_reset_function (struct pci_dev *dev); -void pci_update_resource (struct pci_dev *dev,struct resource *res,int resno); +void pci_update_resource (struct pci_dev *dev,int resno);int __must_check pci_assign_resource (struct pci_dev *dev,int i);int pci_select_bars (struct pci_dev *dev,unsigned long flags);
Signature · 1 line
1.5.6.4
PATCH v7 3/13 PCI: define PCI resource names in an 'enum'
In reply to Yu ZhaoThis patch moves all definitions of the PCI resource names to an 'enum', and also replaces some hard-coded resource variables with symbol names. This change eases introduction of device specific resources.
--- drivers/pci/pci-sysfs.c | 4 +++- drivers/pci/probe.c | 2 +- drivers/pci/proc.c | 7 ++++--- include/linux/pci.h | 37 ++++++++++++++++++++++++------------- 4 files changed, 32 insertions(+), 18 deletions(-)
drivers/pci/pci-sysfs.c +3 -1
@@ -101,11 +101,13 @@ resource_show(struct device * dev, struct device_attribute *attr, char * buf)struct pci_dev * pci_dev =to_pci_dev (dev);char * str = buf;int i; -int max =7 ; +int max;resource_size_t start, end;if (pci_dev->subordinate) max =DEVICE_COUNT_RESOURCE ; +else + max =PCI_BRIDGE_RESOURCES ;for (i =0 ; i < max; i++) {struct resource *res = &pci_dev->resource[i];
drivers/pci/probe.c +1 -1
@@ -423,7 +423,7 @@ static struct pci_bus *pci_alloc_child_bus(struct pci_bus *parent, child->subordinate =0xff ;/* Set up default resource pointers and names.. */ -for (i =0 ; i <4 ; i++) { +for (i =0 ; i <PCI_BRIDGE_RESOURCE_NUM ; i++) { child->resource[i] = &bridge->resource[PCI_BRIDGE_RESOURCES +i]; child->resource[i]->name = child->name; }
drivers/pci/proc.c +4 -3
@@ -352,15 +352,16 @@ static int show_device(struct seq_file *m, void *v) dev->vendor, dev->device, dev->irq); -/* Here should be 7 and not PCI_NUM_RESOURCES as we need to preserve compatibility */ -for (i=0 ; i<7 ; i++) { + +/* only print standard and ROM resources to preserve compatibility */ +for (i =0 ; i <=PCI_ROM_RESOURCE ; i++) {resource_size_t start, end;pci_resource_to_user (dev, i, &dev->resource[i], &start, &end);seq_printf (m,"\t%16llx" , (unsigned long long )(start | (dev->resource[i].flags &PCI_REGION_FLAG_MASK ))); } -for (i=0 ; i<7 ; i++) { +for (i =0 ; i <=PCI_ROM_RESOURCE ; i++) {resource_size_t start, end;pci_resource_to_user (dev, i, &dev->resource[i], &start, &end); seq_printf(m, "\t%16llx",
include/linux/pci.h +24 -13
@@ -82,7 +82,30 @@ enum pci_mmap_state {#define PCI_DMA_FROMDEVICE 2#define PCI_DMA_NONE 3 -#define DEVICE_COUNT_RESOURCE 12 +/* +* For PCI devices, the region numbers are assigned this way: +*/ +enum { +/* #0-5: standard PCI resources */ +PCI_STD_RESOURCES , +PCI_STD_RESOURCE_END =5 , + +/* #6: expansion ROM resource */ +PCI_ROM_RESOURCE , + +/* resources assigned to buses behind the bridge */ +#define PCI_BRIDGE_RESOURCE_NUM 4 + +PCI_BRIDGE_RESOURCES , +PCI_BRIDGE_RESOURCE_END =PCI_BRIDGE_RESOURCES + +PCI_BRIDGE_RESOURCE_NUM -1 , + +/* total resources associated with a PCI device */ +PCI_NUM_RESOURCES , + +/* preserve this for compatibility */ +DEVICE_COUNT_RESOURCE +};typedef int __bitwise pci_power_t ;
@@ -268,18 +291,6 @@ static inline void pci_add_saved_cap(struct pci_dev *pci_dev,hlist_add_head (&new_cap->next, &pci_dev->saved_cap_space); } -/* -* For PCI devices, the region numbers are assigned this way: -* -* 0-5 standard PCI regions -* 6 expansion ROM -* 7-10 bridges: address space assigned to buses behind the bridge -*/ - -#define PCI_ROM_RESOURCE 6 -#define PCI_BRIDGE_RESOURCES 7 -#define PCI_NUM_RESOURCES 11 -#ifndef PCI_BUS_NUM_RESOURCES #define PCI_BUS_NUM_RESOURCES 16#endif
Signature · 1 line
1.5.6.4
PATCH v7 3/13 PCI: define PCI resource names in an 'enum'
In reply to Yu ZhaoThis patch moves all definitions of the PCI resource names to an 'enum', and also replaces some hard-coded resource variables with symbol names. This change eases introduction of device specific resources.
--- drivers/pci/pci-sysfs.c | 4 +++- drivers/pci/probe.c | 2 +- drivers/pci/proc.c | 7 ++++--- include/linux/pci.h | 37 ++++++++++++++++++++++++------------- 4 files changed, 32 insertions(+), 18 deletions(-)
drivers/pci/pci-sysfs.c +3 -1
@@ -101,11 +101,13 @@ resource_show(struct device * dev, struct device_attribute *attr, char * buf)struct pci_dev * pci_dev =to_pci_dev (dev);char * str = buf;int i; -int max =7 ; +int max;resource_size_t start, end;if (pci_dev->subordinate) max =DEVICE_COUNT_RESOURCE ; +else + max =PCI_BRIDGE_RESOURCES ;for (i =0 ; i < max; i++) {struct resource *res = &pci_dev->resource[i];
drivers/pci/probe.c +1 -1
@@ -423,7 +423,7 @@ static struct pci_bus *pci_alloc_child_bus(struct pci_bus *parent, child->subordinate =0xff ;/* Set up default resource pointers and names.. */ -for (i =0 ; i <4 ; i++) { +for (i =0 ; i <PCI_BRIDGE_RESOURCE_NUM ; i++) { child->resource[i] = &bridge->resource[PCI_BRIDGE_RESOURCES +i]; child->resource[i]->name = child->name; }
drivers/pci/proc.c +4 -3
@@ -352,15 +352,16 @@ static int show_device(struct seq_file *m, void *v) dev->vendor, dev->device, dev->irq); -/* Here should be 7 and not PCI_NUM_RESOURCES as we need to preserve compatibility */ -for (i=0 ; i<7 ; i++) { + +/* only print standard and ROM resources to preserve compatibility */ +for (i =0 ; i <=PCI_ROM_RESOURCE ; i++) {resource_size_t start, end;pci_resource_to_user (dev, i, &dev->resource[i], &start, &end);seq_printf (m,"\t%16llx" , (unsigned long long )(start | (dev->resource[i].flags &PCI_REGION_FLAG_MASK ))); } -for (i=0 ; i<7 ; i++) { +for (i =0 ; i <=PCI_ROM_RESOURCE ; i++) {resource_size_t start, end;pci_resource_to_user (dev, i, &dev->resource[i], &start, &end); seq_printf(m, "\t%16llx",
include/linux/pci.h +24 -13
@@ -82,7 +82,30 @@ enum pci_mmap_state {#define PCI_DMA_FROMDEVICE 2#define PCI_DMA_NONE 3 -#define DEVICE_COUNT_RESOURCE 12 +/* +* For PCI devices, the region numbers are assigned this way: +*/ +enum { +/* #0-5: standard PCI resources */ +PCI_STD_RESOURCES , +PCI_STD_RESOURCE_END =5 , + +/* #6: expansion ROM resource */ +PCI_ROM_RESOURCE , + +/* resources assigned to buses behind the bridge */ +#define PCI_BRIDGE_RESOURCE_NUM 4 + +PCI_BRIDGE_RESOURCES , +PCI_BRIDGE_RESOURCE_END =PCI_BRIDGE_RESOURCES + +PCI_BRIDGE_RESOURCE_NUM -1 , + +/* total resources associated with a PCI device */ +PCI_NUM_RESOURCES , + +/* preserve this for compatibility */ +DEVICE_COUNT_RESOURCE +};typedef int __bitwise pci_power_t ;
@@ -268,18 +291,6 @@ static inline void pci_add_saved_cap(struct pci_dev *pci_dev,hlist_add_head (&new_cap->next, &pci_dev->saved_cap_space); } -/* -* For PCI devices, the region numbers are assigned this way: -* -* 0-5 standard PCI regions -* 6 expansion ROM -* 7-10 bridges: address space assigned to buses behind the bridge -*/ - -#define PCI_ROM_RESOURCE 6 -#define PCI_BRIDGE_RESOURCES 7 -#define PCI_NUM_RESOURCES 11 -#ifndef PCI_BUS_NUM_RESOURCES #define PCI_BUS_NUM_RESOURCES 16#endif
Signature · 1 line
1.5.6.4
PATCH v7 4/13 PCI: remove unnecessary condition check in pci_restore_bars()
In reply to Yu ZhaoRemove the unnecessary number of resources condition checks because the pci_update_resource() will check availability of the resources.
--- drivers/pci/pci.c | 19 ++----------------- 1 files changed, 2 insertions(+), 17 deletions(-)
drivers/pci/pci.c +2 -17
@@ -360,24 +360,9 @@ pci_find_parent_resource(const struct pci_dev *dev, struct resource *res)static void pci_restore_bars (struct pci_dev *dev) { -int i, numres; - -switch (dev->hdr_type) { -case PCI_HEADER_TYPE_NORMAL : - numres =6 ; -break ; -case PCI_HEADER_TYPE_BRIDGE : - numres =2 ; -break ; -case PCI_HEADER_TYPE_CARDBUS : - numres =1 ; -break ; -default : -/* Should never get here, but just in case... */ -return ; - } +int i; -for (i =0 ; i < numres; i++) +for (i =0 ; i <PCI_BRIDGE_RESOURCES ; i++)pci_update_resource (dev, i); }
Signature · 1 line
1.5.6.4
PATCH v7 4/13 PCI: remove unnecessary condition check in pci_restore_bars()
In reply to Yu ZhaoRemove the unnecessary number of resources condition checks because the pci_update_resource() will check availability of the resources.
--- drivers/pci/pci.c | 19 ++----------------- 1 files changed, 2 insertions(+), 17 deletions(-)
drivers/pci/pci.c +2 -17
@@ -360,24 +360,9 @@ pci_find_parent_resource(const struct pci_dev *dev, struct resource *res)static void pci_restore_bars (struct pci_dev *dev) { -int i, numres; - -switch (dev->hdr_type) { -case PCI_HEADER_TYPE_NORMAL : - numres =6 ; -break ; -case PCI_HEADER_TYPE_BRIDGE : - numres =2 ; -break ; -case PCI_HEADER_TYPE_CARDBUS : - numres =1 ; -break ; -default : -/* Should never get here, but just in case... */ -return ; - } +int i; -for (i =0 ; i < numres; i++) +for (i =0 ; i <PCI_BRIDGE_RESOURCES ; i++)pci_update_resource (dev, i); }
Signature · 1 line
1.5.6.4
PATCH v7 5/13 PCI: export __pci_read_base()
In reply to Yu ZhaoExport __pci_read_base() so it can be used by whole PCI subsystem.
--- drivers/pci/pci.h | 9 +++++++++ drivers/pci/probe.c | 20 +++++++++----------- 2 files changed, 18 insertions(+), 11 deletions(-)
drivers/pci/pci.h +9 -0
@@ -159,6 +159,15 @@ struct pci_slot_attribute { };#define to_pci_slot_attr (s) container_of(s, struct pci_slot_attribute, attr) +enum pci_bar_type { + pci_bar_unknown,/* Standard PCI BAR probe */ + pci_bar_io,/* An io port BAR */ + pci_bar_mem32,/* A 32-bit memory BAR */ + pci_bar_mem64,/* A 64-bit memory BAR */ +}; + +extern int __pci_read_base (struct pci_dev *dev,enum pci_bar_type type, +struct resource *res,unsigned int reg);extern void pci_enable_ari (struct pci_dev *dev);/** * pci_ari_enabled - query ARI forwarding status
drivers/pci/probe.c +9 -11
@@ -135,13 +135,6 @@ static u64 pci_size(u64 base, u64 maxbase, u64 mask)return size; } -enum pci_bar_type { - pci_bar_unknown,/* Standard PCI BAR probe */ - pci_bar_io,/* An io port BAR */ - pci_bar_mem32,/* A 32-bit memory BAR */ - pci_bar_mem64,/* A 64-bit memory BAR */ -}; -static inline enum pci_bar_type decode_bar (struct resource *res,u32 bar) {if ((bar &PCI_BASE_ADDRESS_SPACE ) ==PCI_BASE_ADDRESS_SPACE_IO ) {
@@ -156,11 +149,16 @@ static inline enum pci_bar_type decode_bar(struct resource *res, u32 bar)return pci_bar_mem32; } -/* -* If the type is not unknown, we assume that the lowest bit is 'enable'. -* Returns 1 if the BAR was 64-bit and 0 if it was 32-bit. +/** +* pci_read_base - read a PCI BAR +* @dev: the PCI device +* @type: type of the BAR +* @res: resource buffer to be filled in +* @pos: BAR position in the config space +* +* Returns 1 if the BAR is 64-bit, or 0 if 32-bit. */ -static int __pci_read_base (struct pci_dev *dev,enum pci_bar_type type, +int __pci_read_base (struct pci_dev *dev,enum pci_bar_type type,struct resource *res,unsigned int pos) {u32 l, sz, mask;
Signature · 1 line
1.5.6.4
PATCH v7 5/13 PCI: export __pci_read_base()
In reply to Yu ZhaoExport __pci_read_base() so it can be used by whole PCI subsystem.
--- drivers/pci/pci.h | 9 +++++++++ drivers/pci/probe.c | 20 +++++++++----------- 2 files changed, 18 insertions(+), 11 deletions(-)
drivers/pci/pci.h +9 -0
@@ -159,6 +159,15 @@ struct pci_slot_attribute { };#define to_pci_slot_attr (s) container_of(s, struct pci_slot_attribute, attr) +enum pci_bar_type { + pci_bar_unknown,/* Standard PCI BAR probe */ + pci_bar_io,/* An io port BAR */ + pci_bar_mem32,/* A 32-bit memory BAR */ + pci_bar_mem64,/* A 64-bit memory BAR */ +}; + +extern int __pci_read_base (struct pci_dev *dev,enum pci_bar_type type, +struct resource *res,unsigned int reg);extern void pci_enable_ari (struct pci_dev *dev);/** * pci_ari_enabled - query ARI forwarding status
drivers/pci/probe.c +9 -11
@@ -135,13 +135,6 @@ static u64 pci_size(u64 base, u64 maxbase, u64 mask)return size; } -enum pci_bar_type { - pci_bar_unknown,/* Standard PCI BAR probe */ - pci_bar_io,/* An io port BAR */ - pci_bar_mem32,/* A 32-bit memory BAR */ - pci_bar_mem64,/* A 64-bit memory BAR */ -}; -static inline enum pci_bar_type decode_bar (struct resource *res,u32 bar) {if ((bar &PCI_BASE_ADDRESS_SPACE ) ==PCI_BASE_ADDRESS_SPACE_IO ) {
@@ -156,11 +149,16 @@ static inline enum pci_bar_type decode_bar(struct resource *res, u32 bar)return pci_bar_mem32; } -/* -* If the type is not unknown, we assume that the lowest bit is 'enable'. -* Returns 1 if the BAR was 64-bit and 0 if it was 32-bit. +/** +* pci_read_base - read a PCI BAR +* @dev: the PCI device +* @type: type of the BAR +* @res: resource buffer to be filled in +* @pos: BAR position in the config space +* +* Returns 1 if the BAR is 64-bit, or 0 if 32-bit. */ -static int __pci_read_base (struct pci_dev *dev,enum pci_bar_type type, +int __pci_read_base (struct pci_dev *dev,enum pci_bar_type type,struct resource *res,unsigned int pos) {u32 l, sz, mask;
Signature · 1 line
1.5.6.4
PATCH v7 6/13 PCI: make pci_alloc_child_bus() be able to handle NULL bridge
In reply to Yu ZhaoMake pci_alloc_child_bus() be able to allocate buses without bridge devices. Some SR-IOV devices can occupy more than one bus number, but there is no explicit bridges because that have internal routing mechanism.
--- drivers/pci/probe.c | 8 ++++++-- 1 files changed, 6 insertions(+), 2 deletions(-)
drivers/pci/probe.c +6 -2
@@ -398,12 +398,10 @@ static struct pci_bus *pci_alloc_child_bus(struct pci_bus *parent,if (!child)return NULL ; - child->self = bridge; child->parent = parent; child->ops = parent->ops; child->sysdata = parent->sysdata; child->bus_flags = parent->bus_flags; - child->bridge =get_device (&bridge->dev);/* initialize some portions of the bus device, but don't register it * now as the parent is not properly set up yet. This device will get
@@ -420,6 +418,12 @@ static struct pci_bus *pci_alloc_child_bus(struct pci_bus *parent, child->primary = parent->secondary; child->subordinate =0xff ; +if (!bridge) +return child; + + child->self = bridge; + child->bridge =get_device (&bridge->dev); +/* Set up default resource pointers and names.. */ for (i =0 ; i <PCI_BRIDGE_RESOURCE_NUM ; i++) { child->resource[i] = &bridge->resource[PCI_BRIDGE_RESOURCES +i];
Signature · 1 line
1.5.6.4
PATCH v7 6/13 PCI: make pci_alloc_child_bus() be able to handle NULL bridge
In reply to Yu ZhaoMake pci_alloc_child_bus() be able to allocate buses without bridge devices. Some SR-IOV devices can occupy more than one bus number, but there is no explicit bridges because that have internal routing mechanism.
--- drivers/pci/probe.c | 8 ++++++-- 1 files changed, 6 insertions(+), 2 deletions(-)
drivers/pci/probe.c +6 -2
@@ -398,12 +398,10 @@ static struct pci_bus *pci_alloc_child_bus(struct pci_bus *parent,if (!child)return NULL ; - child->self = bridge; child->parent = parent; child->ops = parent->ops; child->sysdata = parent->sysdata; child->bus_flags = parent->bus_flags; - child->bridge =get_device (&bridge->dev);/* initialize some portions of the bus device, but don't register it * now as the parent is not properly set up yet. This device will get
@@ -420,6 +418,12 @@ static struct pci_bus *pci_alloc_child_bus(struct pci_bus *parent, child->primary = parent->secondary; child->subordinate =0xff ; +if (!bridge) +return child; + + child->self = bridge; + child->bridge =get_device (&bridge->dev); +/* Set up default resource pointers and names.. */ for (i =0 ; i <PCI_BRIDGE_RESOURCE_NUM ; i++) { child->resource[i] = &bridge->resource[PCI_BRIDGE_RESOURCES +i];
Signature · 1 line
1.5.6.4
PATCH v7 7/13 PCI: add a new function to map BAR offset
In reply to Yu ZhaoAdd a function to map resource number to corresponding register so people can get the offset and type of device specific BARs.
--- drivers/pci/pci.c | 22 ++++++++++++++++++++++ drivers/pci/pci.h | 2 ++ drivers/pci/setup-res.c | 13 +++++-------- 3 files changed, 29 insertions(+), 8 deletions(-)
drivers/pci/pci.c +22 -0
@@ -2007,6 +2007,28 @@ int pci_select_bars(struct pci_dev *dev, unsigned long flags)return bars; } +/** +* pci_resource_bar - get position of the BAR associated with a resource +* @dev: the PCI device +* @resno: the resource number +* @type: the BAR type to be filled in +* +* Returns BAR position in config space, or 0 if the BAR is invalid. +*/ +int pci_resource_bar (struct pci_dev *dev,int resno,enum pci_bar_type *type) +{ +if (resno <PCI_ROM_RESOURCE ) { + *type = pci_bar_unknown; +return PCI_BASE_ADDRESS_0 +4 * resno; + }else if (resno ==PCI_ROM_RESOURCE ) { + *type = pci_bar_mem32; +return dev->rom_base_reg; + } + +dev_err (&dev->dev,"BAR: invalid resource #%d\n" , resno); +return 0 ; +} +static void __devinit pci_no_domains (void ) {#ifdef CONFIG_PCI_DOMAINS
drivers/pci/pci.h +2 -0
@@ -168,6 +168,8 @@ enum pci_bar_type {extern int __pci_read_base (struct pci_dev *dev,enum pci_bar_type type,struct resource *res,unsigned int reg); +extern int pci_resource_bar (struct pci_dev *dev,int resno, +enum pci_bar_type *type);extern void pci_enable_ari (struct pci_dev *dev);/** * pci_ari_enabled - query ARI forwarding status
drivers/pci/setup-res.c +5 -8
@@ -31,6 +31,7 @@ void pci_update_resource(struct pci_dev *dev, int resno)struct pci_bus_region region;u32 new, check, mask;int reg; +enum pci_bar_type type;struct resource *res = dev->resource + resno;/*
@@ -62,17 +63,13 @@ void pci_update_resource(struct pci_dev *dev, int resno)else mask = (u32 )PCI_BASE_ADDRESS_MEM_MASK ; -if (resno <6 ) { - reg =PCI_BASE_ADDRESS_0 +4 * resno; - }else if (resno ==PCI_ROM_RESOURCE ) { + reg =pci_resource_bar (dev, resno, &type); +if (!reg) +return ; +if (type != pci_bar_unknown) {if (!(res->flags &IORESOURCE_ROM_ENABLE ))return ; new |=PCI_ROM_ADDRESS_ENABLE ; - reg = dev->rom_base_reg; - }else { -/* Hmm, non-standard resource. */ - -return ;/* kill uninitialised var warning */ }pci_write_config_dword (dev, reg, new);
Signature · 1 line
1.5.6.4
PATCH v7 7/13 PCI: add a new function to map BAR offset
In reply to Yu ZhaoAdd a function to map resource number to corresponding register so people can get the offset and type of device specific BARs.
--- drivers/pci/pci.c | 22 ++++++++++++++++++++++ drivers/pci/pci.h | 2 ++ drivers/pci/setup-res.c | 13 +++++-------- 3 files changed, 29 insertions(+), 8 deletions(-)
drivers/pci/pci.c +22 -0
@@ -2007,6 +2007,28 @@ int pci_select_bars(struct pci_dev *dev, unsigned long flags)return bars; } +/** +* pci_resource_bar - get position of the BAR associated with a resource +* @dev: the PCI device +* @resno: the resource number +* @type: the BAR type to be filled in +* +* Returns BAR position in config space, or 0 if the BAR is invalid. +*/ +int pci_resource_bar (struct pci_dev *dev,int resno,enum pci_bar_type *type) +{ +if (resno <PCI_ROM_RESOURCE ) { + *type = pci_bar_unknown; +return PCI_BASE_ADDRESS_0 +4 * resno; + }else if (resno ==PCI_ROM_RESOURCE ) { + *type = pci_bar_mem32; +return dev->rom_base_reg; + } + +dev_err (&dev->dev,"BAR: invalid resource #%d\n" , resno); +return 0 ; +} +static void __devinit pci_no_domains (void ) {#ifdef CONFIG_PCI_DOMAINS
drivers/pci/pci.h +2 -0
@@ -168,6 +168,8 @@ enum pci_bar_type {extern int __pci_read_base (struct pci_dev *dev,enum pci_bar_type type,struct resource *res,unsigned int reg); +extern int pci_resource_bar (struct pci_dev *dev,int resno, +enum pci_bar_type *type);extern void pci_enable_ari (struct pci_dev *dev);/** * pci_ari_enabled - query ARI forwarding status
drivers/pci/setup-res.c +5 -8
@@ -31,6 +31,7 @@ void pci_update_resource(struct pci_dev *dev, int resno)struct pci_bus_region region;u32 new, check, mask;int reg; +enum pci_bar_type type;struct resource *res = dev->resource + resno;/*
@@ -62,17 +63,13 @@ void pci_update_resource(struct pci_dev *dev, int resno)else mask = (u32 )PCI_BASE_ADDRESS_MEM_MASK ; -if (resno <6 ) { - reg =PCI_BASE_ADDRESS_0 +4 * resno; - }else if (resno ==PCI_ROM_RESOURCE ) { + reg =pci_resource_bar (dev, resno, &type); +if (!reg) +return ; +if (type != pci_bar_unknown) {if (!(res->flags &IORESOURCE_ROM_ENABLE ))return ; new |=PCI_ROM_ADDRESS_ENABLE ; - reg = dev->rom_base_reg; - }else { -/* Hmm, non-standard resource. */ - -return ;/* kill uninitialised var warning */ }pci_write_config_dword (dev, reg, new);
Signature · 1 line
1.5.6.4
PATCH v7 8/13 PCI: cleanup pci_bus_add_devices()
In reply to Yu ZhaoThis cleanup makes pci_bus_add_devices() easier to read.
--- drivers/pci/bus.c | 55 +++++++++++++++++++++++++++-------------------------- 1 files changed, 28 insertions(+), 27 deletions(-)
drivers/pci/bus.c +28 -27
@@ -71,7 +71,7 @@ pci_bus_alloc_resource(struct pci_bus *bus, struct resource *res, }/** -* add a single device +* pci_bus_add_device - add a single device * @dev: device to add * * This adds a single pci device to the global
@@ -105,7 +105,7 @@ int pci_bus_add_device(struct pci_dev *dev)void pci_bus_add_devices (struct pci_bus *bus) {struct pci_dev *dev; -struct pci_bus *child_bus; +struct pci_bus *child;int retval;list_for_each_entry (dev, &bus->devices, bus_list) {
@@ -120,39 +120,40 @@ void pci_bus_add_devices(struct pci_bus *bus)list_for_each_entry (dev, &bus->devices, bus_list) {BUG_ON (!dev->is_added); + child = dev->subordinate;/* * If there is an unattached subordinate bus, attach * it and then scan for unattached PCI devices. */ -if (dev->subordinate) { -if (list_empty (&dev->subordinate->node)) { -down_write (&pci_bus_sem); -list_add_tail (&dev->subordinate->node, - &dev->bus->children); -up_write (&pci_bus_sem); - } -pci_bus_add_devices (dev->subordinate); - -/* register the bus with sysfs as the parent is now -* properly registered. */ - child_bus = dev->subordinate; -if (child_bus->is_added) -continue ; - child_bus->dev.parent = child_bus->bridge; - retval =device_register (&child_bus->dev); -if (retval) -dev_err (&dev->dev,"Error registering pci_bus," -" continuing...\n" ); -else { - child_bus->is_added =1 ; - retval =device_create_file (&child_bus->dev, - &dev_attr_cpuaffinity); - } +if (!child) +continue ; +if (list_empty (&child->node)) { +down_write (&pci_bus_sem); +list_add_tail (&child->node, &dev->bus->children); +up_write (&pci_bus_sem); + } +pci_bus_add_devices (child); + +/* +* register the bus with sysfs as the parent is now +* properly registered. +*/ +if (child->is_added) +continue ; + child->dev.parent = child->bridge; + retval =device_register (&child->dev); +if (retval) +dev_err (&dev->dev,"Error registering pci_bus," +" continuing...\n" ); +else { + child->is_added =1 ; + retval =device_create_file (&child->dev, + &dev_attr_cpuaffinity);if (retval)dev_err (&dev->dev,"Error creating cpuaffinity" " file, continuing...\n" ); - retval =device_create_file (&child_bus->dev, + retval =device_create_file (&child->dev, &dev_attr_cpulistaffinity);if (retval) dev_err(&dev->dev,
Signature · 1 line
1.5.6.4
PATCH v7 8/13 PCI: cleanup pci_bus_add_devices()
In reply to Yu ZhaoThis cleanup makes pci_bus_add_devices() easier to read.
--- drivers/pci/bus.c | 55 +++++++++++++++++++++++++++-------------------------- 1 files changed, 28 insertions(+), 27 deletions(-)
drivers/pci/bus.c +28 -27
@@ -71,7 +71,7 @@ pci_bus_alloc_resource(struct pci_bus *bus, struct resource *res, }/** -* add a single device +* pci_bus_add_device - add a single device * @dev: device to add * * This adds a single pci device to the global
@@ -105,7 +105,7 @@ int pci_bus_add_device(struct pci_dev *dev)void pci_bus_add_devices (struct pci_bus *bus) {struct pci_dev *dev; -struct pci_bus *child_bus; +struct pci_bus *child;int retval;list_for_each_entry (dev, &bus->devices, bus_list) {
@@ -120,39 +120,40 @@ void pci_bus_add_devices(struct pci_bus *bus)list_for_each_entry (dev, &bus->devices, bus_list) {BUG_ON (!dev->is_added); + child = dev->subordinate;/* * If there is an unattached subordinate bus, attach * it and then scan for unattached PCI devices. */ -if (dev->subordinate) { -if (list_empty (&dev->subordinate->node)) { -down_write (&pci_bus_sem); -list_add_tail (&dev->subordinate->node, - &dev->bus->children); -up_write (&pci_bus_sem); - } -pci_bus_add_devices (dev->subordinate); - -/* register the bus with sysfs as the parent is now -* properly registered. */ - child_bus = dev->subordinate; -if (child_bus->is_added) -continue ; - child_bus->dev.parent = child_bus->bridge; - retval =device_register (&child_bus->dev); -if (retval) -dev_err (&dev->dev,"Error registering pci_bus," -" continuing...\n" ); -else { - child_bus->is_added =1 ; - retval =device_create_file (&child_bus->dev, - &dev_attr_cpuaffinity); - } +if (!child) +continue ; +if (list_empty (&child->node)) { +down_write (&pci_bus_sem); +list_add_tail (&child->node, &dev->bus->children); +up_write (&pci_bus_sem); + } +pci_bus_add_devices (child); + +/* +* register the bus with sysfs as the parent is now +* properly registered. +*/ +if (child->is_added) +continue ; + child->dev.parent = child->bridge; + retval =device_register (&child->dev); +if (retval) +dev_err (&dev->dev,"Error registering pci_bus," +" continuing...\n" ); +else { + child->is_added =1 ; + retval =device_create_file (&child->dev, + &dev_attr_cpuaffinity);if (retval)dev_err (&dev->dev,"Error creating cpuaffinity" " file, continuing...\n" ); - retval =device_create_file (&child_bus->dev, + retval =device_create_file (&child->dev, &dev_attr_cpulistaffinity);if (retval) dev_err(&dev->dev,
Signature · 1 line
1.5.6.4
PATCH v7 9/13 PCI: split a new function from pci_bus_add_devices()
In reply to Yu ZhaoThis patch splits a new function from pci_bus_add_devices(). The new function can be used to register PCI bus to the device core.
--- drivers/pci/bus.c | 49 ++++++++++++++++++++++++++++++------------------- drivers/pci/pci.h | 1 + 2 files changed, 31 insertions(+), 19 deletions(-)
drivers/pci/bus.c +30 -19
@@ -91,6 +91,34 @@ int pci_bus_add_device(struct pci_dev *dev) }/** +* pci_bus_add_child - add a child bus +* @bus: bus to add +* +* This adds sysfs entries for a single bus +*/ +int pci_bus_add_child (struct pci_bus *bus) +{ +int retval; + +if (bus->bridge) + bus->dev.parent = bus->bridge; + + retval =device_register (&bus->dev); +if (retval) +return retval; + + bus->is_added =1 ; + + retval =device_create_file (&bus->dev, &dev_attr_cpuaffinity); +if (retval) +return retval; + + retval =device_create_file (&bus->dev, &dev_attr_cpulistaffinity); + +return retval; +} + +/** * pci_bus_add_devices - insert newly discovered PCI devices * @bus: bus to check for new devices *
@@ -140,26 +168,9 @@ void pci_bus_add_devices(struct pci_bus *bus)*/ if (child->is_added)continue ; - child->dev.parent = child->bridge; - retval =device_register (&child->dev); + retval =pci_bus_add_child (child);if (retval) -dev_err (&dev->dev,"Error registering pci_bus," -" continuing...\n" ); -else { - child->is_added =1 ; - retval =device_create_file (&child->dev, - &dev_attr_cpuaffinity); -if (retval) -dev_err (&dev->dev,"Error creating cpuaffinity" -" file, continuing...\n" ); - - retval =device_create_file (&child->dev, - &dev_attr_cpulistaffinity); -if (retval) -dev_err (&dev->dev, -"Error creating cpulistaffinity" -" file, continuing...\n" ); - } +dev_err (&dev->dev,"Error adding bus, continuing\n" ); } }
drivers/pci/pci.h +1 -0
@@ -170,6 +170,7 @@ extern int __pci_read_base(struct pci_dev *dev, enum pci_bar_type type, struct resource *res, unsigned int reg);extern int pci_resource_bar (struct pci_dev *dev,int resno,enum pci_bar_type *type); +extern int pci_bus_add_child (struct pci_bus *bus);extern void pci_enable_ari (struct pci_dev *dev);/** * pci_ari_enabled - query ARI forwarding status
Signature · 1 line
1.5.6.4
PATCH v7 9/13 PCI: split a new function from pci_bus_add_devices()
In reply to Yu ZhaoThis patch splits a new function from pci_bus_add_devices(). The new function can be used to register PCI bus to the device core.
--- drivers/pci/bus.c | 49 ++++++++++++++++++++++++++++++------------------- drivers/pci/pci.h | 1 + 2 files changed, 31 insertions(+), 19 deletions(-)
drivers/pci/bus.c +30 -19
@@ -91,6 +91,34 @@ int pci_bus_add_device(struct pci_dev *dev) }/** +* pci_bus_add_child - add a child bus +* @bus: bus to add +* +* This adds sysfs entries for a single bus +*/ +int pci_bus_add_child (struct pci_bus *bus) +{ +int retval; + +if (bus->bridge) + bus->dev.parent = bus->bridge; + + retval =device_register (&bus->dev); +if (retval) +return retval; + + bus->is_added =1 ; + + retval =device_create_file (&bus->dev, &dev_attr_cpuaffinity); +if (retval) +return retval; + + retval =device_create_file (&bus->dev, &dev_attr_cpulistaffinity); + +return retval; +} + +/** * pci_bus_add_devices - insert newly discovered PCI devices * @bus: bus to check for new devices *
@@ -140,26 +168,9 @@ void pci_bus_add_devices(struct pci_bus *bus)*/ if (child->is_added)continue ; - child->dev.parent = child->bridge; - retval =device_register (&child->dev); + retval =pci_bus_add_child (child);if (retval) -dev_err (&dev->dev,"Error registering pci_bus," -" continuing...\n" ); -else { - child->is_added =1 ; - retval =device_create_file (&child->dev, - &dev_attr_cpuaffinity); -if (retval) -dev_err (&dev->dev,"Error creating cpuaffinity" -" file, continuing...\n" ); - - retval =device_create_file (&child->dev, - &dev_attr_cpulistaffinity); -if (retval) -dev_err (&dev->dev, -"Error creating cpulistaffinity" -" file, continuing...\n" ); - } +dev_err (&dev->dev,"Error adding bus, continuing\n" ); } }
drivers/pci/pci.h +1 -0
@@ -170,6 +170,7 @@ extern int __pci_read_base(struct pci_dev *dev, enum pci_bar_type type, struct resource *res, unsigned int reg);extern int pci_resource_bar (struct pci_dev *dev,int resno,enum pci_bar_type *type); +extern int pci_bus_add_child (struct pci_bus *bus);extern void pci_enable_ari (struct pci_dev *dev);/** * pci_ari_enabled - query ARI forwarding status
Signature · 1 line
1.5.6.4
PATCH v7 10/13 PCI: support the SR-IOV capability
In reply to Yu ZhaoSupport Single Root I/O Virtualization (SR-IOV) capability.
--- drivers/pci/Kconfig | 13 ++ drivers/pci/Makefile | 3 + drivers/pci/iov.c | 491 ++++++++++++++++++++++++++++++++++++++++++++++ drivers/pci/pci-driver.c | 12 +- drivers/pci/pci.c | 8 + drivers/pci/pci.h | 51 +++++ drivers/pci/probe.c | 4 + include/linux/pci.h | 9 + include/linux/pci_regs.h | 21 ++ 9 files changed, 610 insertions(+), 2 deletions(-) create mode 100644 drivers/pci/iov.c
drivers/pci/Kconfig +13 -0
@@ -50,3 +50,16 @@ config HT_IRQ This allows native hypertransport devices to use interrupts. If unsure say Y. + +config PCI_IOV + bool "PCI IOV support" + depends on PCI + select PCI_MSI + default n + help + PCI-SIG I/O Virtualization (IOV) Specifications support. + Single Root IOV: allows the Physical Function device driver + to enable the hardware capability, so the Virtual Function + is accessible via the PCI configuration space using its own + Bus, Device and Function Number. Each Virtual Function also + has PCI Memory Space to map its own register set.
drivers/pci/Makefile +3 -0
@@ -29,6 +29,9 @@ obj-$(CONFIG_DMAR) += dmar.o iova.o intel-iommu.o obj-$(CONFIG_INTR_REMAP) += dmar.o intr_remapping.o +# PCI IOV support +obj-$(CONFIG_PCI_IOV) += iov.o + # # Some architectures use the generic PCI setup functions #
drivers/pci/iov.c new file +491 -0
491 lines not on this page · raw message
drivers/pci/pci-driver.c +10 -2
@@ -234,6 +234,8 @@ __pci_device_probe(struct pci_driver *drv, struct pci_dev *pci_dev) error =pci_call_probe (drv, pci_dev, id);if (error >=0 ) { pci_dev->driver = drv; +if (drv->virtual) +pci_iov_register (pci_dev); error =0 ; } }
@@ -262,6 +264,8 @@ static int pci_device_remove(struct device * dev)struct pci_driver * drv = pci_dev->driver;if (drv) { +if (drv->virtual) +pci_iov_unregister (pci_dev);if (drv->remove) drv->remove (pci_dev); pci_dev->driver =NULL ;
@@ -292,8 +296,12 @@ static void pci_device_shutdown(struct device *dev)struct pci_dev *pci_dev =to_pci_dev (dev);struct pci_driver *drv = pci_dev->driver; -if (drv && drv->shutdown) - drv->shutdown (pci_dev); +if (drv) { +if (drv->virtual) +pci_iov_unregister (pci_dev); +if (drv->shutdown) + drv->shutdown (pci_dev); + }pci_msi_shutdown (pci_dev);pci_msix_shutdown (pci_dev); }
drivers/pci/pci.c +8 -0
@@ -763,6 +763,7 @@ pci_restore_state(struct pci_dev *dev) }pci_restore_pcix_state (dev);pci_restore_msi_state (dev); +pci_restore_iov_state (dev);return 0 ; }
@@ -2017,12 +2018,19 @@ int pci_select_bars(struct pci_dev *dev, unsigned long flags)*/ int pci_resource_bar (struct pci_dev *dev,int resno,enum pci_bar_type *type) { +int reg; +if (resno <PCI_ROM_RESOURCE ) { *type = pci_bar_unknown;return PCI_BASE_ADDRESS_0 +4 * resno; }else if (resno ==PCI_ROM_RESOURCE ) { *type = pci_bar_mem32;return dev->rom_base_reg; + }else if (resno <PCI_BRIDGE_RESOURCES ) { +/* device specific resource */ + reg =pci_iov_resource_bar (dev, resno, type); +if (reg) +return reg; }dev_err (&dev->dev,"BAR: invalid resource #%d\n" , resno);
drivers/pci/pci.h +51 -0
55 lines not on this page · raw message
drivers/pci/probe.c +4 -0
@@ -785,6 +785,7 @@ static int pci_setup_device(struct pci_dev * dev)static void pci_release_capabilities (struct pci_dev *dev) {pci_vpd_release (dev); +pci_iov_release (dev); }/**
@@ -968,6 +969,9 @@ static void pci_init_capabilities(struct pci_dev *dev)/* Alternative Routing-ID Forwarding */ pci_enable_ari (dev); + +/* Single Root I/O Virtualization */ +pci_iov_init (dev); }void pci_device_add (struct pci_dev *dev,struct pci_bus *bus)
include/linux/pci.h +9 -0
@@ -93,6 +93,12 @@ enum {/* #6: expansion ROM resource */ PCI_ROM_RESOURCE, +/* device specific resources */ +#ifdef CONFIG_PCI_IOV + PCI_IOV_RESOURCES, + PCI_IOV_RESOURCE_END = PCI_IOV_RESOURCES + PCI_IOV_NUM_BAR - 1, +#endif +/* resources assigned to buses behind the bridge */ #define PCI_BRIDGE_RESOURCE_NUM 4
@@ -171,6 +177,7 @@ struct pci_cap_saved_state {struct pcie_link_state ;struct pci_vpd ; +struct pci_iov ;/* * The pci_dev structure is used to describe PCI devices.
@@ -259,6 +266,7 @@ struct pci_dev {struct list_head msi_list;#endif struct pci_vpd *vpd; +struct pci_iov *iov; };extern struct pci_dev *alloc_pci_dev (void );
@@ -426,6 +434,7 @@ struct pci_driver {int (*resume_early) (struct pci_dev *dev);int (*resume) (struct pci_dev *dev);/* Device woken up */ void (*shutdown) (struct pci_dev *dev); +int (*virtual) (struct pci_dev *dev,int nr_virtfn);struct pm_ext_ops *pm;struct pci_error_handlers *err_handler;struct device_driver driver;
include/linux/pci_regs.h +21 -0
@@ -363,6 +363,7 @@#define PCI_EXP_TYPE_UPSTREAM 0x5/* Upstream Port */ #define PCI_EXP_TYPE_DOWNSTREAM 0x6/* Downstream Port */ #define PCI_EXP_TYPE_PCI_BRIDGE 0x7/* PCI/PCI-X Bridge */ +#define PCI_EXP_TYPE_RC_END 0x9/* Root Complex Integrated Endpoint */ #define PCI_EXP_FLAGS_SLOT 0x0100/* Slot implemented */ #define PCI_EXP_FLAGS_IRQ 0x3e00/* Interrupt message number */ #define PCI_EXP_DEVCAP 4/* Device capabilities */
@@ -436,6 +437,7 @@#define PCI_EXT_CAP_ID_DSN 3#define PCI_EXT_CAP_ID_PWR 4#define PCI_EXT_CAP_ID_ARI 14 +#define PCI_EXT_CAP_ID_IOV 16/* Advanced Error Reporting */ #define PCI_ERR_UNCOR_STATUS 4/* Uncorrectable Error Status */
@@ -553,4 +555,23 @@#define PCI_ARI_CTRL_ACS 0x0002/* ACS Function Groups Enable */ #define PCI_ARI_CTRL_FG (x) (((x) >> 4) & 7)/* Function Group */ +/* Single Root I/O Virtualization */ +#define PCI_IOV_CAP 0x04/* SR-IOV Capabilities */ +#define PCI_IOV_CTRL 0x08/* SR-IOV Control */ +#define PCI_IOV_CTRL_VFE 0x01/* VF Enable */ +#define PCI_IOV_CTRL_MSE 0x08/* VF Memory Space Enable */ +#define PCI_IOV_CTRL_ARI 0x10/* ARI Capable Hierarchy */ +#define PCI_IOV_STATUS 0x0a/* SR-IOV Status */ +#define PCI_IOV_INITIAL_VF 0x0c/* Initial VFs */ +#define PCI_IOV_TOTAL_VF 0x0e/* Total VFs */ +#define PCI_IOV_NUM_VF 0x10/* Number of VFs */ +#define PCI_IOV_FUNC_LINK 0x12/* Function Dependency Link */ +#define PCI_IOV_VF_OFFSET 0x14/* First VF Offset */ +#define PCI_IOV_VF_STRIDE 0x16/* Following VF Stride */ +#define PCI_IOV_VF_DID 0x1a/* VF Device ID */ +#define PCI_IOV_SUP_PGSIZE 0x1c/* Supported Page Sizes */ +#define PCI_IOV_SYS_PGSIZE 0x20/* System Page Size */ +#define PCI_IOV_BAR_0 0x24/* VF BAR0 */ +#define PCI_IOV_NUM_BAR 6/* Number of VF BARs */ +#endif /* LINUX_PCI_REGS_H */
Signature · 1 line
1.5.6.4
PATCH v7 10/13 PCI: support the SR-IOV capability
In reply to Yu ZhaoSupport Single Root I/O Virtualization (SR-IOV) capability.
--- drivers/pci/Kconfig | 13 ++ drivers/pci/Makefile | 3 + drivers/pci/iov.c | 491 ++++++++++++++++++++++++++++++++++++++++++++++ drivers/pci/pci-driver.c | 12 +- drivers/pci/pci.c | 8 + drivers/pci/pci.h | 51 +++++ drivers/pci/probe.c | 4 + include/linux/pci.h | 9 + include/linux/pci_regs.h | 21 ++ 9 files changed, 610 insertions(+), 2 deletions(-) create mode 100644 drivers/pci/iov.c
drivers/pci/Kconfig +13 -0
@@ -50,3 +50,16 @@ config HT_IRQ This allows native hypertransport devices to use interrupts. If unsure say Y. + +config PCI_IOV + bool "PCI IOV support" + depends on PCI + select PCI_MSI + default n + help + PCI-SIG I/O Virtualization (IOV) Specifications support. + Single Root IOV: allows the Physical Function device driver + to enable the hardware capability, so the Virtual Function + is accessible via the PCI configuration space using its own + Bus, Device and Function Number. Each Virtual Function also + has PCI Memory Space to map its own register set.
drivers/pci/Makefile +3 -0
@@ -29,6 +29,9 @@ obj-$(CONFIG_DMAR) += dmar.o iova.o intel-iommu.o obj-$(CONFIG_INTR_REMAP) += dmar.o intr_remapping.o +# PCI IOV support +obj-$(CONFIG_PCI_IOV) += iov.o + # # Some architectures use the generic PCI setup functions #
drivers/pci/iov.c new file +491 -0
491 lines not on this page · raw message
drivers/pci/pci-driver.c +10 -2
@@ -234,6 +234,8 @@ __pci_device_probe(struct pci_driver *drv, struct pci_dev *pci_dev) error =pci_call_probe (drv, pci_dev, id);if (error >=0 ) { pci_dev->driver = drv; +if (drv->virtual) +pci_iov_register (pci_dev); error =0 ; } }
@@ -262,6 +264,8 @@ static int pci_device_remove(struct device * dev)struct pci_driver * drv = pci_dev->driver;if (drv) { +if (drv->virtual) +pci_iov_unregister (pci_dev);if (drv->remove) drv->remove (pci_dev); pci_dev->driver =NULL ;
@@ -292,8 +296,12 @@ static void pci_device_shutdown(struct device *dev)struct pci_dev *pci_dev =to_pci_dev (dev);struct pci_driver *drv = pci_dev->driver; -if (drv && drv->shutdown) - drv->shutdown (pci_dev); +if (drv) { +if (drv->virtual) +pci_iov_unregister (pci_dev); +if (drv->shutdown) + drv->shutdown (pci_dev); + }pci_msi_shutdown (pci_dev);pci_msix_shutdown (pci_dev); }
drivers/pci/pci.c +8 -0
@@ -763,6 +763,7 @@ pci_restore_state(struct pci_dev *dev) }pci_restore_pcix_state (dev);pci_restore_msi_state (dev); +pci_restore_iov_state (dev);return 0 ; }
@@ -2017,12 +2018,19 @@ int pci_select_bars(struct pci_dev *dev, unsigned long flags)*/ int pci_resource_bar (struct pci_dev *dev,int resno,enum pci_bar_type *type) { +int reg; +if (resno <PCI_ROM_RESOURCE ) { *type = pci_bar_unknown;return PCI_BASE_ADDRESS_0 +4 * resno; }else if (resno ==PCI_ROM_RESOURCE ) { *type = pci_bar_mem32;return dev->rom_base_reg; + }else if (resno <PCI_BRIDGE_RESOURCES ) { +/* device specific resource */ + reg =pci_iov_resource_bar (dev, resno, type); +if (reg) +return reg; }dev_err (&dev->dev,"BAR: invalid resource #%d\n" , resno);
drivers/pci/pci.h +51 -0
55 lines not on this page · raw message
drivers/pci/probe.c +4 -0
@@ -785,6 +785,7 @@ static int pci_setup_device(struct pci_dev * dev)static void pci_release_capabilities (struct pci_dev *dev) {pci_vpd_release (dev); +pci_iov_release (dev); }/**
@@ -968,6 +969,9 @@ static void pci_init_capabilities(struct pci_dev *dev)/* Alternative Routing-ID Forwarding */ pci_enable_ari (dev); + +/* Single Root I/O Virtualization */ +pci_iov_init (dev); }void pci_device_add (struct pci_dev *dev,struct pci_bus *bus)
include/linux/pci.h +9 -0
@@ -93,6 +93,12 @@ enum {/* #6: expansion ROM resource */ PCI_ROM_RESOURCE, +/* device specific resources */ +#ifdef CONFIG_PCI_IOV + PCI_IOV_RESOURCES, + PCI_IOV_RESOURCE_END = PCI_IOV_RESOURCES + PCI_IOV_NUM_BAR - 1, +#endif +/* resources assigned to buses behind the bridge */ #define PCI_BRIDGE_RESOURCE_NUM 4
@@ -171,6 +177,7 @@ struct pci_cap_saved_state {struct pcie_link_state ;struct pci_vpd ; +struct pci_iov ;/* * The pci_dev structure is used to describe PCI devices.
@@ -259,6 +266,7 @@ struct pci_dev {struct list_head msi_list;#endif struct pci_vpd *vpd; +struct pci_iov *iov; };extern struct pci_dev *alloc_pci_dev (void );
@@ -426,6 +434,7 @@ struct pci_driver {int (*resume_early) (struct pci_dev *dev);int (*resume) (struct pci_dev *dev);/* Device woken up */ void (*shutdown) (struct pci_dev *dev); +int (*virtual) (struct pci_dev *dev,int nr_virtfn);struct pm_ext_ops *pm;struct pci_error_handlers *err_handler;struct device_driver driver;
include/linux/pci_regs.h +21 -0
@@ -363,6 +363,7 @@#define PCI_EXP_TYPE_UPSTREAM 0x5/* Upstream Port */ #define PCI_EXP_TYPE_DOWNSTREAM 0x6/* Downstream Port */ #define PCI_EXP_TYPE_PCI_BRIDGE 0x7/* PCI/PCI-X Bridge */ +#define PCI_EXP_TYPE_RC_END 0x9/* Root Complex Integrated Endpoint */ #define PCI_EXP_FLAGS_SLOT 0x0100/* Slot implemented */ #define PCI_EXP_FLAGS_IRQ 0x3e00/* Interrupt message number */ #define PCI_EXP_DEVCAP 4/* Device capabilities */
@@ -436,6 +437,7 @@#define PCI_EXT_CAP_ID_DSN 3#define PCI_EXT_CAP_ID_PWR 4#define PCI_EXT_CAP_ID_ARI 14 +#define PCI_EXT_CAP_ID_IOV 16/* Advanced Error Reporting */ #define PCI_ERR_UNCOR_STATUS 4/* Uncorrectable Error Status */
@@ -553,4 +555,23 @@#define PCI_ARI_CTRL_ACS 0x0002/* ACS Function Groups Enable */ #define PCI_ARI_CTRL_FG (x) (((x) >> 4) & 7)/* Function Group */ +/* Single Root I/O Virtualization */ +#define PCI_IOV_CAP 0x04/* SR-IOV Capabilities */ +#define PCI_IOV_CTRL 0x08/* SR-IOV Control */ +#define PCI_IOV_CTRL_VFE 0x01/* VF Enable */ +#define PCI_IOV_CTRL_MSE 0x08/* VF Memory Space Enable */ +#define PCI_IOV_CTRL_ARI 0x10/* ARI Capable Hierarchy */ +#define PCI_IOV_STATUS 0x0a/* SR-IOV Status */ +#define PCI_IOV_INITIAL_VF 0x0c/* Initial VFs */ +#define PCI_IOV_TOTAL_VF 0x0e/* Total VFs */ +#define PCI_IOV_NUM_VF 0x10/* Number of VFs */ +#define PCI_IOV_FUNC_LINK 0x12/* Function Dependency Link */ +#define PCI_IOV_VF_OFFSET 0x14/* First VF Offset */ +#define PCI_IOV_VF_STRIDE 0x16/* Following VF Stride */ +#define PCI_IOV_VF_DID 0x1a/* VF Device ID */ +#define PCI_IOV_SUP_PGSIZE 0x1c/* Supported Page Sizes */ +#define PCI_IOV_SYS_PGSIZE 0x20/* System Page Size */ +#define PCI_IOV_BAR_0 0x24/* VF BAR0 */ +#define PCI_IOV_NUM_BAR 6/* Number of VF BARs */ +#endif /* LINUX_PCI_REGS_H */
Signature · 1 line
1.5.6.4
PATCH v7 11/13 PCI: reserve bus range for SR-IOV device
In reply to Yu ZhaoReserve bus range for SR-IOV at device scanning stage when the kernel boot parameter 'pci=assign-busses' is used .
--- drivers/pci/iov.c | 24 ++++++++++++++++++++++++ drivers/pci/pci.h | 6 ++++++ drivers/pci/probe.c | 3 +++ 3 files changed, 33 insertions(+), 0 deletions(-)
drivers/pci/iov.c +24 -0
@@ -489,3 +489,27 @@ void pci_iov_unregister(struct pci_dev *dev)iov_disable (dev);kobject_uevent (&dev->dev.kobj,KOBJ_CHANGE ); } + +/** +* pci_iov_bus_range - find bus range used by the Virtual Function +* @bus: the PCI bus +* +* Returns max number of buses (exclude current one) used by Virtual +* Functions. +*/ +int pci_iov_bus_range (struct pci_bus *bus) +{ +int max =0 ; +u8 busnr, devfn; +struct pci_dev *dev; + +list_for_each_entry (dev, &bus->devices, bus_list) { +if (!dev->iov) +continue ; +virtfn_bdf (dev, dev->iov->total -1 , &busnr, &devfn); +if (busnr > max) + max = busnr; + } + +return max ? max - bus->number :0 ; +}
drivers/pci/pci.h +6 -0
@@ -205,6 +205,7 @@ extern int pci_iov_resource_bar(struct pci_dev *dev, int resno,extern int pci_iov_register (struct pci_dev * dev);extern void pci_iov_unregister (struct pci_dev * dev);extern void pci_restore_iov_state (struct pci_dev *dev); +extern int pci_iov_bus_range (struct pci_bus *bus);#else static inline int pci_iov_init (struct pci_dev *dev) {
@@ -232,6 +233,11 @@ static inline void pci_iov_unregister(struct pci_dev* dev)static inline void pci_restore_iov_state (struct pci_dev *dev) { } + +static inline int pci_iov_bus_range (struct pci_bus *bus) +{ +return 0 ; +}#endif /* CONFIG_PCI_IOV */ #endif /* DRIVERS_PCI_H */
drivers/pci/probe.c +3 -0
@@ -1074,6 +1074,9 @@ unsigned int __devinit pci_scan_child_bus(struct pci_bus *bus)for (devfn =0 ; devfn <0x100 ; devfn +=8 )pci_scan_slot (bus, devfn); +/* Reserve buses for SR-IOV capability. */ + max +=pci_iov_bus_range (bus); +/* * After performing arch-dependent fixup of the bus, look behind * all PCI-to-PCI bridges on this bus.
Signature · 1 line
1.5.6.4
PATCH v7 11/13 PCI: reserve bus range for SR-IOV device
In reply to Yu ZhaoReserve bus range for SR-IOV at device scanning stage when the kernel boot parameter 'pci=assign-busses' is used .
--- drivers/pci/iov.c | 24 ++++++++++++++++++++++++ drivers/pci/pci.h | 6 ++++++ drivers/pci/probe.c | 3 +++ 3 files changed, 33 insertions(+), 0 deletions(-)
drivers/pci/iov.c +24 -0
@@ -489,3 +489,27 @@ void pci_iov_unregister(struct pci_dev *dev)iov_disable (dev);kobject_uevent (&dev->dev.kobj,KOBJ_CHANGE ); } + +/** +* pci_iov_bus_range - find bus range used by the Virtual Function +* @bus: the PCI bus +* +* Returns max number of buses (exclude current one) used by Virtual +* Functions. +*/ +int pci_iov_bus_range (struct pci_bus *bus) +{ +int max =0 ; +u8 busnr, devfn; +struct pci_dev *dev; + +list_for_each_entry (dev, &bus->devices, bus_list) { +if (!dev->iov) +continue ; +virtfn_bdf (dev, dev->iov->total -1 , &busnr, &devfn); +if (busnr > max) + max = busnr; + } + +return max ? max - bus->number :0 ; +}
drivers/pci/pci.h +6 -0
@@ -205,6 +205,7 @@ extern int pci_iov_resource_bar(struct pci_dev *dev, int resno,extern int pci_iov_register (struct pci_dev * dev);extern void pci_iov_unregister (struct pci_dev * dev);extern void pci_restore_iov_state (struct pci_dev *dev); +extern int pci_iov_bus_range (struct pci_bus *bus);#else static inline int pci_iov_init (struct pci_dev *dev) {
@@ -232,6 +233,11 @@ static inline void pci_iov_unregister(struct pci_dev* dev)static inline void pci_restore_iov_state (struct pci_dev *dev) { } + +static inline int pci_iov_bus_range (struct pci_bus *bus) +{ +return 0 ; +}#endif /* CONFIG_PCI_IOV */ #endif /* DRIVERS_PCI_H */
drivers/pci/probe.c +3 -0
@@ -1074,6 +1074,9 @@ unsigned int __devinit pci_scan_child_bus(struct pci_bus *bus)for (devfn =0 ; devfn <0x100 ; devfn +=8 )pci_scan_slot (bus, devfn); +/* Reserve buses for SR-IOV capability. */ + max +=pci_iov_bus_range (bus); +/* * After performing arch-dependent fixup of the bus, look behind * all PCI-to-PCI bridges on this bus.
Signature · 1 line
1.5.6.4
PATCH v7 12/13 PCI: document the SR-IOV sysfs entries
In reply to Yu ZhaoDocument the SR-IOV sysfs entries.
--- Documentation/ABI/testing/sysfs-bus-pci | 26 ++++++++++++++++++++++++++ 1 files changed, 26 insertions(+), 0 deletions(-)
Documentation/ABI/testing/sysfs-bus-pci +26 -0
@@ -9,3 +9,29 @@ Description: that some devices may have malformatted data. If the underlying VPD has a writable section then the corresponding section of this file will be writable. + +What: /sys/bus/pci/devices/.../iov/total_virtfn +Date: November 2008 +Contact: Yu Zhao <[email protected]> +Description: + This file appears when a device has the SR-IOV capability + and the device driver (PF driver) support this operation. + It holds the number of total Virtual Functions (read-only). + +What: /sys/bus/pci/devices/.../iov/initial_virtfn +Date: November 2008 +Contact: Yu Zhao <[email protected]> +Description: + This file appears when a device has the SR-IOV capability + and the device driver (PF driver) support this operation. + It holds the number of initial Virtual Functions (read-only). + +What: /sys/bus/pci/devices/.../iov/nr_virtfn +Date: November 2008 +Contact: Yu Zhao <[email protected]> +Description: + This file appears when a device has the SR-IOV capability + and the device driver (PF driver) support this operation. + It holds the number of available Virtual Functions, and + could be written (0 ~ InitialVFs) to change the number of + the Virtual Functions.
Signature · 1 line
1.5.6.4
PATCH v7 12/13 PCI: document the SR-IOV sysfs entries
In reply to Yu ZhaoDocument the SR-IOV sysfs entries.
--- Documentation/ABI/testing/sysfs-bus-pci | 26 ++++++++++++++++++++++++++ 1 files changed, 26 insertions(+), 0 deletions(-)
Documentation/ABI/testing/sysfs-bus-pci +26 -0
@@ -9,3 +9,29 @@ Description: that some devices may have malformatted data. If the underlying VPD has a writable section then the corresponding section of this file will be writable. + +What: /sys/bus/pci/devices/.../iov/total_virtfn +Date: November 2008 +Contact: Yu Zhao <[email protected]> +Description: + This file appears when a device has the SR-IOV capability + and the device driver (PF driver) support this operation. + It holds the number of total Virtual Functions (read-only). + +What: /sys/bus/pci/devices/.../iov/initial_virtfn +Date: November 2008 +Contact: Yu Zhao <[email protected]> +Description: + This file appears when a device has the SR-IOV capability + and the device driver (PF driver) support this operation. + It holds the number of initial Virtual Functions (read-only). + +What: /sys/bus/pci/devices/.../iov/nr_virtfn +Date: November 2008 +Contact: Yu Zhao <[email protected]> +Description: + This file appears when a device has the SR-IOV capability + and the device driver (PF driver) support this operation. + It holds the number of available Virtual Functions, and + could be written (0 ~ InitialVFs) to change the number of + the Virtual Functions.
Signature · 1 line
1.5.6.4
PATCH v7 13/13 PCI: document for SR-IOV user and developer
In reply to Yu ZhaoCreate how-to for the SR-IOV user and driver developer.
--- Documentation/DocBook/kernel-api.tmpl | 1 + Documentation/PCI/pci-iov-howto.txt | 138 +++++++++++++++++++++++++++++++++ 2 files changed, 139 insertions(+), 0 deletions(-) create mode 100644 Documentation/PCI/pci-iov-howto.txt
Documentation/DocBook/kernel-api.tmpl +1 -0
@@ -251,6 +251,7 @@ X!Edrivers/pci/hotplug.c --> !Edrivers/pci/probe.c !Edrivers/pci/rom.c +!Edrivers/pci/iov.c </sect1> <sect1><title>PCI Hotplug Support Library</title> !Edrivers/pci/hotplug/pci_hotplug_core.c
Documentation/PCI/pci-iov-howto.txt new file +138 -0
138 lines not on this page · raw message
Signature · 1 line
1.5.6.4
PATCH v7 13/13 PCI: document for SR-IOV user and developer
In reply to Yu ZhaoCreate how-to for the SR-IOV user and driver developer.
--- Documentation/DocBook/kernel-api.tmpl | 1 + Documentation/PCI/pci-iov-howto.txt | 138 +++++++++++++++++++++++++++++++++ 2 files changed, 139 insertions(+), 0 deletions(-) create mode 100644 Documentation/PCI/pci-iov-howto.txt
Documentation/DocBook/kernel-api.tmpl +1 -0
@@ -251,6 +251,7 @@ X!Edrivers/pci/hotplug.c --> !Edrivers/pci/probe.c !Edrivers/pci/rom.c +!Edrivers/pci/iov.c </sect1> <sect1><title>PCI Hotplug Support Library</title> !Edrivers/pci/hotplug/pci_hotplug_core.c
Documentation/PCI/pci-iov-howto.txt new file +138 -0
138 lines not on this page · raw message
Signature · 1 line
1.5.6.4
Quoted 10 lines · Yu Zhao
Greetings,
Following patches are intended to support SR-IOV capability in the Linux kernel. With these patches, people can turn a PCI device with the capability into multiple ones from software perspective, which will benefit KVM and achieve other purposes such as QoS, security, and etc.
The Physical Function and Virtual Function drivers using the SR-IOV APIs will come soon!
Thanks for respining these patches, but I think we really need to see a driver using this in order to get an idea of how it will be used.
Also, the Xen and KVM people need to agree on the userspace interface here, perhaps also getting some libvirt involvement as well, as they are going to be the ones having to use this all the time.
thanks,
greg k-h
Quoted 9 lines · Greg KH
Quoted 8 lines · Yu Zhao
Greetings,
Following patches are intended to support SR-IOV capability in the Linux kernel. With these patches, people can turn a PCI device with the capability into multiple ones from software perspective, which will benefit KVM and achieve other purposes such as QoS, security, and etc.
Quoted 5 lines
Quoted 2 lines
The Physical Function and Virtual Function drivers using the SR-IOV APIs will come soon!
Thanks for respining these patches, but I think we really need to see a driver using this in order to get an idea of how it will be used.
Yes, the PF driver patch and the VF driver for Intel 82576 NIC will be available next week. Both PF/VF drivers are testing versions. The PF driver patch is based on the lasted kernel IGB driver (drivers/net/igb/), and uses SR-IOV v7 API. The VF driver is a totally new NIC driver, it looks like other normal PCI NIC drivers.
Quoted 3 lines
Also, the Xen and KVM people need to agree on the userspace interface here, perhaps also getting some libvirt involvement as well, as they are going to be the ones having to use this all the time.
I'll keep KVM/Xen people updated on the user level interface and let them comment based on a real usage model of the Intel 82576 NIC after PF/VF drivers are available.
Regards,
Yu
Quoted 9 lines · Greg KH
Quoted 8 lines · Yu Zhao
Greetings,
Following patches are intended to support SR-IOV capability in the Linux kernel. With these patches, people can turn a PCI device with the capability into multiple ones from software perspective, which will benefit KVM and achieve other purposes such as QoS, security, and etc.
Quoted 5 lines
Quoted 2 lines
The Physical Function and Virtual Function drivers using the SR-IOV APIs will come soon!
Thanks for respining these patches, but I think we really need to see a driver using this in order to get an idea of how it will be used.
Yes, the PF driver patch and the VF driver for Intel 82576 NIC will be available next week. Both PF/VF drivers are testing versions. The PF driver patch is based on the lasted kernel IGB driver (drivers/net/igb/), and uses SR-IOV v7 API. The VF driver is a totally new NIC driver, it looks like other normal PCI NIC drivers.
Quoted 3 lines
Also, the Xen and KVM people need to agree on the userspace interface here, perhaps also getting some libvirt involvement as well, as they are going to be the ones having to use this all the time.
I'll keep KVM/Xen people updated on the user level interface and let them comment based on a real usage model of the Intel 82576 NIC after PF/VF drivers are available.
Regards,
Yu
Quoted 10 lines · Yu Zhao
Greetings,
Following patches are intended to support SR-IOV capability in the Linux kernel. With these patches, people can turn a PCI device with the capability into multiple ones from software perspective, which will benefit KVM and achieve other purposes such as QoS, security, and etc.
The Physical Function and Virtual Function drivers using the SR-IOV APIs will come soon!
Thanks for respining these patches, but I think we really need to see a driver using this in order to get an idea of how it will be used.
Also, the Xen and KVM people need to agree on the userspace interface here, perhaps also getting some libvirt involvement as well, as they are going to be the ones having to use this all the time.
thanks,
greg k-h
cover introduction
In reply to Yu ZhaoSR-IOV drivers of Intel 82576 NIC are available. There are two parts of the drivers: Physical Function driver and Virtual Function driver. The PF driver is based on the IGB driver and is used to control PF to allocate hardware specific resources and interface with the SR-IOV core. The VF driver is a new NIC driver that is same as the traditional PCI device driver. It works in both the host and the guest (Xen and KVM) environment.
These two drivers are testing versions and they are *only* intended to show how to use SR-IOV API.
Intel 82576 NIC specification can be found at:
http://download.intel.com/design/network/datashts/82576_Datasheet_v2p1.pdf
[SR-IOV driver example 1/3] PF driver: allocate hardware specific resource [SR-IOV driver example 2/3] PF driver: integrate with SR-IOV core [SR-IOV driver example 3/3] VF driver tar ball
PATCH 1/3 PF driver: allocate hardware specific resource
This patch makes the IGB driver allocate hardware resource (rx/tx queues) for Virtual Functions. All operations in this patch are hardware specific.
--- drivers/net/igb/Makefile | 2 +- drivers/net/igb/e1000_82575.c | 1 + drivers/net/igb/e1000_82575.h | 61 ++++ drivers/net/igb/e1000_defines.h | 7 + drivers/net/igb/e1000_hw.h | 2 + drivers/net/igb/e1000_regs.h | 13 + drivers/net/igb/e1000_vf.c | 223 ++++++++++++++ drivers/net/igb/igb.h | 10 + drivers/net/igb/igb_main.c | 604 ++++++++++++++++++++++++++++++++++++++- 9 files changed, 910 insertions(+), 13 deletions(-) create mode 100644 drivers/net/igb/e1000_vf.c
drivers/net/igb/Makefile +1 -1
@@ -33,5 +33,5 @@ obj-$(CONFIG_IGB) += igb.o igb-objs := igb_main.o igb_ethtool.o e1000_82575.o \ - e1000_mac.o e1000_nvm.o e1000_phy.o + e1000_mac.o e1000_nvm.o e1000_phy.o e1000_vf.o
drivers/net/igb/e1000_82575.c +1 -0
@@ -87,6 +87,7 @@ static s32 igb_get_invariants_82575(struct e1000_hw *hw)case E1000_DEV_ID_82576 :case E1000_DEV_ID_82576_FIBER :case E1000_DEV_ID_82576_SERDES : +case E1000_DEV_ID_82576_QUAD_COPPER : mac->type = e1000_82576;break ;default :
drivers/net/igb/e1000_82575.h +61 -0
65 lines not on this page · raw message
drivers/net/igb/e1000_defines.h +7 -0
@@ -389,6 +389,7 @@#define E1000_ICR_RXDMT0 0x00000010/* rx desc min. threshold (0) */ #define E1000_ICR_RXO 0x00000040/* rx overrun */ #define E1000_ICR_RXT0 0x00000080/* rx timer intr (ring 0) */ +#define E1000_ICR_VMMB 0x00000100/* VM MB event */ #define E1000_ICR_MDAC 0x00000200/* MDIO access complete */ #define E1000_ICR_RXCFG 0x00000400/* Rx /c/ ordered set */ #define E1000_ICR_GPI_EN0 0x00000800/* GP Int 0 */
@@ -451,6 +452,7 @@/* Interrupt Mask Set */ #define E1000_IMS_TXDW E1000_ICR_TXDW/* Transmit desc written back */ #define E1000_IMS_LSC E1000_ICR_LSC/* Link Status Change */ +#define E1000_IMS_VMMB E1000_ICR_VMMB/* Mail box activity */ #define E1000_IMS_RXSEQ E1000_ICR_RXSEQ/* rx sequence error */ #define E1000_IMS_RXDMT0 E1000_ICR_RXDMT0/* rx desc min. threshold */ #define E1000_IMS_RXT0 E1000_ICR_RXT0/* rx timer intr */
@@ -768,4 +770,9 @@#define E1000_GEN_CTL_ADDRESS_SHIFT 8#define E1000_GEN_POLL_TIMEOUT 640 +#define E1000_WRITE_FLUSH (a) (readl((a)->hw_addr + E1000_STATUS)) +#define E1000_MRQC_ENABLE_MASK 0x00000007 +#define E1000_MRQC_ENABLE_VMDQ 0x00000003 +#define E1000_CTRL_EXT_PFRSTD 0x00004000 +#endif
drivers/net/igb/e1000_hw.h +2 -0
@@ -41,6 +41,7 @@ struct e1000_hw;#define E1000_DEV_ID_82576 0x10C9#define E1000_DEV_ID_82576_FIBER 0x10E6#define E1000_DEV_ID_82576_SERDES 0x10E7 +#define E1000_DEV_ID_82576_QUAD_COPPER 0x10E8#define E1000_DEV_ID_82575EB_COPPER 0x10A7#define E1000_DEV_ID_82575EB_FIBER_SERDES 0x10A9#define E1000_DEV_ID_82575GB_QUAD_COPPER 0x10D6
@@ -91,6 +92,7 @@ enum e1000_phy_type { e1000_phy_gg82563, e1000_phy_igp_3, e1000_phy_ife, + e1000_phy_vf, }; enum e1000_bus_type {
drivers/net/igb/e1000_regs.h +13 -0
@@ -262,6 +262,19 @@#define E1000_RETA (_i) (0x05C00 + ((_i) * 4))#define E1000_RSSRK (_i) (0x05C80 + ((_i) * 4))/* RSS Random Key - RW Array */ +/* VT Registers */ +#define E1000_MBVFICR 0x00C80/* Mailbox VF Cause - RWC */ +#define E1000_MBVFIMR 0x00C84/* Mailbox VF int Mask - RW */ +#define E1000_VFLRE 0x00C88/* VF Register Events - RWC */ +#define E1000_VFRE 0x00C8C/* VF Receive Enables */ +#define E1000_VFTE 0x00C90/* VF Transmit Enables */ +#define E1000_DTXSWC 0x03500/* DMA Tx Switch Control - RW */ +/* These act per VF so an array friendly macro is used */ +#define E1000_P2VMAILBOX (_n) (0x00C00 + (4 * (_n))) +#define E1000_VMBMEM (_n) (0x00800 + (64 * (_n))) +#define E1000_VMOLR (_n) (0x05AD0 + (4 * (_n))) +#define E1000_VLVF (_n) (0x05D00 + (4 * (_n)))/* VLAN Virtual Machine */ +#define wr32 (reg, value) (writel(value, hw->hw_addr + reg))#define rd32 (reg) (readl(hw->hw_addr + reg))#define wrfl () ((void)rd32(E1000_STATUS))
drivers/net/igb/e1000_vf.c new file +223 -0
223 lines not on this page · raw message
drivers/net/igb/igb.h +10 -0
@@ -294,6 +294,16 @@ struct igb_adapter {unsigned int lro_flushed;unsigned int lro_no_desc;#endif +#ifdef CONFIG_PCI_IOV +unsigned int vfs_allocated_count; +struct work_struct msg_task; +u32 vf_icr; +u32 vflre; +unsigned char vf_mac_addresses[8 ][6 ]; +u8 vfta_tracking_entry[128 ]; +int int0counter; +int int1counter; +#endif };#define IGB_FLAG_HAS_MSI (1 << 0)
drivers/net/igb/igb_main.c +592 -12
793 lines not on this page · raw message
Signature · 1 line
1.5.4.4
PATCH 1/3 PF driver: allocate hardware specific resource
In reply to Yu ZhaoThis patch makes the IGB driver allocate hardware resource (rx/tx queues) for Virtual Functions. All operations in this patch are hardware specific.
--- drivers/net/igb/Makefile | 2 +- drivers/net/igb/e1000_82575.c | 1 + drivers/net/igb/e1000_82575.h | 61 ++++ drivers/net/igb/e1000_defines.h | 7 + drivers/net/igb/e1000_hw.h | 2 + drivers/net/igb/e1000_regs.h | 13 + drivers/net/igb/e1000_vf.c | 223 ++++++++++++++ drivers/net/igb/igb.h | 10 + drivers/net/igb/igb_main.c | 604 ++++++++++++++++++++++++++++++++++++++- 9 files changed, 910 insertions(+), 13 deletions(-) create mode 100644 drivers/net/igb/e1000_vf.c
drivers/net/igb/Makefile +1 -1
@@ -33,5 +33,5 @@ obj-$(CONFIG_IGB) += igb.o igb-objs := igb_main.o igb_ethtool.o e1000_82575.o \ - e1000_mac.o e1000_nvm.o e1000_phy.o + e1000_mac.o e1000_nvm.o e1000_phy.o e1000_vf.o
drivers/net/igb/e1000_82575.c +1 -0
@@ -87,6 +87,7 @@ static s32 igb_get_invariants_82575(struct e1000_hw *hw)case E1000_DEV_ID_82576 :case E1000_DEV_ID_82576_FIBER :case E1000_DEV_ID_82576_SERDES : +case E1000_DEV_ID_82576_QUAD_COPPER : mac->type = e1000_82576;break ;default :
drivers/net/igb/e1000_82575.h +61 -0
65 lines not on this page · raw message
drivers/net/igb/e1000_defines.h +7 -0
@@ -389,6 +389,7 @@#define E1000_ICR_RXDMT0 0x00000010/* rx desc min. threshold (0) */ #define E1000_ICR_RXO 0x00000040/* rx overrun */ #define E1000_ICR_RXT0 0x00000080/* rx timer intr (ring 0) */ +#define E1000_ICR_VMMB 0x00000100/* VM MB event */ #define E1000_ICR_MDAC 0x00000200/* MDIO access complete */ #define E1000_ICR_RXCFG 0x00000400/* Rx /c/ ordered set */ #define E1000_ICR_GPI_EN0 0x00000800/* GP Int 0 */
@@ -451,6 +452,7 @@/* Interrupt Mask Set */ #define E1000_IMS_TXDW E1000_ICR_TXDW/* Transmit desc written back */ #define E1000_IMS_LSC E1000_ICR_LSC/* Link Status Change */ +#define E1000_IMS_VMMB E1000_ICR_VMMB/* Mail box activity */ #define E1000_IMS_RXSEQ E1000_ICR_RXSEQ/* rx sequence error */ #define E1000_IMS_RXDMT0 E1000_ICR_RXDMT0/* rx desc min. threshold */ #define E1000_IMS_RXT0 E1000_ICR_RXT0/* rx timer intr */
@@ -768,4 +770,9 @@#define E1000_GEN_CTL_ADDRESS_SHIFT 8#define E1000_GEN_POLL_TIMEOUT 640 +#define E1000_WRITE_FLUSH (a) (readl((a)->hw_addr + E1000_STATUS)) +#define E1000_MRQC_ENABLE_MASK 0x00000007 +#define E1000_MRQC_ENABLE_VMDQ 0x00000003 +#define E1000_CTRL_EXT_PFRSTD 0x00004000 +#endif
drivers/net/igb/e1000_hw.h +2 -0
@@ -41,6 +41,7 @@ struct e1000_hw;#define E1000_DEV_ID_82576 0x10C9#define E1000_DEV_ID_82576_FIBER 0x10E6#define E1000_DEV_ID_82576_SERDES 0x10E7 +#define E1000_DEV_ID_82576_QUAD_COPPER 0x10E8#define E1000_DEV_ID_82575EB_COPPER 0x10A7#define E1000_DEV_ID_82575EB_FIBER_SERDES 0x10A9#define E1000_DEV_ID_82575GB_QUAD_COPPER 0x10D6
@@ -91,6 +92,7 @@ enum e1000_phy_type { e1000_phy_gg82563, e1000_phy_igp_3, e1000_phy_ife, + e1000_phy_vf, }; enum e1000_bus_type {
drivers/net/igb/e1000_regs.h +13 -0
@@ -262,6 +262,19 @@#define E1000_RETA (_i) (0x05C00 + ((_i) * 4))#define E1000_RSSRK (_i) (0x05C80 + ((_i) * 4))/* RSS Random Key - RW Array */ +/* VT Registers */ +#define E1000_MBVFICR 0x00C80/* Mailbox VF Cause - RWC */ +#define E1000_MBVFIMR 0x00C84/* Mailbox VF int Mask - RW */ +#define E1000_VFLRE 0x00C88/* VF Register Events - RWC */ +#define E1000_VFRE 0x00C8C/* VF Receive Enables */ +#define E1000_VFTE 0x00C90/* VF Transmit Enables */ +#define E1000_DTXSWC 0x03500/* DMA Tx Switch Control - RW */ +/* These act per VF so an array friendly macro is used */ +#define E1000_P2VMAILBOX (_n) (0x00C00 + (4 * (_n))) +#define E1000_VMBMEM (_n) (0x00800 + (64 * (_n))) +#define E1000_VMOLR (_n) (0x05AD0 + (4 * (_n))) +#define E1000_VLVF (_n) (0x05D00 + (4 * (_n)))/* VLAN Virtual Machine */ +#define wr32 (reg, value) (writel(value, hw->hw_addr + reg))#define rd32 (reg) (readl(hw->hw_addr + reg))#define wrfl () ((void)rd32(E1000_STATUS))
drivers/net/igb/e1000_vf.c new file +223 -0
223 lines not on this page · raw message
drivers/net/igb/igb.h +10 -0
@@ -294,6 +294,16 @@ struct igb_adapter {unsigned int lro_flushed;unsigned int lro_no_desc;#endif +#ifdef CONFIG_PCI_IOV +unsigned int vfs_allocated_count; +struct work_struct msg_task; +u32 vf_icr; +u32 vflre; +unsigned char vf_mac_addresses[8 ][6 ]; +u8 vfta_tracking_entry[128 ]; +int int0counter; +int int1counter; +#endif };#define IGB_FLAG_HAS_MSI (1 << 0)
drivers/net/igb/igb_main.c +592 -12
793 lines not on this page · raw message
Signature · 1 line
1.5.4.4
PATCH 2/3 PF driver: integrate with SR-IOV core
In reply to Yu ZhaoThis patch integrates the IGB driver with the SR-IOV core. It shows how the SR-IOV API is used to support the capability. Obviously people does not need to put much effort to integrate the PF driver with SR-IOV core. All SR-IOV standard stuff are handled by SR-IOV core and PF driver only concerns the device specific resource allocation and deallocation once it gets the necessary information (i.e. number of Virtual Functions) from the callback function.
--- drivers/net/igb/igb_main.c | 30 ++++++++++++++++++++++++++++++ 1 files changed, 30 insertions(+), 0 deletions(-)
drivers/net/igb/igb_main.c +30 -0
@@ -139,6 +139,7 @@ void igb_set_mc_list_pools(struct igb_adapter *, struct e1000_hw *, int, u16);static int igb_vmm_control (struct igb_adapter *,bool );static int igb_set_vf_mac (struct net_device *,int ,u8 *);static void igb_mbox_handler (struct igb_adapter *); +static int igb_virtual (struct pci_dev *,int );#endif static int igb_suspend (struct pci_dev *,pm_message_t );
@@ -184,6 +185,9 @@ static struct pci_driver igb_driver = {#endif .shutdown = igb_shutdown, .err_handler = &igb_err_handler, +#ifdef CONFIG_PCI_IOV + .virtual = igb_virtual +#endif };static int global_quad_port_a;/* global quad port a indication */
@@ -5107,6 +5111,32 @@ void igb_set_mc_list_pools(struct igb_adapter *adapter, reg_data |= (1 <<25 );wr32 (E1000_VMOLR (pool), reg_data); } + +static int +igb_virtual (struct pci_dev *pdev,int nr_virtfn) +{ +unsigned char my_mac_addr[6 ] = {0x00 ,0xDE ,0xAD ,0xBE ,0xEF ,0xFF }; +struct net_device *netdev =pci_get_drvdata (pdev); +struct igb_adapter *adapter =netdev_priv (netdev); +int i; + +if (nr_virtfn >7 ) +return -EINVAL ; + +if (nr_virtfn) { +for (i =0 ; i < nr_virtfn; i++) { +printk (KERN_INFO "SR-IOV: VF %d is enabled\n" , i); + my_mac_addr[5 ] = (unsigned char )i; +igb_set_vf_mac (netdev, i, my_mac_addr); +igb_set_vf_vmolr (adapter, i); + } + }else +printk (KERN_INFO "SR-IOV is disabled\n" ); + + adapter->vfs_allocated_count = nr_virtfn; + +return 0 ; +}#endif /* igb_main.c */
Signature · 1 line
1.5.4.4
Re: [SR-IOV driver example 2/3] PF driver: integrate with SR-IOV core
Quoted 25 lines · Yu Zhao
This patch integrates the IGB driver with the SR-IOV core. It shows how the SR-IOV API is used to support the capability. Obviously people does not need to put much effort to integrate the PF driver with SR-IOV core. All SR-IOV standard stuff are handled by SR-IOV core and PF driver only concerns the device specific resource allocation and deallocation once it gets the necessary information (i.e. number of Virtual Functions) from the callback function.
--- drivers/net/igb/igb_main.c | 30 ++++++++++++++++++++++++++++++ 1 files changed, 30 insertions(+), 0 deletions(-)drivers/net/igb/igb_main.c +1
-0@@ -139,6 +139,7 @@ void igb_set_mc_list_pools(struct igb_adapter *, struct e1000_hw *, int, u16);static int igb_vmm_control (struct igb_adapter *,bool );static int igb_set_vf_mac (struct net_device *,int ,u8 *);static void igb_mbox_handler (struct igb_adapter *); +static int igb_virtual (struct pci_dev *,int );#endif static int igb_suspend (struct pci_dev *,pm_message_t );
drivers/net/igb/igb_main.c static struct pci_driver igb_driver = { · line 190
Quoted 6 lines
drivers/net/igb/igb_main.c +3
-0@@ -184,6 +185,9 @@ static struct pci_driver igb_driver = {#endif .shutdown = igb_shutdown, .err_handler = &igb_err_handler, +#ifdef CONFIG_PCI_IOV + .virtual = igb_virtual +#endif
#ifdef should not be needed, right?
Quoted 8 lines
drivers/net/igb/igb_main.c +1
-0};static int global_quad_port_a;/* global quad port a indication */ @@ -5107,6 +5111,32 @@ void igb_set_mc_list_pools(struct igb_adapter *adapter, reg_data |= (1 <<25 );wr32 (E1000_VMOLR (pool), reg_data); } +
drivers/net/igb/igb_main.c void igb_set_mc_list_pools(struct igb_adapter *adapter, · line 5124
Quoted 10 lines
drivers/net/igb/igb_main.c +10
-0+static int +igb_virtual (struct pci_dev *pdev,int nr_virtfn) +{ +unsigned char my_mac_addr[6 ] = {0x00 ,0xDE ,0xAD ,0xBE ,0xEF ,0xFF }; +struct net_device *netdev =pci_get_drvdata (pdev); +struct igb_adapter *adapter =netdev_priv (netdev); +int i; + +if (nr_virtfn >7 ) +return -EINVAL ;
Why the check for 7? Is that the max virtual functions for this card? Shouldn't that be a define somewhere so it's easier to fix in future versions of this hardware? :)
drivers/net/igb/igb_main.c void igb_set_mc_list_pools(struct igb_adapter *adapter, · line 5128
Quoted 4 lines
drivers/net/igb/igb_main.c +4
-0+ +if (nr_virtfn) { +for (i =0 ; i < nr_virtfn; i++) { +printk (KERN_INFO "SR-IOV: VF %d is enabled\n" , i);
Use dev_info() please, that shows the exact pci device and driver that emitted the message.
drivers/net/igb/igb_main.c void igb_set_mc_list_pools(struct igb_adapter *adapter, · line 5134
Quoted 6 lines
drivers/net/igb/igb_main.c +6
-0+ my_mac_addr[5 ] = (unsigned char )i; +igb_set_vf_mac (netdev, i, my_mac_addr); +igb_set_vf_vmolr (adapter, i); + } + }else +printk (KERN_INFO "SR-IOV is disabled\n" );
Is that really true? (oh, use dev_info as well.) What happens if you had called this with "5" and then later with "0", you never destroyed those existing virtual functions, yet the code does:
drivers/net/igb/igb_main.c void igb_set_mc_list_pools(struct igb_adapter *adapter, · line 5136
Quoted 1 line
drivers/net/igb/igb_main.c +1
-0+ adapter->vfs_allocated_count = nr_virtfn;
Which makes the driver think they are not present. What happens when the driver later goes to shut down? Are those resources freed up properly?
thanks,
greg k-h
Re: [SR-IOV driver example 2/3] PF driver: integrate with SR-IOV core
Quoted 6 lines · * Greg KH ([email protected]) wrote:
Quoted 6 lines
drivers/net/igb/igb_main.c +6
-0+static int +igb_virtual (struct pci_dev *pdev,int nr_virtfn) +{ +unsigned char my_mac_addr[6 ] = {0x00 ,0xDE ,0xAD ,0xBE ,0xEF ,0xFF }; +struct net_device *netdev =pci_get_drvdata (pdev); +struct igb_adapter *adapter =netdev_priv (netdev);
Quoted 8 lines
drivers/net/igb/igb_main.c void igb_set_mc_list_pools(struct igb_adapter *adapter, · line 5124
Quoted 4 lines
drivers/net/igb/igb_main.c +4
-0+int i; + +if (nr_virtfn >7 ) +return -EINVAL ;Why the check for 7? Is that the max virtual functions for this card? Shouldn't that be a define somewhere so it's easier to fix in future versions of this hardware? :)
IIRC it's 8 for the card, 1 reserved for PF. I think both notions should be captured w/ commented constants.
thanks,
-chris
Re: [SR-IOV driver example 2/3] PF driver: integrate with SR-IOV core
Quoted 16 lines · Chris Wright
Quoted 6 lines · * Greg KH ([email protected]) wrote:
Quoted 6 lines
drivers/net/igb/igb_main.c +6
-0+static int +igb_virtual (struct pci_dev *pdev,int nr_virtfn) +{ +unsigned char my_mac_addr[6 ] = {0x00 ,0xDE ,0xAD ,0xBE ,0xEF ,0xFF }; +struct net_device *netdev =pci_get_drvdata (pdev); +struct igb_adapter *adapter =netdev_priv (netdev);Quoted 8 lines
drivers/net/igb/igb_main.c void igb_set_mc_list_pools(struct igb_adapter *adapter, · line 5124
Quoted 4 lines
drivers/net/igb/igb_main.c +4
-0+int i; + +if (nr_virtfn >7 ) +return -EINVAL ;Why the check for 7? Is that the max virtual functions for this card? Shouldn't that be a define somewhere so it's easier to fix in future versions of this hardware? :)
Quoted 2 lines
IIRC it's 8 for the card, 1 reserved for PF. I think both notions should be captured w/ commented constants.
You remember correctly! I'll put some comments there as suggested.
Thanks,
Yu
Re: [SR-IOV driver example 2/3] PF driver: integrate with SR-IOV core
In reply to Chris WrightQuoted 16 lines · Chris Wright
Quoted 6 lines · * Greg KH ([email protected]) wrote:
Quoted 6 lines
drivers/net/igb/igb_main.c +6
-0+static int +igb_virtual (struct pci_dev *pdev,int nr_virtfn) +{ +unsigned char my_mac_addr[6 ] = {0x00 ,0xDE ,0xAD ,0xBE ,0xEF ,0xFF }; +struct net_device *netdev =pci_get_drvdata (pdev); +struct igb_adapter *adapter =netdev_priv (netdev);Quoted 8 lines
drivers/net/igb/igb_main.c void igb_set_mc_list_pools(struct igb_adapter *adapter, · line 5124
Quoted 4 lines
drivers/net/igb/igb_main.c +4
-0+int i; + +if (nr_virtfn >7 ) +return -EINVAL ;Why the check for 7? Is that the max virtual functions for this card? Shouldn't that be a define somewhere so it's easier to fix in future versions of this hardware? :)
Quoted 2 lines
IIRC it's 8 for the card, 1 reserved for PF. I think both notions should be captured w/ commented constants.
You remember correctly! I'll put some comments there as suggested.
Thanks,
Yu
Re: [SR-IOV driver example 2/3] PF driver: integrate with SR-IOV core
In reply to Greg KHQuoted 6 lines · * Greg KH ([email protected]) wrote:
Quoted 6 lines
drivers/net/igb/igb_main.c +6
-0+static int +igb_virtual (struct pci_dev *pdev,int nr_virtfn) +{ +unsigned char my_mac_addr[6 ] = {0x00 ,0xDE ,0xAD ,0xBE ,0xEF ,0xFF }; +struct net_device *netdev =pci_get_drvdata (pdev); +struct igb_adapter *adapter =netdev_priv (netdev);
Quoted 8 lines
drivers/net/igb/igb_main.c void igb_set_mc_list_pools(struct igb_adapter *adapter, · line 5124
Quoted 4 lines
drivers/net/igb/igb_main.c +4
-0+int i; + +if (nr_virtfn >7 ) +return -EINVAL ;Why the check for 7? Is that the max virtual functions for this card? Shouldn't that be a define somewhere so it's easier to fix in future versions of this hardware? :)
IIRC it's 8 for the card, 1 reserved for PF. I think both notions should be captured w/ commented constants.
thanks,
-chris
RE: [SR-IOV driver example 2/3] PF driver: integrate with SR-IOV core
In reply to Greg KHQuoted 24 lines · Greg KH
Quoted 23 lines · Yu Zhao
This patch integrates the IGB driver with the SR-IOV core. It shows how the SR-IOV API is used to support the capability. Obviously people does not need to put much effort to integrate the PF driver with SR-IOV core. All SR-IOV standard stuff are handled by SR-IOV core and PF driver once it gets the necessary information (i.e. number of Virtual
Functions) from the callback function.--- drivers/net/igb/igb_main.c | 30 ++++++++++++++++++++++++++++++ 1 files changed, 30 insertions(+), 0 deletions(-)drivers/net/igb/igb_main.c +0
-0@@ -139,6 +139,7 @@ void igb_set_mc_list_pools(struct igb_adapter *,struct
e1000_hw*, int, u16); static intigb_vmm_control(struct igb_adapter *, bool); static intigb_set_vf_mac(struct net_device *, int, u8*); static voidigb_mbox_handler(struct igb_adapter *); +static intigb_virtual(struct pci_dev *, int);
#endif
Quoted 10 lines
drivers/net/igb/igb_main.c static struct pci_driver igb_driver = { · line 190
Quoted 7 lines
static int
igb_suspend(struct pci_dev *, pm_message_t); @@ -184,6
+185,9 @@ static structpci_driverigb_driver= { #endif
.shutdown =igb_shutdown,
.err_handler =&igb_err_handler,drivers/net/igb/igb_main.c +3
-0+#ifdef CONFIG_PCI_IOV + .virtual = igb_virtual +#endif #ifdef should not be needed, right?
Good point. I think this is because the driver is expected to build on older kernels also, but the problem is that the driver (and probably others) is broken unless the kernel is built with CONFIG_PCI_IOV because of the following hunk, for example.
However, we don't want to use #ifdef for the (*virtual) field in the header. One option would be to define a constant like the following along with those changes. #define PCI_DEV_IOV
Any better idea?
Thanks,
.
Jun Nakajima | Intel Open Source Technology Center
----
@@ -259,6 +266,7 @@ struct pci_dev {
struct list_head msi_list;
#endif
struct pci_vpd *vpd;+ struct pci_iov *iov;
};
extern struct pci_dev *alloc_pci_dev(void); @@ -426,6 +434,7 @@ struct pci_driver {
int (*resume_early) (struct pci_dev *dev);
int (*resume) (struct pci_dev *dev); /* Device woken up */
void (*shutdown) (struct pci_dev *dev);
+ int (*virtual) (struct pci_dev *dev, int nr_virtfn);
struct pm_ext_ops *pm;
struct pci_error_handlers *err_handler;
struct device_driver driver;Re: [SR-IOV driver example 2/3] PF driver: integrate with SR-IOV core
Quoted 36 lines · Jun
Quoted 24 lines · Greg KH
Quoted 23 lines · Yu Zhao
This patch integrates the IGB driver with the SR-IOV core. It shows how the SR-IOV API is used to support the capability. Obviously people does not need to put much effort to integrate the PF driver with SR-IOV core. All SR-IOV standard stuff are handled by SR-IOV core and PF driver once it gets the necessary information (i.e. number of Virtual
Functions) from the callback function.--- drivers/net/igb/igb_main.c | 30 ++++++++++++++++++++++++++++++ 1 files changed, 30 insertions(+), 0 deletions(-)drivers/net/igb/igb_main.c +0
-0@@ -139,6 +139,7 @@ void igb_set_mc_list_pools(struct igb_adapter *,struct
e1000_hw*, int, u16); static intigb_vmm_control(struct igb_adapter *, bool); static intigb_set_vf_mac(struct net_device *, int, u8*); static voidigb_mbox_handler(struct igb_adapter *); +static intigb_virtual(struct pci_dev *, int);
#endifQuoted 10 lines
drivers/net/igb/igb_main.c static struct pci_driver igb_driver = { · line 190
Quoted 7 lines
static int
igb_suspend(struct pci_dev *, pm_message_t); @@ -184,6
+185,9 @@ static structpci_driverigb_driver= { #endif
.shutdown =igb_shutdown,
.err_handler =&igb_err_handler,drivers/net/igb/igb_main.c +3
-0+#ifdef CONFIG_PCI_IOV + .virtual = igb_virtual +#endif #ifdef should not be needed, right?
Quoted 2 lines
Good point. I think this is because the driver is expected to build on older kernels also,
That should not be an issue for patches that are being submitted, right?
And if this is the case, shouldn't it be called out in the changelog entry?
Quoted 10 lines
but the problem is that the driver (and probably others) is broken unless the kernel is built with
CONFIG_PCI_IOVbecause of the following hunk, for example.However, we don't want to use #ifdef for the (*virtual) field in the header. One option would be to define a constant like the following along with those changes.
#definePCI_DEV_IOVAny better idea?
Just always declare it in your driver, which will be added _after_ this field gets added to the kernel tree as well. It's not a big deal, just an ordering of patches issue.
Because remember, don't add #ifdefs to drivers, they should not be needed at all.
thanks,
greg k-h
Re: [SR-IOV driver example 2/3] PF driver: integrate with SR-IOV core
In reply to Nakajima, JunQuoted 36 lines · Jun
Quoted 24 lines · Greg KH
Quoted 23 lines · Yu Zhao
This patch integrates the IGB driver with the SR-IOV core. It shows how the SR-IOV API is used to support the capability. Obviously people does not need to put much effort to integrate the PF driver with SR-IOV core. All SR-IOV standard stuff are handled by SR-IOV core and PF driver once it gets the necessary information (i.e. number of Virtual
Functions) from the callback function.--- drivers/net/igb/igb_main.c | 30 ++++++++++++++++++++++++++++++ 1 files changed, 30 insertions(+), 0 deletions(-)drivers/net/igb/igb_main.c +0
-0@@ -139,6 +139,7 @@ void igb_set_mc_list_pools(struct igb_adapter *,struct
e1000_hw*, int, u16); static intigb_vmm_control(struct igb_adapter *, bool); static intigb_set_vf_mac(struct net_device *, int, u8*); static voidigb_mbox_handler(struct igb_adapter *); +static intigb_virtual(struct pci_dev *, int);
#endifQuoted 10 lines
drivers/net/igb/igb_main.c static struct pci_driver igb_driver = { · line 190
Quoted 7 lines
static int
igb_suspend(struct pci_dev *, pm_message_t); @@ -184,6
+185,9 @@ static structpci_driverigb_driver= { #endif
.shutdown =igb_shutdown,
.err_handler =&igb_err_handler,drivers/net/igb/igb_main.c +3
-0+#ifdef CONFIG_PCI_IOV + .virtual = igb_virtual +#endif #ifdef should not be needed, right?
Quoted 2 lines
Good point. I think this is because the driver is expected to build on older kernels also,
That should not be an issue for patches that are being submitted, right?
And if this is the case, shouldn't it be called out in the changelog entry?
Quoted 10 lines
but the problem is that the driver (and probably others) is broken unless the kernel is built with
CONFIG_PCI_IOVbecause of the following hunk, for example.However, we don't want to use #ifdef for the (*virtual) field in the header. One option would be to define a constant like the following along with those changes.
#definePCI_DEV_IOVAny better idea?
Just always declare it in your driver, which will be added _after_ this field gets added to the kernel tree as well. It's not a big deal, just an ordering of patches issue.
Because remember, don't add #ifdefs to drivers, they should not be needed at all.
thanks,
greg k-h
RE: [SR-IOV driver example 2/3] PF driver: integrate with SR-IOV core
In reply to Greg KHQuoted 24 lines · Greg KH
Quoted 23 lines · Yu Zhao
This patch integrates the IGB driver with the SR-IOV core. It shows how the SR-IOV API is used to support the capability. Obviously people does not need to put much effort to integrate the PF driver with SR-IOV core. All SR-IOV standard stuff are handled by SR-IOV core and PF driver once it gets the necessary information (i.e. number of Virtual
Functions) from the callback function.--- drivers/net/igb/igb_main.c | 30 ++++++++++++++++++++++++++++++ 1 files changed, 30 insertions(+), 0 deletions(-)drivers/net/igb/igb_main.c +0
-0@@ -139,6 +139,7 @@ void igb_set_mc_list_pools(struct igb_adapter *,struct
e1000_hw*, int, u16); static intigb_vmm_control(struct igb_adapter *, bool); static intigb_set_vf_mac(struct net_device *, int, u8*); static voidigb_mbox_handler(struct igb_adapter *); +static intigb_virtual(struct pci_dev *, int);
#endif
Quoted 10 lines
drivers/net/igb/igb_main.c static struct pci_driver igb_driver = { · line 190
Quoted 7 lines
static int
igb_suspend(struct pci_dev *, pm_message_t); @@ -184,6
+185,9 @@ static structpci_driverigb_driver= { #endif
.shutdown =igb_shutdown,
.err_handler =&igb_err_handler,drivers/net/igb/igb_main.c +3
-0+#ifdef CONFIG_PCI_IOV + .virtual = igb_virtual +#endif #ifdef should not be needed, right?
Good point. I think this is because the driver is expected to build on older kernels also, but the problem is that the driver (and probably others) is broken unless the kernel is built with CONFIG_PCI_IOV because of the following hunk, for example.
However, we don't want to use #ifdef for the (*virtual) field in the header. One option would be to define a constant like the following along with those changes. #define PCI_DEV_IOV
Any better idea?
Thanks,
.
Jun Nakajima | Intel Open Source Technology Center
----
@@ -259,6 +266,7 @@ struct pci_dev {
struct list_head msi_list;
#endif
struct pci_vpd *vpd;+ struct pci_iov *iov;
};
extern struct pci_dev *alloc_pci_dev(void); @@ -426,6 +434,7 @@ struct pci_driver {
int (*resume_early) (struct pci_dev *dev);
int (*resume) (struct pci_dev *dev); /* Device woken up */
void (*shutdown) (struct pci_dev *dev);
+ int (*virtual) (struct pci_dev *dev, int nr_virtfn);
struct pm_ext_ops *pm;
struct pci_error_handlers *err_handler;
struct device_driver driver;Re: [SR-IOV driver example 2/3] PF driver: integrate with SR-IOV core
In reply to Greg KHQuoted 12 lines · Greg KH
drivers/net/igb/igb_main.c void igb_set_mc_list_pools(struct igb_adapter *adapter, · line 5134
Quoted 6 lines · Yu Zhao
drivers/net/igb/igb_main.c +6
-0+ my_mac_addr[5 ] = (unsigned char )i; +igb_set_vf_mac (netdev, i, my_mac_addr); +igb_set_vf_vmolr (adapter, i); + } + }else +printk (KERN_INFO "SR-IOV is disabled\n" );Is that really true? (oh, use
dev_infoas well.) What happens if you had called this with "5" and then later with "0", you never destroyed those existing virtual functions, yet the code does:
Quoted 5 lines
drivers/net/igb/igb_main.c void igb_set_mc_list_pools(struct igb_adapter *adapter, · line 5136
Quoted 1 line
drivers/net/igb/igb_main.c +1
-0+ adapter->vfs_allocated_count = nr_virtfn;Which makes the driver think they are not present. What happens when the driver later goes to shut down? Are those resources freed up properly?
For now we hard-code the tx/rx queues allocation so this doesn't matter. Eventually this will become dynamic allocation: when number of VFs changes the corresponding resources need to be freed.
I'll put more comments here.
Thanks,
Yu
Re: [SR-IOV driver example 2/3] PF driver: integrate with SR-IOV core
In reply to Greg KHQuoted 12 lines · Greg KH
drivers/net/igb/igb_main.c void igb_set_mc_list_pools(struct igb_adapter *adapter, · line 5134
Quoted 6 lines · Yu Zhao
drivers/net/igb/igb_main.c +6
-0+ my_mac_addr[5 ] = (unsigned char )i; +igb_set_vf_mac (netdev, i, my_mac_addr); +igb_set_vf_vmolr (adapter, i); + } + }else +printk (KERN_INFO "SR-IOV is disabled\n" );Is that really true? (oh, use
dev_infoas well.) What happens if you had called this with "5" and then later with "0", you never destroyed those existing virtual functions, yet the code does:
Quoted 5 lines
drivers/net/igb/igb_main.c void igb_set_mc_list_pools(struct igb_adapter *adapter, · line 5136
Quoted 1 line
drivers/net/igb/igb_main.c +1
-0+ adapter->vfs_allocated_count = nr_virtfn;Which makes the driver think they are not present. What happens when the driver later goes to shut down? Are those resources freed up properly?
For now we hard-code the tx/rx queues allocation so this doesn't matter. Eventually this will become dynamic allocation: when number of VFs changes the corresponding resources need to be freed.
I'll put more comments here.
Thanks,
Yu
Re: [SR-IOV driver example 2/3] PF driver: integrate with SR-IOV core
In reply to Yu ZhaoQuoted 25 lines · Yu Zhao
This patch integrates the IGB driver with the SR-IOV core. It shows how the SR-IOV API is used to support the capability. Obviously people does not need to put much effort to integrate the PF driver with SR-IOV core. All SR-IOV standard stuff are handled by SR-IOV core and PF driver only concerns the device specific resource allocation and deallocation once it gets the necessary information (i.e. number of Virtual Functions) from the callback function.
--- drivers/net/igb/igb_main.c | 30 ++++++++++++++++++++++++++++++ 1 files changed, 30 insertions(+), 0 deletions(-)drivers/net/igb/igb_main.c +1
-0@@ -139,6 +139,7 @@ void igb_set_mc_list_pools(struct igb_adapter *, struct e1000_hw *, int, u16);static int igb_vmm_control (struct igb_adapter *,bool );static int igb_set_vf_mac (struct net_device *,int ,u8 *);static void igb_mbox_handler (struct igb_adapter *); +static int igb_virtual (struct pci_dev *,int );#endif static int igb_suspend (struct pci_dev *,pm_message_t );
drivers/net/igb/igb_main.c static struct pci_driver igb_driver = { · line 190
Quoted 6 lines
drivers/net/igb/igb_main.c +3
-0@@ -184,6 +185,9 @@ static struct pci_driver igb_driver = {#endif .shutdown = igb_shutdown, .err_handler = &igb_err_handler, +#ifdef CONFIG_PCI_IOV + .virtual = igb_virtual +#endif
#ifdef should not be needed, right?
Quoted 8 lines
drivers/net/igb/igb_main.c +1
-0};static int global_quad_port_a;/* global quad port a indication */ @@ -5107,6 +5111,32 @@ void igb_set_mc_list_pools(struct igb_adapter *adapter, reg_data |= (1 <<25 );wr32 (E1000_VMOLR (pool), reg_data); } +
drivers/net/igb/igb_main.c void igb_set_mc_list_pools(struct igb_adapter *adapter, · line 5124
Quoted 10 lines
drivers/net/igb/igb_main.c +10
-0+static int +igb_virtual (struct pci_dev *pdev,int nr_virtfn) +{ +unsigned char my_mac_addr[6 ] = {0x00 ,0xDE ,0xAD ,0xBE ,0xEF ,0xFF }; +struct net_device *netdev =pci_get_drvdata (pdev); +struct igb_adapter *adapter =netdev_priv (netdev); +int i; + +if (nr_virtfn >7 ) +return -EINVAL ;
Why the check for 7? Is that the max virtual functions for this card? Shouldn't that be a define somewhere so it's easier to fix in future versions of this hardware? :)
drivers/net/igb/igb_main.c void igb_set_mc_list_pools(struct igb_adapter *adapter, · line 5128
Quoted 4 lines
drivers/net/igb/igb_main.c +4
-0+ +if (nr_virtfn) { +for (i =0 ; i < nr_virtfn; i++) { +printk (KERN_INFO "SR-IOV: VF %d is enabled\n" , i);
Use dev_info() please, that shows the exact pci device and driver that emitted the message.
drivers/net/igb/igb_main.c void igb_set_mc_list_pools(struct igb_adapter *adapter, · line 5134
Quoted 6 lines
drivers/net/igb/igb_main.c +6
-0+ my_mac_addr[5 ] = (unsigned char )i; +igb_set_vf_mac (netdev, i, my_mac_addr); +igb_set_vf_vmolr (adapter, i); + } + }else +printk (KERN_INFO "SR-IOV is disabled\n" );
Is that really true? (oh, use dev_info as well.) What happens if you had called this with "5" and then later with "0", you never destroyed those existing virtual functions, yet the code does:
drivers/net/igb/igb_main.c void igb_set_mc_list_pools(struct igb_adapter *adapter, · line 5136
Quoted 1 line
drivers/net/igb/igb_main.c +1
-0+ adapter->vfs_allocated_count = nr_virtfn;
Which makes the driver think they are not present. What happens when the driver later goes to shut down? Are those resources freed up properly?
thanks,
greg k-h
PATCH 2/3 PF driver: integrate with SR-IOV core
In reply to Yu ZhaoThis patch integrates the IGB driver with the SR-IOV core. It shows how the SR-IOV API is used to support the capability. Obviously people does not need to put much effort to integrate the PF driver with SR-IOV core. All SR-IOV standard stuff are handled by SR-IOV core and PF driver only concerns the device specific resource allocation and deallocation once it gets the necessary information (i.e. number of Virtual Functions) from the callback function.
--- drivers/net/igb/igb_main.c | 30 ++++++++++++++++++++++++++++++ 1 files changed, 30 insertions(+), 0 deletions(-)
drivers/net/igb/igb_main.c +30 -0
@@ -139,6 +139,7 @@ void igb_set_mc_list_pools(struct igb_adapter *, struct e1000_hw *, int, u16);static int igb_vmm_control (struct igb_adapter *,bool );static int igb_set_vf_mac (struct net_device *,int ,u8 *);static void igb_mbox_handler (struct igb_adapter *); +static int igb_virtual (struct pci_dev *,int );#endif static int igb_suspend (struct pci_dev *,pm_message_t );
@@ -184,6 +185,9 @@ static struct pci_driver igb_driver = {#endif .shutdown = igb_shutdown, .err_handler = &igb_err_handler, +#ifdef CONFIG_PCI_IOV + .virtual = igb_virtual +#endif };static int global_quad_port_a;/* global quad port a indication */
@@ -5107,6 +5111,32 @@ void igb_set_mc_list_pools(struct igb_adapter *adapter, reg_data |= (1 <<25 );wr32 (E1000_VMOLR (pool), reg_data); } + +static int +igb_virtual (struct pci_dev *pdev,int nr_virtfn) +{ +unsigned char my_mac_addr[6 ] = {0x00 ,0xDE ,0xAD ,0xBE ,0xEF ,0xFF }; +struct net_device *netdev =pci_get_drvdata (pdev); +struct igb_adapter *adapter =netdev_priv (netdev); +int i; + +if (nr_virtfn >7 ) +return -EINVAL ; + +if (nr_virtfn) { +for (i =0 ; i < nr_virtfn; i++) { +printk (KERN_INFO "SR-IOV: VF %d is enabled\n" , i); + my_mac_addr[5 ] = (unsigned char )i; +igb_set_vf_mac (netdev, i, my_mac_addr); +igb_set_vf_vmolr (adapter, i); + } + }else +printk (KERN_INFO "SR-IOV is disabled\n" ); + + adapter->vfs_allocated_count = nr_virtfn; + +return 0 ; +}#endif /* igb_main.c */
Signature · 1 line
1.5.4.4
PATCH 3/3 VF driver tar ball
In reply to Yu ZhaoThe attachment is the VF driver for Intel 82576 NIC. Since the VF appears as the normal PCI device driver, this VF driver has no difference from other PCI NIC drivers. It handles interrupt, DMA operations, etc. to perform packet receiving and transmission.
How the design of the VF internals is up to the hardware vendor. So the VF may have totally different register set from the PF, which means the VF driver may have its own logics (rather than derive from PF driver) to handle the hardware specific stuff.
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel
Re: [SR-IOV driver example 3/3] VF driver tar ball
Quoted 1 line · Yu Zhao
The attachment is the VF driver for Intel 82576 NIC.
Please don't attach things as tarballs, we can't review or easily read them at all.
Care to resend it?
thanks,
greg k-h
Re: [SR-IOV driver example 3/3] VF driver tar ball
In reply to Yu ZhaoQuoted 1 line · Yu Zhao
The attachment is the VF driver for Intel 82576 NIC.
Please don't attach things as tarballs, we can't review or easily read them at all.
Care to resend it?
thanks,
greg k-h
PATCH 3/3 VF driver tar ball
In reply to Yu ZhaoThe attachment is the VF driver for Intel 82576 NIC. Since the VF appears as the normal PCI device driver, this VF driver has no difference from other PCI NIC drivers. It handles interrupt, DMA operations, etc. to perform packet receiving and transmission.
How the design of the VF internals is up to the hardware vendor. So the VF may have totally different register set from the PF, which means the VF driver may have its own logics (rather than derive from PF driver) to handle the hardware specific stuff.
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linux-foundation.org/mailman/listinfo/virtualization
Re: [SR-IOV driver example 0/3] introduction
In reply to Yu ZhaoQuoted 10 lines · Yu Zhao
SR-IOV drivers of Intel 82576 NIC are available. There are two parts of the drivers: Physical Function driver and Virtual Function driver. The PF driver is based on the IGB driver and is used to control PF to allocate hardware specific resources and interface with the SR-IOV core. The VF driver is a new NIC driver that is same as the traditional PCI device driver. It works in both the host and the guest (Xen and KVM) environment.
These two drivers are testing versions and they are *only* intended to show how to use SR-IOV API.
That's funny, as some distros are already shipping this driver. You might want to tell them that this is an "example only" driver and not to be used "for real"... :(
greg k-h
Re: [SR-IOV driver example 0/3] introduction
Quoted 9 lines · Greg KH
Quoted 8 lines · Yu Zhao
SR-IOV drivers of Intel 82576 NIC are available. There are two parts of the drivers: Physical Function driver and Virtual Function driver. The PF driver is based on the IGB driver and is used to control PF to allocate hardware specific resources and interface with the SR-IOV core. The VF driver is a new NIC driver that is same as the traditional PCI device driver. It works in both the host and the guest (Xen and KVM) environment.
Quoted 6 lines
Quoted 2 lines
These two drivers are testing versions and they are *only* intended to show how to use SR-IOV API.
That's funny, as some distros are already shipping this driver. You might want to tell them that this is an "example only" driver and not to be used "for real"... :(
Maybe they are shipping another version, not this one. This one is really a experimental patch, it's just created a week before...
Re: [SR-IOV driver example 0/3] introduction
In reply to Greg KHQuoted 9 lines · Greg KH
Quoted 8 lines · Yu Zhao
SR-IOV drivers of Intel 82576 NIC are available. There are two parts of the drivers: Physical Function driver and Virtual Function driver. The PF driver is based on the IGB driver and is used to control PF to allocate hardware specific resources and interface with the SR-IOV core. The VF driver is a new NIC driver that is same as the traditional PCI device driver. It works in both the host and the guest (Xen and KVM) environment.
Quoted 6 lines
Quoted 2 lines
These two drivers are testing versions and they are *only* intended to show how to use SR-IOV API.
That's funny, as some distros are already shipping this driver. You might want to tell them that this is an "example only" driver and not to be used "for real"... :(
Maybe they are shipping another version, not this one. This one is really a experimental patch, it's just created a week before...
Re: [SR-IOV driver example 0/3] introduction
In reply to Yu ZhaoQuoted 10 lines · Yu Zhao
SR-IOV drivers of Intel 82576 NIC are available. There are two parts of the drivers: Physical Function driver and Virtual Function driver. The PF driver is based on the IGB driver and is used to control PF to allocate hardware specific resources and interface with the SR-IOV core. The VF driver is a new NIC driver that is same as the traditional PCI device driver. It works in both the host and the guest (Xen and KVM) environment.
These two drivers are testing versions and they are *only* intended to show how to use SR-IOV API.
That's funny, as some distros are already shipping this driver. You might want to tell them that this is an "example only" driver and not to be used "for real"... :(
greg k-h
Re: [SR-IOV driver example 0/3] introduction
In reply to Yu ZhaoQuoted 11 lines · Yu Zhao
SR-IOV drivers of Intel 82576 NIC are available. There are two parts of the drivers: Physical Function driver and Virtual Function driver. The PF driver is based on the IGB driver and is used to control PF to allocate hardware specific resources and interface with the SR-IOV core. The VF driver is a new NIC driver that is same as the traditional PCI device driver. It works in both the host and the guest (Xen and KVM) environment.
These two drivers are testing versions and they are *only* intended to show how to use SR-IOV API.
Quoted 6 lines
Intel 82576 NIC specification can be found at:
http://download.intel.com/design/network/datashts/82576_Datasheet_v2p1.pdf[SR-IOV driver example 1/3] PF driver: allocate hardware specific resource [SR-IOV driver example 2/3] PF driver: integrate with SR-IOV core [SR-IOV driver example 3/3] VF driver tar ball
Please copy [email protected] on all network-related patches. This is where the network developers live, and all patches on this list are automatically archived for review and handling at http://patchwork.ozlabs.org/project/netdev/list/
Jeff
Re: [SR-IOV driver example 0/3] introduction
Quoted 15 lines · Jeff Garzik
Quoted 14 lines · Yu Zhao
SR-IOV drivers of Intel 82576 NIC are available. There are two parts of the drivers: Physical Function driver and Virtual Function driver. The PF driver is based on the IGB driver and is used to control PF to allocate hardware specific resources and interface with the SR-IOV core. The VF driver is a new NIC driver that is same as the traditional PCI device driver. It works in both the host and the guest (Xen and KVM) environment.
These two drivers are testing versions and they are *only* intended to show how to use SR-IOV API.
Intel 82576 NIC specification can be found at:
http://download.intel.com/design/network/datashts/82576_Datasheet_v2p1.pdf
Quoted 8 lines
Quoted 3 lines
[SR-IOV driver example 1/3] PF driver: allocate hardware specific resource [SR-IOV driver example 2/3] PF driver: integrate with SR-IOV core [SR-IOV driver example 3/3] VF driver tar ball
Please copy [email protected] on all network-related patches. This is where the network developers live, and all patches on this list are automatically archived for review and handling at
http://patchwork.ozlabs.org/project/netdev/list/
Will do.
Thanks,
Yu
Re: [SR-IOV driver example 0/3] introduction
In reply to Jeff GarzikQuoted 15 lines · Jeff Garzik
Quoted 14 lines · Yu Zhao
SR-IOV drivers of Intel 82576 NIC are available. There are two parts of the drivers: Physical Function driver and Virtual Function driver. The PF driver is based on the IGB driver and is used to control PF to allocate hardware specific resources and interface with the SR-IOV core. The VF driver is a new NIC driver that is same as the traditional PCI device driver. It works in both the host and the guest (Xen and KVM) environment.
These two drivers are testing versions and they are *only* intended to show how to use SR-IOV API.
Intel 82576 NIC specification can be found at:
http://download.intel.com/design/network/datashts/82576_Datasheet_v2p1.pdf
Quoted 8 lines
Quoted 3 lines
[SR-IOV driver example 1/3] PF driver: allocate hardware specific resource [SR-IOV driver example 2/3] PF driver: integrate with SR-IOV core [SR-IOV driver example 3/3] VF driver tar ball
Please copy [email protected] on all network-related patches. This is where the network developers live, and all patches on this list are automatically archived for review and handling at
http://patchwork.ozlabs.org/project/netdev/list/
Will do.
Thanks,
Yu
Re: [SR-IOV driver example 0/3] introduction
In reply to Yu ZhaoQuoted 11 lines · Yu Zhao
SR-IOV drivers of Intel 82576 NIC are available. There are two parts of the drivers: Physical Function driver and Virtual Function driver. The PF driver is based on the IGB driver and is used to control PF to allocate hardware specific resources and interface with the SR-IOV core. The VF driver is a new NIC driver that is same as the traditional PCI device driver. It works in both the host and the guest (Xen and KVM) environment.
These two drivers are testing versions and they are *only* intended to show how to use SR-IOV API.
Quoted 6 lines
Intel 82576 NIC specification can be found at:
http://download.intel.com/design/network/datashts/82576_Datasheet_v2p1.pdf[SR-IOV driver example 1/3] PF driver: allocate hardware specific resource [SR-IOV driver example 2/3] PF driver: integrate with SR-IOV core [SR-IOV driver example 3/3] VF driver tar ball
Please copy [email protected] on all network-related patches. This is where the network developers live, and all patches on this list are automatically archived for review and handling at
http://patchwork.ozlabs.org/project/netdev/list/
Jeff
cover introduction
In reply to Yu ZhaoSR-IOV drivers of Intel 82576 NIC are available. There are two parts of the drivers: Physical Function driver and Virtual Function driver. The PF driver is based on the IGB driver and is used to control PF to allocate hardware specific resources and interface with the SR-IOV core. The VF driver is a new NIC driver that is same as the traditional PCI device driver. It works in both the host and the guest (Xen and KVM) environment.
These two drivers are testing versions and they are *only* intended to show how to use SR-IOV API.
Intel 82576 NIC specification can be found at:
http://download.intel.com/design/network/datashts/82576_Datasheet_v2p1.pdf
[SR-IOV driver example 1/3] PF driver: allocate hardware specific resource [SR-IOV driver example 2/3] PF driver: integrate with SR-IOV core [SR-IOV driver example 3/3] VF driver tar ball
cover introduction
In reply to Yu ZhaoSR-IOV drivers of Intel 82576 NIC are available. There are two parts of the drivers: Physical Function driver and Virtual Function driver. The PF driver is based on the IGB driver and is used to control PF to allocate hardware specific resources and interface with the SR-IOV core. The VF driver is a new NIC driver that is same as the traditional PCI device driver. It works in both the host and the guest (Xen and KVM) environment.
These two drivers are testing versions and they are *only* intended to show how to use SR-IOV API.
Intel 82576 NIC specification can be found at:
http://download.intel.com/design/network/datashts/82576_Datasheet_v2p1.pdf
[SR-IOV driver example 0/3 resend] introduction
[SR-IOV driver example 1/3 resend] PF driver: hardware specific operations [SR-IOV driver example 2/3 resend] PF driver: integrate with SR-IOV core [SR-IOV driver example 3/3 resend] VF driver: an independent PCI NIC driver
PATCH RESEND 1/3 PF driver: hardware specific operations
This patch makes the IGB driver allocate hardware resource (rx/tx queues) for Virtual Functions. All operations in this patch are hardware specific.
From: Intel Corporation, LAN Access Division <[email protected]>
--- drivers/net/igb/Makefile | 2 +- drivers/net/igb/e1000_82575.c | 1 + drivers/net/igb/e1000_82575.h | 61 +++++ drivers/net/igb/e1000_defines.h | 7 + drivers/net/igb/e1000_hw.h | 2 + drivers/net/igb/e1000_regs.h | 13 + drivers/net/igb/igb.h | 8 + drivers/net/igb/igb_main.c | 567 +++++++++++++++++++++++++++++++++++++- drivers/pci/iov.c | 6 +- 9 files changed, 649 insertions(+), 18 deletions(-)
drivers/net/igb/Makefile +1 -1
@@ -33,5 +33,5 @@ obj-$(CONFIG_IGB) += igb.o igb-objs := igb_main.o igb_ethtool.o e1000_82575.o \ - e1000_mac.o e1000_nvm.o e1000_phy.o + e1000_mac.o e1000_nvm.o e1000_phy.o e1000_vf.o
drivers/net/igb/e1000_82575.c +1 -0
@@ -87,6 +87,7 @@ static s32 igb_get_invariants_82575(struct e1000_hw *hw)case E1000_DEV_ID_82576 :case E1000_DEV_ID_82576_FIBER :case E1000_DEV_ID_82576_SERDES : +case E1000_DEV_ID_82576_QUAD_COPPER : mac->type = e1000_82576;break ;default :
drivers/net/igb/e1000_82575.h +61 -0
65 lines not on this page · raw message
drivers/net/igb/e1000_defines.h +7 -0
@@ -389,6 +389,7 @@#define E1000_ICR_RXDMT0 0x00000010/* rx desc min. threshold (0) */ #define E1000_ICR_RXO 0x00000040/* rx overrun */ #define E1000_ICR_RXT0 0x00000080/* rx timer intr (ring 0) */ +#define E1000_ICR_VMMB 0x00000100/* VM MB event */ #define E1000_ICR_MDAC 0x00000200/* MDIO access complete */ #define E1000_ICR_RXCFG 0x00000400/* Rx /c/ ordered set */ #define E1000_ICR_GPI_EN0 0x00000800/* GP Int 0 */
@@ -451,6 +452,7 @@/* Interrupt Mask Set */ #define E1000_IMS_TXDW E1000_ICR_TXDW/* Transmit desc written back */ #define E1000_IMS_LSC E1000_ICR_LSC/* Link Status Change */ +#define E1000_IMS_VMMB E1000_ICR_VMMB/* Mail box activity */ #define E1000_IMS_RXSEQ E1000_ICR_RXSEQ/* rx sequence error */ #define E1000_IMS_RXDMT0 E1000_ICR_RXDMT0/* rx desc min. threshold */ #define E1000_IMS_RXT0 E1000_ICR_RXT0/* rx timer intr */
@@ -768,4 +770,9 @@#define E1000_GEN_CTL_ADDRESS_SHIFT 8#define E1000_GEN_POLL_TIMEOUT 640 +#define E1000_WRITE_FLUSH (a) (readl((a)->hw_addr + E1000_STATUS)) +#define E1000_MRQC_ENABLE_MASK 0x00000007 +#define E1000_MRQC_ENABLE_VMDQ 0x00000003 +#define E1000_CTRL_EXT_PFRSTD 0x00004000 +#endif
drivers/net/igb/e1000_hw.h +2 -0
@@ -41,6 +41,7 @@ struct e1000_hw;#define E1000_DEV_ID_82576 0x10C9#define E1000_DEV_ID_82576_FIBER 0x10E6#define E1000_DEV_ID_82576_SERDES 0x10E7 +#define E1000_DEV_ID_82576_QUAD_COPPER 0x10E8#define E1000_DEV_ID_82575EB_COPPER 0x10A7#define E1000_DEV_ID_82575EB_FIBER_SERDES 0x10A9#define E1000_DEV_ID_82575GB_QUAD_COPPER 0x10D6
@@ -91,6 +92,7 @@ enum e1000_phy_type { e1000_phy_gg82563, e1000_phy_igp_3, e1000_phy_ife, + e1000_phy_vf, }; enum e1000_bus_type {
drivers/net/igb/e1000_regs.h +13 -0
@@ -262,6 +262,19 @@#define E1000_RETA (_i) (0x05C00 + ((_i) * 4))#define E1000_RSSRK (_i) (0x05C80 + ((_i) * 4))/* RSS Random Key - RW Array */ +/* VT Registers */ +#define E1000_MBVFICR 0x00C80/* Mailbox VF Cause - RWC */ +#define E1000_MBVFIMR 0x00C84/* Mailbox VF int Mask - RW */ +#define E1000_VFLRE 0x00C88/* VF Register Events - RWC */ +#define E1000_VFRE 0x00C8C/* VF Receive Enables */ +#define E1000_VFTE 0x00C90/* VF Transmit Enables */ +#define E1000_DTXSWC 0x03500/* DMA Tx Switch Control - RW */ +/* These act per VF so an array friendly macro is used */ +#define E1000_P2VMAILBOX (_n) (0x00C00 + (4 * (_n))) +#define E1000_VMBMEM (_n) (0x00800 + (64 * (_n))) +#define E1000_VMOLR (_n) (0x05AD0 + (4 * (_n))) +#define E1000_VLVF (_n) (0x05D00 + (4 * (_n)))/* VLAN Virtual Machine */ +#define wr32 (reg, value) (writel(value, hw->hw_addr + reg))#define rd32 (reg) (readl(hw->hw_addr + reg))#define wrfl () ((void)rd32(E1000_STATUS))
drivers/net/igb/igb.h +8 -0
@@ -294,6 +294,14 @@ struct igb_adapter {unsigned int lro_flushed;unsigned int lro_no_desc;#endif +unsigned int vfs_allocated_count; +struct work_struct msg_task; +u32 vf_icr; +u32 vflre; +unsigned char vf_mac_addresses[8 ][6 ]; +u8 vfta_tracking_entry[128 ]; +int int0counter; +int int1counter; };#define IGB_FLAG_HAS_MSI (1 << 0)
drivers/net/igb/igb_main.c +552 -15
754 lines not on this page · raw message
drivers/pci/iov.c +4 -2
@@ -487,9 +487,11 @@ void pci_iov_unregister(struct pci_dev *dev)sysfs_remove_group (&dev->dev.kobj, &iov_attr_group); -mutex_lock (&pdev->iov->physfn->iov->lock); +mutex_lock (&dev->iov->physfn->iov->lock); +iov_disable (dev); -mutex_unlock (&pdev->iov->physfn->iov->lock); + +mutex_unlock (&dev->iov->physfn->iov->lock);kobject_uevent (&dev->dev.kobj,KOBJ_CHANGE ); }
Signature · 1 line
1.5.6.4
PATCH RESEND 1/3 PF driver: hardware specific operations
In reply to Yu ZhaoThis patch makes the IGB driver allocate hardware resource (rx/tx queues) for Virtual Functions. All operations in this patch are hardware specific.
From: Intel Corporation, LAN Access Division <[email protected]>
--- drivers/net/igb/Makefile | 2 +- drivers/net/igb/e1000_82575.c | 1 + drivers/net/igb/e1000_82575.h | 61 +++++ drivers/net/igb/e1000_defines.h | 7 + drivers/net/igb/e1000_hw.h | 2 + drivers/net/igb/e1000_regs.h | 13 + drivers/net/igb/igb.h | 8 + drivers/net/igb/igb_main.c | 567 +++++++++++++++++++++++++++++++++++++- drivers/pci/iov.c | 6 +- 9 files changed, 649 insertions(+), 18 deletions(-)
drivers/net/igb/Makefile +1 -1
@@ -33,5 +33,5 @@ obj-$(CONFIG_IGB) += igb.o igb-objs := igb_main.o igb_ethtool.o e1000_82575.o \ - e1000_mac.o e1000_nvm.o e1000_phy.o + e1000_mac.o e1000_nvm.o e1000_phy.o e1000_vf.o
drivers/net/igb/e1000_82575.c +1 -0
@@ -87,6 +87,7 @@ static s32 igb_get_invariants_82575(struct e1000_hw *hw)case E1000_DEV_ID_82576 :case E1000_DEV_ID_82576_FIBER :case E1000_DEV_ID_82576_SERDES : +case E1000_DEV_ID_82576_QUAD_COPPER : mac->type = e1000_82576;break ;default :
drivers/net/igb/e1000_82575.h +61 -0
65 lines not on this page · raw message
drivers/net/igb/e1000_defines.h +7 -0
@@ -389,6 +389,7 @@#define E1000_ICR_RXDMT0 0x00000010/* rx desc min. threshold (0) */ #define E1000_ICR_RXO 0x00000040/* rx overrun */ #define E1000_ICR_RXT0 0x00000080/* rx timer intr (ring 0) */ +#define E1000_ICR_VMMB 0x00000100/* VM MB event */ #define E1000_ICR_MDAC 0x00000200/* MDIO access complete */ #define E1000_ICR_RXCFG 0x00000400/* Rx /c/ ordered set */ #define E1000_ICR_GPI_EN0 0x00000800/* GP Int 0 */
@@ -451,6 +452,7 @@/* Interrupt Mask Set */ #define E1000_IMS_TXDW E1000_ICR_TXDW/* Transmit desc written back */ #define E1000_IMS_LSC E1000_ICR_LSC/* Link Status Change */ +#define E1000_IMS_VMMB E1000_ICR_VMMB/* Mail box activity */ #define E1000_IMS_RXSEQ E1000_ICR_RXSEQ/* rx sequence error */ #define E1000_IMS_RXDMT0 E1000_ICR_RXDMT0/* rx desc min. threshold */ #define E1000_IMS_RXT0 E1000_ICR_RXT0/* rx timer intr */
@@ -768,4 +770,9 @@#define E1000_GEN_CTL_ADDRESS_SHIFT 8#define E1000_GEN_POLL_TIMEOUT 640 +#define E1000_WRITE_FLUSH (a) (readl((a)->hw_addr + E1000_STATUS)) +#define E1000_MRQC_ENABLE_MASK 0x00000007 +#define E1000_MRQC_ENABLE_VMDQ 0x00000003 +#define E1000_CTRL_EXT_PFRSTD 0x00004000 +#endif
drivers/net/igb/e1000_hw.h +2 -0
@@ -41,6 +41,7 @@ struct e1000_hw;#define E1000_DEV_ID_82576 0x10C9#define E1000_DEV_ID_82576_FIBER 0x10E6#define E1000_DEV_ID_82576_SERDES 0x10E7 +#define E1000_DEV_ID_82576_QUAD_COPPER 0x10E8#define E1000_DEV_ID_82575EB_COPPER 0x10A7#define E1000_DEV_ID_82575EB_FIBER_SERDES 0x10A9#define E1000_DEV_ID_82575GB_QUAD_COPPER 0x10D6
@@ -91,6 +92,7 @@ enum e1000_phy_type { e1000_phy_gg82563, e1000_phy_igp_3, e1000_phy_ife, + e1000_phy_vf, }; enum e1000_bus_type {
drivers/net/igb/e1000_regs.h +13 -0
@@ -262,6 +262,19 @@#define E1000_RETA (_i) (0x05C00 + ((_i) * 4))#define E1000_RSSRK (_i) (0x05C80 + ((_i) * 4))/* RSS Random Key - RW Array */ +/* VT Registers */ +#define E1000_MBVFICR 0x00C80/* Mailbox VF Cause - RWC */ +#define E1000_MBVFIMR 0x00C84/* Mailbox VF int Mask - RW */ +#define E1000_VFLRE 0x00C88/* VF Register Events - RWC */ +#define E1000_VFRE 0x00C8C/* VF Receive Enables */ +#define E1000_VFTE 0x00C90/* VF Transmit Enables */ +#define E1000_DTXSWC 0x03500/* DMA Tx Switch Control - RW */ +/* These act per VF so an array friendly macro is used */ +#define E1000_P2VMAILBOX (_n) (0x00C00 + (4 * (_n))) +#define E1000_VMBMEM (_n) (0x00800 + (64 * (_n))) +#define E1000_VMOLR (_n) (0x05AD0 + (4 * (_n))) +#define E1000_VLVF (_n) (0x05D00 + (4 * (_n)))/* VLAN Virtual Machine */ +#define wr32 (reg, value) (writel(value, hw->hw_addr + reg))#define rd32 (reg) (readl(hw->hw_addr + reg))#define wrfl () ((void)rd32(E1000_STATUS))
drivers/net/igb/igb.h +8 -0
@@ -294,6 +294,14 @@ struct igb_adapter {unsigned int lro_flushed;unsigned int lro_no_desc;#endif +unsigned int vfs_allocated_count; +struct work_struct msg_task; +u32 vf_icr; +u32 vflre; +unsigned char vf_mac_addresses[8 ][6 ]; +u8 vfta_tracking_entry[128 ]; +int int0counter; +int int1counter; };#define IGB_FLAG_HAS_MSI (1 << 0)
drivers/net/igb/igb_main.c +552 -15
754 lines not on this page · raw message
drivers/pci/iov.c +4 -2
@@ -487,9 +487,11 @@ void pci_iov_unregister(struct pci_dev *dev)sysfs_remove_group (&dev->dev.kobj, &iov_attr_group); -mutex_lock (&pdev->iov->physfn->iov->lock); +mutex_lock (&dev->iov->physfn->iov->lock); +iov_disable (dev); -mutex_unlock (&pdev->iov->physfn->iov->lock); + +mutex_unlock (&dev->iov->physfn->iov->lock);kobject_uevent (&dev->dev.kobj,KOBJ_CHANGE ); }
Signature · 1 line
1.5.6.4
PATCH RESEND 2/3 PF driver: integrate with SR-IOV core
In reply to Yu ZhaoThis patch integrates the IGB driver with the SR-IOV core. It shows how the SR-IOV API is used to support the capability. Obviously people does not need to put much effort to integrate the PF driver with SR-IOV core. All SR-IOV standard stuff are handled by SR-IOV core and PF driver only concerns the device specific resource allocation and deallocation once it gets the necessary information (i.e. number of Virtual Functions) from the callback function.
From: Intel Corporation, LAN Access Division <[email protected]>
--- drivers/net/igb/igb_main.c | 46 ++++++++++++++++++++++++++++++++++++++++++++ 1 files changed, 46 insertions(+), 0 deletions(-)
drivers/net/igb/igb_main.c +46 -0
@@ -138,6 +138,7 @@ void igb_set_mc_list_pools(struct igb_adapter *, struct e1000_hw *, int, u16);static int igb_vmm_control (struct igb_adapter *,bool );static int igb_set_vf_mac (struct net_device *,int ,u8 *);static void igb_mbox_handler (struct igb_adapter *); +static int igb_virtual (struct pci_dev *,int );static int igb_suspend (struct pci_dev *,pm_message_t );#ifdef CONFIG_PM
@@ -182,6 +183,7 @@ static struct pci_driver igb_driver = {#endif .shutdown = igb_shutdown, .err_handler = &igb_err_handler, + .virtual = igb_virtual };static int global_quad_port_a;/* global quad port a indication */
@@ -5066,4 +5068,48 @@ void igb_set_mc_list_pools(struct igb_adapter *adapter,wr32 (E1000_VMOLR (pool), reg_data); } +static int +igb_virtual (struct pci_dev *pdev,int nr_virtfn) +{ +int i; +struct net_device *netdev =pci_get_drvdata (pdev); +struct igb_adapter *adapter =netdev_priv (netdev); +/* the VFs' MAC addresses are hard-coded */ +unsigned char my_mac_addr[6 ] = {0x00 ,0xDE ,0xAD ,0xBE ,0xEF ,0xFF }; + +/* +* the 82576 NIC supports 1-PF NIC + 7-VF NICs mode and 8-VF NICs +* mode. In the 8-VF NICs mode, the PF can't tx/rx packets -- it +* only behaves as 'VF supervisor'. For now we use the 1-PF NIC + +* 7-VF NICs mode to preserve PF's tx/rx capability for the debug +* purpose. +*/ +if (nr_virtfn > (MAX_NUM_VFS -1 )) +return -EINVAL ; + +if (nr_virtfn) { +dev_info (&pdev->dev,"SR-IOV is enabled\n" ); +/* +* Currently VFs resources are pre-allocated, so just set +* the MAC addresses of each VF here. +*/ +for (i =0 ; i < nr_virtfn; i++) { + my_mac_addr[5 ] = (unsigned char )i; +igb_set_vf_mac (netdev, i, my_mac_addr); +igb_set_vf_vmolr (adapter, i); + } + }else { +/* +* Since we statically allocate tx/rx queues for the PF +* and the VFs, so we don't need to free any VF related +* resources here. +*/ +dev_info (&pdev->dev,"SR-IOV is disabled\n" ); + } + + adapter->vfs_allocated_count = nr_virtfn; + +return 0 ; +} +/* igb_main.c */
Signature · 1 line
1.5.6.4
PATCH RESEND 2/3 PF driver: integrate with SR-IOV core
In reply to Yu ZhaoThis patch integrates the IGB driver with the SR-IOV core. It shows how the SR-IOV API is used to support the capability. Obviously people does not need to put much effort to integrate the PF driver with SR-IOV core. All SR-IOV standard stuff are handled by SR-IOV core and PF driver only concerns the device specific resource allocation and deallocation once it gets the necessary information (i.e. number of Virtual Functions) from the callback function.
From: Intel Corporation, LAN Access Division <[email protected]>
--- drivers/net/igb/igb_main.c | 46 ++++++++++++++++++++++++++++++++++++++++++++ 1 files changed, 46 insertions(+), 0 deletions(-)
drivers/net/igb/igb_main.c +46 -0
@@ -138,6 +138,7 @@ void igb_set_mc_list_pools(struct igb_adapter *, struct e1000_hw *, int, u16);static int igb_vmm_control (struct igb_adapter *,bool );static int igb_set_vf_mac (struct net_device *,int ,u8 *);static void igb_mbox_handler (struct igb_adapter *); +static int igb_virtual (struct pci_dev *,int );static int igb_suspend (struct pci_dev *,pm_message_t );#ifdef CONFIG_PM
@@ -182,6 +183,7 @@ static struct pci_driver igb_driver = {#endif .shutdown = igb_shutdown, .err_handler = &igb_err_handler, + .virtual = igb_virtual };static int global_quad_port_a;/* global quad port a indication */
@@ -5066,4 +5068,48 @@ void igb_set_mc_list_pools(struct igb_adapter *adapter,wr32 (E1000_VMOLR (pool), reg_data); } +static int +igb_virtual (struct pci_dev *pdev,int nr_virtfn) +{ +int i; +struct net_device *netdev =pci_get_drvdata (pdev); +struct igb_adapter *adapter =netdev_priv (netdev); +/* the VFs' MAC addresses are hard-coded */ +unsigned char my_mac_addr[6 ] = {0x00 ,0xDE ,0xAD ,0xBE ,0xEF ,0xFF }; + +/* +* the 82576 NIC supports 1-PF NIC + 7-VF NICs mode and 8-VF NICs +* mode. In the 8-VF NICs mode, the PF can't tx/rx packets -- it +* only behaves as 'VF supervisor'. For now we use the 1-PF NIC + +* 7-VF NICs mode to preserve PF's tx/rx capability for the debug +* purpose. +*/ +if (nr_virtfn > (MAX_NUM_VFS -1 )) +return -EINVAL ; + +if (nr_virtfn) { +dev_info (&pdev->dev,"SR-IOV is enabled\n" ); +/* +* Currently VFs resources are pre-allocated, so just set +* the MAC addresses of each VF here. +*/ +for (i =0 ; i < nr_virtfn; i++) { + my_mac_addr[5 ] = (unsigned char )i; +igb_set_vf_mac (netdev, i, my_mac_addr); +igb_set_vf_vmolr (adapter, i); + } + }else { +/* +* Since we statically allocate tx/rx queues for the PF +* and the VFs, so we don't need to free any VF related +* resources here. +*/ +dev_info (&pdev->dev,"SR-IOV is disabled\n" ); + } + + adapter->vfs_allocated_count = nr_virtfn; + +return 0 ; +} +/* igb_main.c */
Signature · 1 line
1.5.6.4
PATCH RESEND 3/3 VF driver: an independent PCI NIC driver
In reply to Yu ZhaoThis patch is the VF driver for Intel 82576 NIC. Since the VF appears as the normal PCI device driver, this VF driver has no difference from other PCI NIC drivers. It handles interrupt, DMA operations, etc. to perform packet receiving and transmission.
How the design of the VF internals is up to the hardware vendor. So the VF may have totally different register set from the PF, which means the VF driver may have its own logics (rather than derive from PF driver) to handle the hardware specific stuff.
This driver is a totally new driver, currently there is no logical-split patches available. The attached single diff file includes code.
From: Intel Corporation, LAN Access Division <[email protected]>
COPYING +339 -0
@@ -0,0 +1,339 @@ + +"This software program is licensed subject to the GNU General Public License +(GPL). Version 2, June 1991, available at +<http://www.fsf.org/copyleft/gpl.html>" + +GNU General Public License + +Version 2, June 1991 + +Copyright (C) 1989, 1991 Free Software Foundation, Inc. +59 Temple Place - Suite 330, Boston, MA 02111-1307, USA + +Everyone is permitted to copy and distribute verbatim copies of this license +document, but changing it is not allowed. + +Preamble + +The licenses for most software are designed to take away your freedom to +share and change it. By contrast, the GNU General Public License is intended +to guarantee your freedom to share and change free software--to make sure +the software is free for all its users. This General Public License applies +to most of the Free Software Foundation's software and to any other program +whose authors commit to using it. (Some other Free Software Foundation +software is covered by the GNU Library General Public License instead.) You +can apply it to your programs, too. + +When we speak of free software, we are referring to freedom, not price. Our +General Public Licenses are designed to make sure that you have the freedom +to distribute copies of free software (and charge for this service if you +wish), that you receive source code or can get it if you want it, that you +can change the software or use pieces of it in new free programs; and that +you know you can do these things. + +To protect your rights, we need to make restrictions that forbid anyone to +deny you these rights or to ask you to surrender the rights. These +restrictions translate to certain responsibilities for you if you distribute +copies of the software, or if you modify it. + +For example, if you distribute copies of such a program, whether gratis or +for a fee, you must give the recipients all the rights that you have. You +must make sure that they, too, receive or can get the source code. And you +must show them these terms so they know their rights. + +We protect your rights with two steps: (1) copyright the software, and (2) +offer you this license which gives you legal permission to copy, distribute +and/or modify the software. + +Also, for each author's protection and ours, we want to make certain that +everyone understands that there is no warranty for this free software. If +the software is modified by someone else and passed on, we want its +recipients to know that what they have is not the original, so that any +problems introduced by others will not reflect on the original authors' +reputations. + +Finally, any free program is threatened constantly by software patents. We +wish to avoid the danger that redistributors of a free program will +individually obtain patent licenses, in effect making the program +proprietary. To prevent this, we have made it clear that any patent must be +licensed for everyone's free use or not licensed at all. + +The precise terms and conditions for copying, distribution and modification +follow. + +TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION + +0. This License applies to any program or other work which contains a notice + placed by the copyright holder saying it may be distributed under the + terms of this General Public License. The "Program", below, refers to any + such program or work, and a "work based on the Program" means either the + Program or any derivative work under copyright law: that is to say, a + work containing the Program or a portion of it, either verbatim or with + modifications and/or translated into another language. (Hereinafter, + translation is included without limitation in the term "modification".) + Each licensee is addressed as "you". + + Activities other than copying, distribution and modification are not + covered by this License; they are outside its scope. The act of running + the Program is not restricted, and the output from the Program is covered + only if its contents constitute a work based on the Program (independent + of having been made by running the Program). Whether that is true depends + on what the Program does. + +1. You may copy and distribute verbatim copies of the Program's source code + as you receive it, in any medium, provided that you conspicuously and + appropriately publish on each copy an appropriate copyright notice and + disclaimer of warranty; keep intact all the notices that refer to this + License and to the absence of any warranty; and give any other recipients + of the Program a copy of this License along with the Program. + + You may charge a fee for the physical act of transferring a copy, and you + may at your option offer warranty protection in exchange for a fee. + +2. You may modify your copy or copies of the Program or any portion of it, + thus forming a work based on the Program, and copy and distribute such + modifications or work under the terms of Section 1 above, provided that + you also meet all of these conditions: + + * a) You must cause the modified files to carry prominent notices stating + that you changed the files and the date of any change. + + * b) You must cause any work that you distribute or publish, that in + whole or in part contains or is derived from the Program or any part + thereof, to be licensed as a whole at no charge to all third parties + under the terms of this License. + + * c) If the modified program normally reads commands interactively when + run, you must cause it, when started running for such interactive + use in the most ordinary way, to print or display an announcement + including an appropriate copyright notice and a notice that there is + no warranty (or else, saying that you provide a warranty) and that + users may redistribute the program under these conditions, and + telling the user how to view a copy of this License. (Exception: if + the Program itself is interactive but does not normally print such + an announcement, your work based on the Program is not required to + print an announcement.) + + These requirements apply to the modified work as a whole. If identifiable + sections of that work are not derived from the Program, and can be + reasonably considered independent and separate works in themselves, then + this License, and its terms, do not apply to those sections when you + distribute them as separate works. But when you distribute the same + sections as part of a whole which is a work based on the Program, the + distribution of the whole must be on the terms of this License, whose + permissions for other licensees extend to the entire whole, and thus to + each and every part regardless of who wrote it. + + Thus, it is not the intent of this section to claim rights or contest + your rights to work written entirely by you; rather, the intent is to + exercise the right to control the distribution of derivative or + collective works based on the Program. + + In addition, mere aggregation of another work not based on the Program + with the Program (or with a work based on the Program) on a volume of a + storage or distribution medium does not bring the other work under the + scope of this License. + +3. You may copy and distribute the Program (or a work based on it, under + Section 2) in object code or executable form under the terms of Sections + 1 and 2 above provided that you also do one of the following: + + * a) Accompany it with the complete corresponding machine-readable source + code, which must be distributed under the terms of Sections 1 and 2 + above on a medium customarily used for software interchange; or, + + * b) Accompany it with a written offer, valid for at least three years, + to give any third party, for a charge no more than your cost of + physically performing source distribution, a complete machine- + readable copy of the corresponding source code, to be distributed + under the terms of Sections 1 and 2 above on a medium customarily + used for software interchange; or, + + * c) Accompany it with the information you received as to the offer to + distribute corresponding source code. (This alternative is allowed + only for noncommercial distribution and only if you received the + program in object code or executable form with such an offer, in + accord with Subsection b above.) + + The source code for a work means the preferred form of the work for + making modifications to it. For an executable work, complete source code + means all the source code for all modules it contains, plus any + associated interface definition files, plus the scripts used to control + compilation and installation of the executable. However, as a special + exception, the source code distributed need not include anything that is + normally distributed (in either source or binary form) with the major + components (compiler, kernel, and so on) of the operating system on which + the executable runs, unless that component itself accompanies the + executable. + + If distribution of executable or object code is made by offering access + to copy from a designated place, then offering equivalent access to copy + the source code from the same place counts as distribution of the source + code, even though third parties are not compelled to copy the source + along with the object code. + +4. You may not copy, modify, sublicense, or distribute the Program except as + expressly provided under this License. Any attempt otherwise to copy, + modify, sublicense or distribute the Program is void, and will + automatically terminate your rights under this License. However, parties + who have received copies, or rights, from you under this License will not + have their licenses terminated so long as such parties remain in full + compliance. + +5. You are not required to accept this License, since you have not signed + it. However, nothing else grants you permission to modify or distribute + the Program or its derivative works. These actions are prohibited by law + if you do not accept this License. Therefore, by modifying or + distributing the Program (or any work based on the Program), you + indicate your acceptance of this License to do so, and all its terms and + conditions for copying, distributing or modifying the Program or works + based on it. + +6. Each time you redistribute the Program (or any work based on the + Program), the recipient automatically receives a license from the + original licensor to copy, distribute or modify the Program subject to + these terms and conditions. You may not impose any further restrictions + on the recipients' exercise of the rights granted herein. You are not + responsible for enforcing compliance by third parties to this License. + +7. If, as a consequence of a court judgment or allegation of patent + infringement or for any other reason (not limited to patent issues), + conditions are imposed on you (whether by court order, agreement or + otherwise) that contradict the conditions of this License, they do not + excuse you from the conditions of this License. If you cannot distribute + so as to satisfy simultaneously your obligations under this License and + any other pertinent obligations, then as a consequence you may not + distribute the Program at all. For example, if a patent license would + not permit royalty-free redistribution of the Program by all those who + receive copies directly or indirectly through you, then the only way you + could satisfy both it and this License would be to refrain entirely from + distribution of the Program. + + If any portion of this section is held invalid or unenforceable under any + particular circumstance, the balance of the section is intended to apply + and the section as a whole is intended to apply in other circumstances. + + It is not the purpose of this section to induce you to infringe any + patents or other property right claims or to contest validity of any + such claims; this section has the sole purpose of protecting the + integrity of the free software distribution system, which is implemented + by public license practices. Many people have made generous contributions + to the wide range of software distributed through that system in + reliance on consistent application of that system; it is up to the + author/donor to decide if he or she is willing to distribute software + through any other system and a licensee cannot impose that choice. + + This section is intended to make thoroughly clear what is believed to be + a consequence of the rest of this License. + +8. If the distribution and/or use of the Program is restricted in certain + countries either by patents or by copyrighted interfaces, the original + copyright holder who places the Program under this License may add an + explicit geographical distribution limitation excluding those countries, + so that distribution is permitted only in or among countries not thus + excluded. In such case, this License incorporates the limitation as if + written in the body of this License. + +9. The Free Software Foundation may publish revised and/or new versions of + the General Public License from time to time. Such new versions will be + similar in spirit to the present version, but may differ in detail to + address new problems or concerns. + + Each version is given a distinguishing version number. If the Program + specifies a version number of this License which applies to it and "any + later version", you have the option of following the terms and + conditions either of that version or of any later version published by + the Free Software Foundation. If the Program does not specify a version + number of this License, you may choose any version ever published by the + Free Software Foundation. + +10. If you wish to incorporate parts of the Program into other free programs + whose distribution conditions are different, write to the author to ask + for permission. For software which is copyrighted by the Free Software + Foundation, write to the Free Software Foundation; we sometimes make + exceptions for this. Our decision will be guided by the two goals of + preserving the free status of all derivatives of our free software and + of promoting the sharing and reuse of software generally. + + NO WARRANTY + +11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY + FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN + OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES + PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER + EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED + WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE + ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH + YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL + NECESSARY SERVICING, REPAIR OR CORRECTION. + +12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING + WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR + REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR + DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL + DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM + (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED + INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF + THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER OR + OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. + +END OF TERMS AND CONDITIONS + +How to Apply These Terms to Your New Programs + +If you develop a new program, and you want it to be of the greatest +possible use to the public, the best way to achieve this is to make it free +software which everyone can redistribute and change under these terms. + +To do so, attach the following notices to the program. It is safest to +attach them to the start of each source file to most effectively convey the +exclusion of warranty; and each file should have at least the "copyright" +line and a pointer to where the full notice is found. + +one line to give the program's name and an idea of what it does. +Copyright (C) yyyy name of author + +This program is free software; you can redistribute it and/or modify it +under the terms of the GNU General Public License as published by the Free +Software Foundation; either version 2 of the License, or (at your option) +any later version. + +This program is distributed in the hope that it will be useful, but WITHOUT +ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or +FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for +more details. + +You should have received a copy of the GNU General Public License along with +this program; if not, write to the Free Software Foundation, Inc., 59 +Temple Place - Suite 330, Boston, MA 02111-1307, USA. + +Also add information on how to contact you by electronic and paper mail. + +If the program is interactive, make it output a short notice like this when +it starts in an interactive mode: + +Gnomovision version 69, Copyright (C) year name of author Gnomovision comes +with ABSOLUTELY NO WARRANTY; for details type 'show w'. This is free +software, and you are welcome to redistribute it under certain conditions; +type 'show c' for details. + +The hypothetical commands 'show w' and 'show c' should show the appropriate +parts of the General Public License. Of course, the commands you use may be +called something other than 'show w' and 'show c'; they could even be +mouse-clicks or menu items--whatever suits your program. + +You should also get your employer (if you work as a programmer) or your +school, if any, to sign a "copyright disclaimer" for the program, if +necessary. Here is a sample; alter the names: + +Yoyodyne, Inc., hereby disclaims all copyright interest in the program +'Gnomovision' (which makes passes at compilers) written by James Hacker. + +signature of Ty Coon, 1 April 1989 +Ty Coon, President of Vice + +This General Public License does not permit incorporating your program into +proprietary programs. If your program is a subroutine library, you may +consider it more useful to permit linking proprietary applications with the +library. If this is what you want to do, use the GNU Library General Public +License instead of this License.
README +10 -0
@@ -0,0 +1,10 @@ +Linux* Base Driver README for the Intel(R) PRO/1000 Family of Adapters +=============================================================== + +This is a placeholder file, if you're reading this you got a pre-pre-alpha +copy of a driver tarball. + +This program is distributed in the hope that it will be useful, but WITHOUT +ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or +FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for +more details.
SUMS +27 -0
@@ -0,0 +1,27 @@ +37528 24 src/e1000_vf.c +40974 11 src/Makefile +05583 48 src/kcompat.h +43811 88 src/netdev.c +16366 73 src/e1000_defines.h +05472 11 src/kcompat.c +43671 59 src/e1000_phy.c +52675 61 src/e1000_mac.c +64115 3 src/e1000_nvm.h +54788 17 src/ethtool.c +31462 31 src/e1000_regs.h +29628 17 src/e1000_nvm.c +35406 15 src/e1000_hw.h +08899 5 src/e1000_mac.h +64697 4 src/e1000_osdep.h +41156 8 src/e1000_phy.h +61263 10 src/igbvf.h +44123 11 src/e1000_manage.c +09481 21 src/e1000_82575.h +56017 29 src/kcompat_ethtool.c +27223 4 src/e1000_manage.h +61211 10 src/param.c +12485 1 igbvf.7 +40802 1 README +21679 1 pci.updates +50366 12 igbvf.spec +39773 19 COPYING
igbvf.7 +11 -0
@@ -0,0 +1,11 @@ +.\" LICENSE +.\" +.\" This software program is released under the terms of a license agreement between you ('Licensee') and Intel. Do not use or load this software or any associated materials (collectively, the 'Software') until you have carefully read the full terms and conditions of the LICENSE located in this software package. By loading or using the Software, you agree to the terms of this Agreement. If you do not agree with the terms of this Agreement, do not install or use the Software. +.\" +.\" * Other names and brands may be claimed as the property of others. +.\" +.TH e1000e 1 "August 9999" + + +.SH NAME +e1000e \-This is a placeholder file, if you're reading this you got a pre-pre-alpha copy of a driver tarball.
igbvf.spec +471 -0
471 lines not on this page · raw message
pci.updates +9 -0
@@ -0,0 +1,9 @@ +# updates for the system pci.ids file +# +# IMPORTANT! Entries in this list must be sorted as they +# would appear in the system pci.ids file. Entries +# are sorted by ven, dev, subven, subdev +# (numerical order). +# +8086 Intel Corporation + 10ca 82576 Virtual Function Network Connection
src/Makefile +345 -0
345 lines not on this page · raw message
src/e1000_82575.h +478 -0
478 lines not on this page · raw message
src/e1000_defines.h +1460 -0
1,310 lines not on this page · raw message
src/e1000_hw.h +669 -0
669 lines not on this page · raw message
src/e1000_mac.c +2126 -0
2,126 lines not on this page · raw message
src/e1000_mac.h +90 -0
90 lines not on this page · raw message
src/e1000_manage.c +384 -0
384 lines not on this page · raw message
src/e1000_manage.h +82 -0
82 lines not on this page · raw message
src/e1000_nvm.c +681 -0
681 lines not on this page · raw message
src/e1000_nvm.h +55 -0
55 lines not on this page · raw message
src/e1000_osdep.h +121 -0
121 lines not on this page · raw message
src/e1000_phy.c +2170 -0
2,170 lines not on this page · raw message
src/e1000_phy.h +158 -0
158 lines not on this page · raw message
src/e1000_regs.h +477 -0
477 lines not on this page · raw message
src/e1000_vf.c +821 -0
821 lines not on this page · raw message
src/ethtool.c +620 -0
620 lines not on this page · raw message
src/igbvf.h +368 -0
368 lines not on this page · raw message
src/kcompat.c +417 -0
417 lines not on this page · raw message
src/kcompat.h +1668 -0
1,668 lines not on this page · raw message
src/kcompat_ethtool.c +1169 -0
1,169 lines not on this page · raw message
src/netdev.c +3372 -0
3,372 lines not on this page · raw message
src/param.c +351 -0
351 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linux-foundation.org/mailman/listinfo/virtualization
Part of the diff is not shown here. View the raw message.
Re: [SR-IOV driver example 0/3 resend] introduction
In reply to Yu ZhaoQuoted 11 lines · Yu Zhao
SR-IOV drivers of Intel 82576 NIC are available. There are two parts of the drivers: Physical Function driver and Virtual Function driver. The PF driver is based on the IGB driver and is used to control PF to allocate hardware specific resources and interface with the SR-IOV core. The VF driver is a new NIC driver that is same as the traditional PCI device driver. It works in both the host and the guest (Xen and KVM) environment.
These two drivers are testing versions and they are *only* intended to show how to use SR-IOV API.
Quoted 9 lines
Intel 82576 NIC specification can be found at:
http://download.intel.com/design/network/datashts/82576_Datasheet_v2p1.pdf[SR-IOV driver example 0/3 resend] introduction
[SR-IOV driver example 1/3 resend] PF driver: hardware specific operations [SR-IOV driver example 2/3 resend] PF driver: integrate with SR-IOV core [SR-IOV driver example 3/3 resend] VF driver: an independent PCI NIC driver --
First of all, we (e1000-devel) do support the SR-IOV API.
With that said, NAK on the driver changes. We were not involved in these changes and are currently working on a version of the drivers that will make them acceptable for kernel inclusion.
Signature · 2 lines
Cheers, Jeff
Re: [SR-IOV driver example 0/3 resend] introduction
In reply to Yu ZhaoQuoted 11 lines · Yu Zhao
SR-IOV drivers of Intel 82576 NIC are available. There are two parts of the drivers: Physical Function driver and Virtual Function driver. The PF driver is based on the IGB driver and is used to control PF to allocate hardware specific resources and interface with the SR-IOV core. The VF driver is a new NIC driver that is same as the traditional PCI device driver. It works in both the host and the guest (Xen and KVM) environment.
These two drivers are testing versions and they are *only* intended to show how to use SR-IOV API.
Quoted 9 lines
Intel 82576 NIC specification can be found at:
http://download.intel.com/design/network/datashts/82576_Datasheet_v2p1.pdf[SR-IOV driver example 0/3 resend] introduction
[SR-IOV driver example 1/3 resend] PF driver: hardware specific operations [SR-IOV driver example 2/3 resend] PF driver: integrate with SR-IOV core [SR-IOV driver example 3/3 resend] VF driver: an independent PCI NIC driver --
First of all, we (e1000-devel) do support the SR-IOV API.
With that said, NAK on the driver changes. We were not involved in these changes and are currently working on a version of the drivers that will make them acceptable for kernel inclusion.
Signature · 2 lines
Cheers, Jeff
cover introduction
In reply to Yu ZhaoSR-IOV drivers of Intel 82576 NIC are available. There are two parts of the drivers: Physical Function driver and Virtual Function driver. The PF driver is based on the IGB driver and is used to control PF to allocate hardware specific resources and interface with the SR-IOV core. The VF driver is a new NIC driver that is same as the traditional PCI device driver. It works in both the host and the guest (Xen and KVM) environment.
These two drivers are testing versions and they are *only* intended to show how to use SR-IOV API.
Intel 82576 NIC specification can be found at:
http://download.intel.com/design/network/datashts/82576_Datasheet_v2p1.pdf
[SR-IOV driver example 0/3 resend] introduction
[SR-IOV driver example 1/3 resend] PF driver: hardware specific operations [SR-IOV driver example 2/3 resend] PF driver: integrate with SR-IOV core [SR-IOV driver example 3/3 resend] VF driver: an independent PCI NIC driver
Quoted 11 lines · Yu Zhao
Greetings,
Following patches are intended to support SR-IOV capability in the Linux kernel. With these patches, people can turn a PCI device with the capability into multiple ones from software perspective, which will benefit KVM and achieve other purposes such as QoS, security, and etc.
The Physical Function and Virtual Function drivers using the SR-IOV APIs will come soon!
Quoted 6 lines
Major changes from v6 to v7:
1, remove boot-time resource rebalancing support. (Greg KH) 2, emit uevent upon the PF driver is loaded. (Greg KH)
3, put SR-IOV callback function into the 'pci_driver'. (Matthew Wilcox) 4, register SR-IOV service at the PF loading stage.
5, remove unnecessary APIs (pci_iov_enable/disable).
Thanks for your patience with this, Yu, I know it's been a long haul. :)
I applied 1-9 to my linux-next branch; and at least patch #10 needs a respin, so can you re-do 10-13 as a new patch set?
On re-reading the last thread, there was a lot of smoke, but very little fire afaict. The main questions I saw were:
1) do we need SR-IOV at all? why not just make each subsystem export devices to guests?
This is a bit of a red herring. Nothing about SR-IOV prevents us from making subsystems more v12n friendly. And since SR-IOV is a hardware feature supported by devices these days, we should make Linux support it.
2) should the PF/VF drivers be the same or not?
Again, the SR-IOV patchset and PCI spec don't dictate this. We're free to do what we want here.
3) should VF devices be represented by pci_dev structs? Yes. (This is an easy one :)
4) can VF devices be used on the host?
Yet again, SR-IOV doesn't dictate this. Developers can make PF/VF combo drivers or split them, and export the resulting devices however they want. Some subsystem work may be needed to make this efficient, but SR-IOV itself is agnostic about it.
So overall I didn't see many objections to the actual code in the last post, and the issues above certainly don't merit a NAK IMO...
Given a respin of 10-13 I think it's reasonable to merge this into 2.6.29, but I'd be much happier about it if we got some driver code along with it, so as not to have an unused interface sitting around for who knows how many releases. Is that reasonable? Do you know if any of the corresponding PF/VF driver bits are ready yet?
Thanks,
Signature · 1 line
Jesse Barnes, Intel Open Source Technology Center
Quoted 5 lines · Jesse Barnes
Given a respin of 10-13 I think it's reasonable to merge this into 2.6.29, but I'd be much happier about it if we got some driver code along with it, so as not to have an unused interface sitting around for who knows how many releases. Is that reasonable? Do you know if any of the corresponding PF/VF driver bits are ready yet?
Hi Jesse,
Yu Zhao has posted a patch set with subject "SR-IOV driver example" at November 26, which illustrated the usage of SR-IOV API in Intel 82576 VF/PF drivers;-)
--
Thanks,
Jike
Quoted 5 lines · Jesse Barnes
Given a respin of 10-13 I think it's reasonable to merge this into 2.6.29, but I'd be much happier about it if we got some driver code along with it, so as not to have an unused interface sitting around for who knows how many releases. Is that reasonable? Do you know if any of the corresponding PF/VF driver bits are ready yet?
Hi Jesse,
Yu Zhao has posted a patch set with subject "SR-IOV driver example" at November 26, which illustrated the usage of SR-IOV API in Intel 82576 VF/PF drivers;-)
--
Thanks,
Jike
Quoted 7 lines · Jike Song
Quoted 5 lines · Jesse Barnes
Given a respin of 10-13 I think it's reasonable to merge this into 2.6.29, but I'd be much happier about it if we got some driver code along with it, so as not to have an unused interface sitting around for who knows how many releases. Is that reasonable? Do you know if any of the corresponding PF/VF driver bits are ready yet?
Quoted 5 lines
Hi Jesse,
Yu Zhao has posted a patch set with subject "SR-IOV driver example" at November 26, which illustrated the usage of SR-IOV API in Intel 82576 VF/PF drivers;-)
Yes, but that driver was soundly rejected by the network driver maintainers, so I wouldn't go around showing that as your primary example of how to use this interface :)
The point is valid, I don't think these apis should go into the tree without a driver or some other code using them. Otherwise they make no sense at all to have in-tree.
thanks,
greg k-h
Quoted 8 lines · Greg KH
Quoted 6 lines · Jike Song
Quoted 1 line · Jesse Barnes
Given a respin of 10-13 I think it's reasonable to merge this into 2.6.29, but I'd be much happier about it if we got some driver code along with it, so as not to have an unused interface sitting around for who knows how many releases. Is that reasonable? Do you know if any of the corresponding PF/VF driver bits are ready yet?
Hi Jesse,
Yu Zhao has posted a patch set with subject "SR-IOV driver example" at November 26, which illustrated the usage of SR-IOV API in Intel 82576 VF/PF drivers;-)
Quoted 7 lines
Yes, but that driver was soundly rejected by the network driver maintainers, so I wouldn't go around showing that as your primary example of how to use this interface :)
The point is valid, I don't think these apis should go into the tree without a driver or some other code using them. Otherwise they make no sense at all to have in-tree.
I agree the point is valid, but on another hand this is a 'the chicken & the egg' problem -- if we don't have the SR-IOV base, people who are developing PF drivers can not get their changes in-tree. Maybe they are holding the patches and waiting on the infrastructure... :-)
Quoted 20 lines · Yu
Quoted 12 lines · Greg KH
Quoted 11 lines · Jike Song
Quoted 5 lines · Jesse Barnes
Given a respin of 10-13 I think it's reasonable to merge this into 2.6.29, but I'd be much happier about it if we got some driver code along with it, so as not to have an unused interface sitting around for who knows how many releases. Is that reasonable? Do you know if any of the corresponding PF/VF driver bits are ready yet?
Hi Jesse,
Yu Zhao has posted a patch set with subject "SR-IOV driver example" at November 26, which illustrated the usage of SR-IOV API in Intel 82576 VF/PF
drivers;-)Quoted 6 lines
Yes, but that driver was soundly rejected by the network driver maintainers, so I wouldn't go around showing that as your primary example of how to use this interface :)
The point is valid, I don't think these apis should go into the tree without a driver or some other code using them. Otherwise they make no sense at all to have in-tree.
Quoted 4 lines
I agree the point is valid, but on another hand this is a 'the chicken & the egg' problem -- if we don't have the SR-IOV base, people who are developing PF drivers can not get their changes in-tree. Maybe they are holding the patches and waiting on the infrastructure... :-)
Are they? They can both go in at the same time, like almost every other api addition to the kernel, right?
thanks,
greg k-h
Quoted 20 lines · Yu
Quoted 12 lines · Greg KH
Quoted 11 lines · Jike Song
Quoted 5 lines · Jesse Barnes
Given a respin of 10-13 I think it's reasonable to merge this into 2.6.29, but I'd be much happier about it if we got some driver code along with it, so as not to have an unused interface sitting around for who knows how many releases. Is that reasonable? Do you know if any of the corresponding PF/VF driver bits are ready yet?
Hi Jesse,
Yu Zhao has posted a patch set with subject "SR-IOV driver example" at November 26, which illustrated the usage of SR-IOV API in Intel 82576 VF/PF
drivers;-)Quoted 6 lines
Yes, but that driver was soundly rejected by the network driver maintainers, so I wouldn't go around showing that as your primary example of how to use this interface :)
The point is valid, I don't think these apis should go into the tree without a driver or some other code using them. Otherwise they make no sense at all to have in-tree.
Quoted 4 lines
I agree the point is valid, but on another hand this is a 'the chicken & the egg' problem -- if we don't have the SR-IOV base, people who are developing PF drivers can not get their changes in-tree. Maybe they are holding the patches and waiting on the infrastructure... :-)
Are they? They can both go in at the same time, like almost every other api addition to the kernel, right?
thanks,
greg k-h
Quoted 13 lines · Greg KH
Quoted 11 lines · Jike Song
Quoted 5 lines · Jesse Barnes
Given a respin of 10-13 I think it's reasonable to merge this into 2.6.29, but I'd be much happier about it if we got some driver code along with it, so as not to have an unused interface sitting around for who knows how many releases. Is that reasonable? Do you know if any of the corresponding PF/VF driver bits are ready yet?
Hi Jesse,
Yu Zhao has posted a patch set with subject "SR-IOV driver example" at November 26, which illustrated the usage of SR-IOV API in Intel 82576 VF/PF drivers;-)
Quoted 7 lines
Yes, but that driver was soundly rejected by the network driver maintainers, so I wouldn't go around showing that as your primary example of how to use this interface :)
The point is valid, I don't think these apis should go into the tree without a driver or some other code using them. Otherwise they make no sense at all to have in-tree.
I agree the point is valid, but on another hand this is a 'the chicken & the egg' problem -- if we don't have the SR-IOV base, people who are developing PF drivers can not get their changes in-tree. Maybe they are holding the patches and waiting on the infrastructure... :-)
As noted in the attached email to the netdev list, we (e1000_devel) will support the API.
- Greg Rose
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Greg KH
Sent: Tuesday, December 16, 2008 10:06 PM
To: Jike Song
Cc: Jesse Barnes; Zhao, Yu; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 0/13 v7] PCI: Linux kernel SR-IOV support
Quoted 7 lines · Jike Song
Quoted 5 lines · Jesse Barnes
Given a respin of 10-13 I think it's reasonable to merge this into 2.6.29, but I'd be much happier about it if we got some driver code along with it, so as not to have an unused interface sitting around for who knows how many releases. Is that reasonable? Do you know if any of the corresponding PF/VF driver bits are ready yet?
Quoted 5 lines
Hi Jesse,
Yu Zhao has posted a patch set with subject "SR-IOV driver example" at November 26, which illustrated the usage of SR-IOV API in Intel 82576 VF/PF drivers;-)
Yes, but that driver was soundly rejected by the network driver maintainers, so I wouldn't go around showing that as your primary example of how to use this interface :)
The point is valid, I don't think these apis should go into the tree without a driver or some other code using them. Otherwise they make no sense at all to have in-tree.
thanks,
greg k-h
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 11 lines · Yu Zhao
SR-IOV drivers of Intel 82576 NIC are available. There are two parts of the drivers: Physical Function driver and Virtual Function driver. The PF driver is based on the IGB driver and is used to control PF to allocate hardware specific resources and interface with the SR-IOV core. The VF driver is a new NIC driver that is same as the traditional PCI device driver. It works in both the host and the guest (Xen and KVM) environment.
These two drivers are testing versions and they are *only* intended to show how to use SR-IOV API.
Quoted 9 lines
Intel 82576 NIC specification can be found at:
http://download.intel.com/design/network/datashts/82576_Datasheet_v2p1.pdf[SR-IOV driver example 0/3 resend] introduction
[SR-IOV driver example 1/3 resend] PF driver: hardware specific operations [SR-IOV driver example 2/3 resend] PF driver: integrate with SR-IOV core [SR-IOV driver example 3/3 resend] VF driver: an independent PCI NIC driver --
First of all, we (e1000-devel) do support the SR-IOV API.
With that said, NAK on the driver changes. We were not involved in these changes and are currently working on a version of the drivers that will make them acceptable for kernel inclusion.
--
Cheers,
Jeff
A: No.
Q: Should I include quotations after my reply?
Quoted 1 line · Gregory V
As noted in the attached email to the netdev list, we (
e1000_devel) will support the API.
Great, will you have patches for the existing e1000 drivers soon to use it? Or will they be a while before they can be available?
As it is, the one posted user of this api is for a driver that has been rejected, so as there are no users of the api, I feel it should be deferrred until there is a user to make sure it all works and feels proper.
thanks,
greg k-h
A: No.
Q: Should I include quotations after my reply?
Quoted 1 line · Gregory V
As noted in the attached email to the netdev list, we (
e1000_devel) will support the API.
Great, will you have patches for the existing e1000 drivers soon to use it? Or will they be a while before they can be available?
As it is, the one posted user of this api is for a driver that has been rejected, so as there are no users of the api, I feel it should be deferrred until there is a user to make sure it all works and feels proper.
thanks,
greg k-h
Quoted 2 lines · Gregory V
As noted in the attached email to the netdev list, we (
e1000_devel) will support the API.
Do you think you'll have those changes ready for 2.6.29? Would merging core SR-IOV support now make that any more likely?
Thanks,
Jesse
-----Original Message-----
From: Jesse Barnes [mailto:[email protected]]
Quoted 2 lines · Gregory V
As noted in the attached email to the netdev list, we (
e1000_devel) will support the API.
Do you think you'll have those changes ready for 2.6.29? Would merging core SR-IOV support now make that any more likely?
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
I'm not sure about readiness for 2.6.29. I can tell you that as soon as I get a Xen Dom0 kernel with these API's included it will take me less than a day to convert over to them from the current drivers I have that are using an older API from back in August. The drivers are mostly functional, they have a few bugs. I could do some quick regression testing to make sure that the API changes haven't broken anything and then some bug fixes to get everything ready for release. Maybe two or three weeks for the major bugs. I'll be out over the Christmas holidays so that puts us into middle or late January if I got the Xen Dom0 kernel today. That seems unlikely but it gives you an idea of the time required.
- Greg
Quoted 9 lines · Gregory V
-----Original Message-----
From: Jesse Barnes [mailto:[email protected]]Quoted 3 lines · Gregory V
As noted in the attached email to the netdev list, we (
e1000_devel) will support the API.
Quoted 6 lines
Do you think you'll have those changes ready for 2.6.29? Would merging core SR-IOV support now make that any more likely?
I'm not sure about readiness for 2.6.29. I can tell you that as soon as I get a Xen Dom0 kernel with these API's included it will take me less than a day to convert over to them from the current drivers I have that are using an older API from back in August.
Which dom0 kernel are you using? Is it based on my pvops-based dom0 work?
J
Quoted 1 line · Jeremy Fitzhardinge
Which dom0 kernel are you using? Is it based on my pvops-based dom0 work?
The kernel I'm currently using is an ad-hoc patchwork of changes to the 2.6.18 Xen Dom0 kernel that was available back in August. The folks from OTC in Intel (Zhao Yu and his team) would be able to provide you more background on it as they did the work to enable MSI-X, SR-IOV and VT-d in that kernel so that my drivers would function. I don't see Zhao Yu on the distro list for this email so I'll add him.
- Greg
Quoted 1 line · Jeremy Fitzhardinge
Which dom0 kernel are you using? Is it based on my pvops-based dom0 work?
The kernel I'm currently using is an ad-hoc patchwork of changes to the 2.6.18 Xen Dom0 kernel that was available back in August. The folks from OTC in Intel (Zhao Yu and his team) would be able to provide you more background on it as they did the work to enable MSI-X, SR-IOV and VT-d in that kernel so that my drivers would function. I don't see Zhao Yu on the distro list for this email so I'll add him.
- Greg
Quoted 12 lines · Gregory V
-----Original Message-----
From: Jesse Barnes [mailto:[email protected]]Quoted 2 lines · Gregory V
As noted in the attached email to the netdev list, we (
e1000_devel) will support the API.Do you think you'll have those changes ready for 2.6.29? Would merging core SR-IOV support now make that any more likely?
Quoted 10 lines
I'm not sure about readiness for 2.6.29. I can tell you that as soon as I get a Xen Dom0 kernel with these API's included it will take me less than a day to convert over to them from the current drivers I have that are using an older API from back in August. The drivers are mostly functional, they have a few bugs. I could do some quick regression testing to make sure that the API changes haven't broken anything and then some bug fixes to get everything ready for release. Maybe two or three weeks for the major bugs.
I'll be out over the Christmas holidays so that puts us into middle or
late January if I got the Xen Dom0 kernel today. That seems unlikely but it gives you an idea of the time required.
Hm, that's not the answer I was hoping for. :) (Was looking for, "Yeah we just need this bits queued and we'll send an update for e1000 right away." :)
I really don't want the SR-IOV stuff to sit out another merge cycle though... Arg.
Signature · 1 line
Jesse Barnes, Intel Open Source Technology Center
Quoted 12 lines · Gregory V
-----Original Message-----
From: Jesse Barnes [mailto:[email protected]]Quoted 2 lines · Gregory V
As noted in the attached email to the netdev list, we (
e1000_devel) will support the API.Do you think you'll have those changes ready for 2.6.29? Would merging core SR-IOV support now make that any more likely?
Quoted 10 lines
I'm not sure about readiness for 2.6.29. I can tell you that as soon as I get a Xen Dom0 kernel with these API's included it will take me less than a day to convert over to them from the current drivers I have that are using an older API from back in August. The drivers are mostly functional, they have a few bugs. I could do some quick regression testing to make sure that the API changes haven't broken anything and then some bug fixes to get everything ready for release. Maybe two or three weeks for the major bugs.
I'll be out over the Christmas holidays so that puts us into middle or
late January if I got the Xen Dom0 kernel today. That seems unlikely but it gives you an idea of the time required.
Hm, that's not the answer I was hoping for. :) (Was looking for, "Yeah we just need this bits queued and we'll send an update for e1000 right away." :)
I really don't want the SR-IOV stuff to sit out another merge cycle though... Arg.
Signature · 1 line
Jesse Barnes, Intel Open Source Technology Center
Quoted 3 lines · Jesse Barnes
I really don't want the SR-IOV stuff to sit out another merge cycle though... Arg.
Why, is there some rush to get it in? As there is no in-kernel users of it, I don't see the problem with postponing it until someone actually needs it.
thanks,
greg k-h
Quoted 7 lines · Greg KH
Quoted 2 lines · Jesse Barnes
I really don't want the SR-IOV stuff to sit out another merge cycle though... Arg.
Why, is there some rush to get it in? As there is no in-kernel users of it, I don't see the problem with postponing it until someone actually needs it.
Well it *does* make development of SR-IOV drivers that much harder. As you know, out of tree development is a pain. OTOH if any changes end up being required, they can be done before the code is merged.
Anyway, hopefully we won't have to worry about it because some driver will come along soon that uses Yu's code. :) If not, Yu might have to maintain a separate git tree or something until the drivers are ready to be merged.
Signature · 1 line
Jesse Barnes, Intel Open Source Technology Center
Quoted 11 lines · Jesse Barnes
Quoted 6 lines · Greg KH
Quoted 2 lines · Jesse Barnes
I really don't want the SR-IOV stuff to sit out another merge cycle though... Arg.
Why, is there some rush to get it in? As there is no in-kernel users of it, I don't see the problem with postponing it until someone actually needs it.
Well it *does* make development of SR-IOV drivers that much harder. As you know, out of tree development is a pain. OTOH if any changes end up being required, they can be done before the code is merged.
Yes, people write to me asking for the SR-IOV patch or update everyday -- I guess they don't want to let their competitors know they are working on it so they can't bring their questions up on the mailing list.
And I personally also have dozen of other patches related to PCI and KVM subsystems which depend on the SR-IOV change.
Quoted 3 lines
Anyway, hopefully we won't have to worry about it because some driver will come along soon that uses Yu's code. :) If not, Yu might have to maintain a separate git tree or something until the drivers are ready to be merged.
Quoted 9 lines · Jesse Barnes
Quoted 6 lines · Greg KH
Quoted 2 lines · Jesse Barnes
I really don't want the SR-IOV stuff to sit out another merge cycle though... Arg.
Why, is there some rush to get it in? As there is no in-kernel users of it, I don't see the problem with postponing it until someone actually needs it.
Well it *does* make development of SR-IOV drivers that much harder. As you know, out of tree development is a pain. OTOH if any changes end up being required, they can be done before the code is merged.
Yes, people write to me asking for the SR-IOV patch or update everyday -- I guess they don't want to let their competitors know they are working on it so they can't bring their questions up on the mailing list.
And I personally also have dozen of other patches related to PCI and KVM subsystems which depend on the SR-IOV change.
Quoted 1 line
Anyway, hopefully we won't have to worry about it because some driver will come along soon that uses Yu's code. :) If not, Yu might have to maintain a separate git tree or something until the drivers are ready to be merged.
Quoted 7 lines · Greg KH
Quoted 2 lines · Jesse Barnes
I really don't want the SR-IOV stuff to sit out another merge cycle though... Arg.
Why, is there some rush to get it in? As there is no in-kernel users of it, I don't see the problem with postponing it until someone actually needs it.
Well it *does* make development of SR-IOV drivers that much harder. As you know, out of tree development is a pain. OTOH if any changes end up being required, they can be done before the code is merged.
Anyway, hopefully we won't have to worry about it because some driver will come along soon that uses Yu's code. :) If not, Yu might have to maintain a separate git tree or something until the drivers are ready to be merged.
Signature · 1 line
Jesse Barnes, Intel Open Source Technology Center
Quoted 3 lines · Jesse Barnes
I really don't want the SR-IOV stuff to sit out another merge cycle though... Arg.
Why, is there some rush to get it in? As there is no in-kernel users of it, I don't see the problem with postponing it until someone actually needs it.
thanks,
greg k-h
Quoted 7 lines · Jesse Barnes
Hm, that's not the answer I was hoping for. :) (Was looking for, "Yeah we just need this bits queued and we'll send an update for e1000 right away." :)
I really don't want the SR-IOV stuff to sit out another merge cycle though... Arg.
We will have drivers that support these API's posted to the lists within two or three days. These drivers are RFC only and not to be pushed upstream. More non-Xen testing needs to happen with the 82576 HW.
- Greg
Quoted 7 lines · Jesse Barnes
Hm, that's not the answer I was hoping for. :) (Was looking for, "Yeah we just need this bits queued and we'll send an update for e1000 right away." :)
I really don't want the SR-IOV stuff to sit out another merge cycle though... Arg.
We will have drivers that support these API's posted to the lists within two or three days. These drivers are RFC only and not to be pushed upstream. More non-Xen testing needs to happen with the 82576 HW.
- Greg
-----Original Message-----
From: Jesse Barnes [mailto:[email protected]]
Quoted 2 lines · Gregory V
As noted in the attached email to the netdev list, we (
e1000_devel) will support the API.
Do you think you'll have those changes ready for 2.6.29? Would merging core SR-IOV support now make that any more likely?
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
I'm not sure about readiness for 2.6.29. I can tell you that as soon as I get a Xen Dom0 kernel with these API's included it will take me less than a day to convert over to them from the current drivers I have that are using an older API from back in August. The drivers are mostly functional, they have a few bugs. I could do some quick regression testing to make sure that the API changes haven't broken anything and then some bug fixes to get everything ready for release. Maybe two or three weeks for the major bugs. I'll be out over the Christmas holidays so that puts us into middle or late January if I got the Xen Dom0 kernel today. That seems unlikely but it gives you an idea of the time required.
- Greg
Quoted 2 lines · Gregory V
As noted in the attached email to the netdev list, we (
e1000_devel) will support the API.
Do you think you'll have those changes ready for 2.6.29? Would merging core SR-IOV support now make that any more likely?
Thanks,
Jesse
As noted in the attached email to the netdev list, we (e1000_devel) will support the API.
- Greg Rose
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Greg KH
Sent: Tuesday, December 16, 2008 10:06 PM
To: Jike Song
Cc: Jesse Barnes; Zhao, Yu; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 0/13 v7] PCI: Linux kernel SR-IOV support
Quoted 7 lines · Jike Song
Quoted 5 lines · Jesse Barnes
Given a respin of 10-13 I think it's reasonable to merge this into 2.6.29, but I'd be much happier about it if we got some driver code along with it, so as not to have an unused interface sitting around for who knows how many releases. Is that reasonable? Do you know if any of the corresponding PF/VF driver bits are ready yet?
Quoted 5 lines
Hi Jesse,
Yu Zhao has posted a patch set with subject "SR-IOV driver example" at November 26, which illustrated the usage of SR-IOV API in Intel 82576 VF/PF drivers;-)
Yes, but that driver was soundly rejected by the network driver maintainers, so I wouldn't go around showing that as your primary example of how to use this interface :)
The point is valid, I don't think these apis should go into the tree without a driver or some other code using them. Otherwise they make no sense at all to have in-tree.
thanks,
greg k-h
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 11 lines · Yu Zhao
SR-IOV drivers of Intel 82576 NIC are available. There are two parts of the drivers: Physical Function driver and Virtual Function driver. The PF driver is based on the IGB driver and is used to control PF to allocate hardware specific resources and interface with the SR-IOV core. The VF driver is a new NIC driver that is same as the traditional PCI device driver. It works in both the host and the guest (Xen and KVM) environment.
These two drivers are testing versions and they are *only* intended to show how to use SR-IOV API.
Quoted 9 lines
Intel 82576 NIC specification can be found at:
http://download.intel.com/design/network/datashts/82576_Datasheet_v2p1.pdf[SR-IOV driver example 0/3 resend] introduction
[SR-IOV driver example 1/3 resend] PF driver: hardware specific operations [SR-IOV driver example 2/3 resend] PF driver: integrate with SR-IOV core [SR-IOV driver example 3/3 resend] VF driver: an independent PCI NIC driver --
First of all, we (e1000-devel) do support the SR-IOV API.
With that said, NAK on the driver changes. We were not involved in these changes and are currently working on a version of the drivers that will make them acceptable for kernel inclusion.
--
Cheers,
Jeff
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linux-foundation.org/mailman/listinfo/virtualization
Quoted 7 lines · Jike Song
Quoted 5 lines · Jesse Barnes
Given a respin of 10-13 I think it's reasonable to merge this into 2.6.29, but I'd be much happier about it if we got some driver code along with it, so as not to have an unused interface sitting around for who knows how many releases. Is that reasonable? Do you know if any of the corresponding PF/VF driver bits are ready yet?
Quoted 5 lines
Hi Jesse,
Yu Zhao has posted a patch set with subject "SR-IOV driver example" at November 26, which illustrated the usage of SR-IOV API in Intel 82576 VF/PF drivers;-)
Yes, but that driver was soundly rejected by the network driver maintainers, so I wouldn't go around showing that as your primary example of how to use this interface :)
The point is valid, I don't think these apis should go into the tree without a driver or some other code using them. Otherwise they make no sense at all to have in-tree.
thanks,
greg k-h
Quoted 68 lines
From: [email protected] [mailto:linux-pci- [email protected]] On Behalf Of Jesse Barnes
Sent: 16 December 2008 23:24
To: Yu Zhao
Cc: [email protected]; Chiang, Alexander; Helgaas, Bjorn; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 0/13 v7] PCI: Linux kernel SR-IOV supportQuoted 8 lines · Yu Zhao
Greetings,
Following patches are intended to support SR-IOV capability in the Linux kernel. With these patches, people can turn a PCI device with the capability into multiple ones from software perspective, which will benefit KVM and achieve other purposes such as QoS, security, and etc.
Quoted 7 lines
The Physical Function and Virtual Function drivers using the SR-IOV APIs will come soon!
Major changes from v6 to v7:
1, remove boot-time resource rebalancing support. (Greg KH) 2, emit uevent upon the PF driver is loaded. (Greg KH) 3, put SR-IOV callback function into the 'pci_driver'. (MatthewWilcox)
Quoted 2 lines
4, register SR-IOV service at the PF loading stage. 5, remove unnecessary APIs (pci_iov_enable/disable).
Thanks for your patience with this, Yu, I know it's been a long haul. :)
I applied 1-9 to my linux-next branch; and at least patch #10 needs a respin,
so can you re-do 10-13 as a new patch set?On re-reading the last thread, there was a lot of smoke, but very little fire
afaict. The main questions I saw were:1) do we need SR-IOV at all? why not just make each subsystem export devices to guests?
This is a bit of a red herring. Nothing about SR-IOV prevents us
from
making subsystems more v12n friendly. And since SR-IOV is a
hardware
feature supported by devices these days, we should make Linux
support it.2) should the PF/VF drivers be the same or not?
Again, the SR-IOV patchset and PCI spec don't dictate this. We're
free to
do what we want here.3) should VF devices be represented by
pci_devstructs? Yes. (This is an easy one :)4) can VF devices be used on the host?
Yet again, SR-IOV doesn't dictate this. Developers can make PF/VF
combo
drivers or split them, and export the resulting devices however
they want.
Some subsystem work may be needed to make this efficient, but SR-
IOV
itself is agnostic about it.
Quoted 3 lines
So overall I didn't see many objections to the actual code in the last post,
and the issues above certainly don't merit a NAK IMO...
I have two minor comments on this topic.
1) Currently the PF driver is called before the kernel initializes VFs and their resources, and the current API does not allow the PF driver to detect that easily if the allocation of the VFs and their resources has succeeded or not. It would be quite useful if the PF driver gets notified when the VFs have been created successfully as it might have to do further device-specific work *after* IOV has been enabled.
2) Configuration of SR-IOV: the current API allows to enable/disable VFs from userspace via SYSFS. At the moment I am not quite clear what exactly is supposed to control these capabilities. This could be Linux tools or, on a virtualized system, hypervisor control tools. One thing I am missing though is an in-kernel API for this which I think might be useful. After all the PF driver controls the device, and, for example, when a device error occurs (e.g. a hardware failure which only the PF driver will be able to detect, not Linux), then the PF driver might have to de-allocate all resources, shut down VFs and reset the device, or something like that. In that case the PF driver needs to have a way to notify the Linux SR-IOV code about this and initiate cleaning up of VFs and their resources. At the moment, this would have to go through userspace, I believe, and I think that is not an optimal solution. Yu, do you have an opinion on how this would be realized?
Anna
Quoted 8 lines · Anna
I have two minor comments on this topic.
1) Currently the PF driver is called before the kernel initializes VFs and their resources, and the current API does not allow the PF driver to detect that easily if the allocation of the VFs and their resources has succeeded or not. It would be quite useful if the PF driver gets notified when the VFs have been created successfully as it might have to do further device-specific work *after* IOV has been enabled.
You're thinking of after the VFs are created the VF drivers (which may or may not be part of the PF driver) may not be able to communicate back to the PF driver that something else needs to be done (I remember seeing this in the earlier thread, should have included it in my post, sorry)? I'm not sure if it makes sense to add an interface like that to the core until we have feel for what the PF/VF drivers are going to want... Or do you have something specific in mind right now? If/until we have something in the core, it seems like this could be done on a per PF/VF driver basis for now.
Quoted 7 lines
2) Configuration of SR-IOV: the current API allows to enable/disable VFs from userspace via SYSFS. At the moment I am not quite clear what exactly is supposed to control these capabilities. This could be Linux tools or, on a virtualized system, hypervisor control tools. One thing I am missing though is an in-kernel API for this which I think might be useful. After all the PF driver controls the device, and, for example, when a device error occurs (e.g. a hardware failure
Quoted 8 lines
which only the PF driver will be able to detect, not Linux), then the PF driver might have to de-allocate all resources, shut down VFs and reset the device, or something like that. In that case the PF driver needs to have a way to notify the Linux SR-IOV code about this and initiate cleaning up of VFs and their resources. At the moment, this would have to go through userspace, I believe, and I think that is not an optimal solution. Yu, do you have an opinion on how this would be realized?
That's a good point, Yu?
Signature · 1 line
Jesse Barnes, Intel Open Source Technology Center
Quoted 8 lines · Anna
I have two minor comments on this topic.
1) Currently the PF driver is called before the kernel initializes VFs and their resources, and the current API does not allow the PF driver to detect that easily if the allocation of the VFs and their resources has succeeded or not. It would be quite useful if the PF driver gets notified when the VFs have been created successfully as it might have to do further device-specific work *after* IOV has been enabled.
You're thinking of after the VFs are created the VF drivers (which may or may not be part of the PF driver) may not be able to communicate back to the PF driver that something else needs to be done (I remember seeing this in the earlier thread, should have included it in my post, sorry)? I'm not sure if it makes sense to add an interface like that to the core until we have feel for what the PF/VF drivers are going to want... Or do you have something specific in mind right now? If/until we have something in the core, it seems like this could be done on a per PF/VF driver basis for now.
Quoted 7 lines
2) Configuration of SR-IOV: the current API allows to enable/disable VFs from userspace via SYSFS. At the moment I am not quite clear what exactly is supposed to control these capabilities. This could be Linux tools or, on a virtualized system, hypervisor control tools. One thing I am missing though is an in-kernel API for this which I think might be useful. After all the PF driver controls the device, and, for example, when a device error occurs (e.g. a hardware failure
Quoted 8 lines
which only the PF driver will be able to detect, not Linux), then the PF driver might have to de-allocate all resources, shut down VFs and reset the device, or something like that. In that case the PF driver needs to have a way to notify the Linux SR-IOV code about this and initiate cleaning up of VFs and their resources. At the moment, this would have to go through userspace, I believe, and I think that is not an optimal solution. Yu, do you have an opinion on how this would be realized?
That's a good point, Yu?
Signature · 1 line
Jesse Barnes, Intel Open Source Technology Center
Quoted 8 lines · Anna
I have two minor comments on this topic.
1) Currently the PF driver is called before the kernel initializes VFs and their resources, and the current API does not allow the PF driver to detect that easily if the allocation of the VFs and their resources has succeeded or not. It would be quite useful if the PF driver gets notified when the VFs have been created successfully as it might have to do further device-specific work *after* IOV has been enabled.
If the VF allocation fails in the PCI layer, then the SR-IOV core will invokes the callback again to notify the PF driver with zero VF count. The PF driver does not have to concern about this even the PCI layer code fails (and actually it's very rare).
And I'm not sure why the PF driver wants to do further work *after* the VF is allocated. Does this mean PF driver have to set up some internal resources related to SR-IOV/VF? If yes, I suggest the PF driver do it before VF allocation. The design philosophy of SR-IOV/VF is that VF is treated as hot-plug device, which means it should be immediately usable by VF driver (e.g. VF driver is pre-loaded) after it appears in the PCI subsystem. If that is not the purpose, then PF driver should handle it not depending on the SR-IOV, right?
If you could elaborate your SR-IOV PF/VF h/w specific requirement, it would be help for me to answer this question :-)
Quoted 4 lines
2) Configuration of SR-IOV: the current API allows to enable/disable VFs from userspace via SYSFS. At the moment I am not quite clear what exactly is supposed to control these capabilities. This could be Linux tools or, on a virtualized system, hypervisor control tools.
This depends on user application, you know, which depends on the usage environment (i.e. native, KVM or Xen).
Quoted 11 lines
One thing I am missing though is an in-kernel API for this which I think might be useful. After all the PF driver controls the device, and, for example, when a device error occurs (e.g. a hardware failure which only the PF driver will be able to detect, not Linux), then the PF driver might have to de-allocate all resources, shut down VFs and reset the device, or something like that. In that case the PF driver needs to have a way to notify the Linux SR-IOV code about this and initiate cleaning up of VFs and their resources. At the moment, this would have to go through userspace, I believe, and I think that is not an optimal solution. Yu, do you have an opinion on how this would be realized?
Yes, the PF driver can use pci_iov_unregister to disable SR-IOV in case the fatal error occurs. This function also sends notification to user level through 'uevent' so user application can aware the change.
Thanks,
Yu
Quoted 8 lines · Anna
I have two minor comments on this topic.
1) Currently the PF driver is called before the kernel initializes VFs and their resources, and the current API does not allow the PF driver to detect that easily if the allocation of the VFs and their resources has succeeded or not. It would be quite useful if the PF driver gets notified when the VFs have been created successfully as it might have to do further device-specific work *after* IOV has been enabled.
If the VF allocation fails in the PCI layer, then the SR-IOV core will invokes the callback again to notify the PF driver with zero VF count. The PF driver does not have to concern about this even the PCI layer code fails (and actually it's very rare).
And I'm not sure why the PF driver wants to do further work *after* the VF is allocated. Does this mean PF driver have to set up some internal resources related to SR-IOV/VF? If yes, I suggest the PF driver do it before VF allocation. The design philosophy of SR-IOV/VF is that VF is treated as hot-plug device, which means it should be immediately usable by VF driver (e.g. VF driver is pre-loaded) after it appears in the PCI subsystem. If that is not the purpose, then PF driver should handle it not depending on the SR-IOV, right?
If you could elaborate your SR-IOV PF/VF h/w specific requirement, it would be help for me to answer this question :-)
Quoted 4 lines
2) Configuration of SR-IOV: the current API allows to enable/disable VFs from userspace via SYSFS. At the moment I am not quite clear what exactly is supposed to control these capabilities. This could be Linux tools or, on a virtualized system, hypervisor control tools.
This depends on user application, you know, which depends on the usage environment (i.e. native, KVM or Xen).
Quoted 11 lines
One thing I am missing though is an in-kernel API for this which I think might be useful. After all the PF driver controls the device, and, for example, when a device error occurs (e.g. a hardware failure which only the PF driver will be able to detect, not Linux), then the PF driver might have to de-allocate all resources, shut down VFs and reset the device, or something like that. In that case the PF driver needs to have a way to notify the Linux SR-IOV code about this and initiate cleaning up of VFs and their resources. At the moment, this would have to go through userspace, I believe, and I think that is not an optimal solution. Yu, do you have an opinion on how this would be realized?
Yes, the PF driver can use pci_iov_unregister to disable SR-IOV in case the fatal error occurs. This function also sends notification to user level through 'uevent' so user application can aware the change.
Thanks,
Yu
Fischer, Anna
Acked-byQuoted 22 lines
From: Zhao, Yu [mailto:[email protected]]
Sent: 18 December 2008 02:14
To: Fischer, Anna
Cc: Jesse Barnes; [email protected]; Chiang, Alexander; Helgaas, Bjorn; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; linux- [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 0/13 v7] PCI: Linux kernel SR-IOV supportQuoted 3 lines · Anna
I have two minor comments on this topic.
1) Currently the PF driver is called before the kernel initializes
VFs and
Quoted 5 lines
their resources, and the current API does not allow the PF driver to detect that easily if the allocation of the VFs and their resources has succeeded or not. It would be quite useful if the PF driver gets notified when the VFs have been created successfully as it might have to do further device-specific work *after* IOV has been enabled.
Quoted 4 lines
If the VF allocation fails in the PCI layer, then the SR-IOV core will invokes the callback again to notify the PF driver with zero VF count. The PF driver does not have to concern about this even the PCI layer code fails (and actually it's very rare).
Yes, this is good.
Quoted 8 lines
And I'm not sure why the PF driver wants to do further work *after* the VF is allocated. Does this mean PF driver have to set up some internal resources related to SR-IOV/VF? If yes, I suggest the PF driver do it before VF allocation. The design philosophy of SR-IOV/VF is that VF is treated as hot-plug device, which means it should be immediately usable by VF driver (e.g. VF driver is pre-loaded) after it appears in the PCI subsystem. If that is not the purpose, then PF driver should handle it not depending on the SR-IOV, right?
Yes, you are right. In fact I was assuming in this case that the PF driver might have to allocate VF specific resources before a PF <-> VF communication can be established but this can be done before the VF PCI device appears, so I was wrong with this. The current API is sufficient to handle all of this, so I am withdrawing my concern here ;-)
Quoted 20 lines
If you could elaborate your SR-IOV PF/VF h/w specific requirement, it would be help for me to answer this question :-)
Quoted 4 lines
2) Configuration of SR-IOV: the current API allows to enable/disable VFs from userspace via SYSFS. At the moment I am not quite clear what exactly is supposed to control these capabilities. This could be Linux tools or, on a virtualized system, hypervisor control tools.
This depends on user application, you know, which depends on the usage environment (i.e. native, KVM or Xen).
Quoted 9 lines
One thing I am missing though is an in-kernel API for this which I think might be useful. After all the PF driver controls the device, and, for example, when a device error occurs (e.g. a hardware failure which only the PF driver will be able to detect, not Linux), then the PF driver might have to de-allocate all resources, shut down VFs and reset the device, or something like that. In that case the PF driver needs to have a way to notify the Linux SR-IOV code about this and initiate cleaning up of VFs and their resources. At the moment, this would have to go through userspace, I believe, and I think that is
Quoted 7 lines
not
Quoted 2 lines
an optimal solution. Yu, do you have an opinion on how this would be realized?
Yes, the PF driver can use
pci_iov_unregisterto disable SR-IOV in case the fatal error occurs. This function also sends notification to user level through 'uevent' so user application can aware the change.
If pci_iov_unregister is accessible for kernel drivers than this is in fact all we need. Thanks for the clarification.
I think the patchset looks very good.
Fischer, Anna
Acked-byQuoted 22 lines
From: Zhao, Yu [mailto:[email protected]]
Sent: 18 December 2008 02:14
To: Fischer, Anna
Cc: Jesse Barnes; [email protected]; Chiang, Alexander; Helgaas, Bjorn; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; linux- [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 0/13 v7] PCI: Linux kernel SR-IOV supportQuoted 3 lines · Anna
I have two minor comments on this topic.
1) Currently the PF driver is called before the kernel initializes
VFs and
Quoted 5 lines
their resources, and the current API does not allow the PF driver to detect that easily if the allocation of the VFs and their resources has succeeded or not. It would be quite useful if the PF driver gets notified when the VFs have been created successfully as it might have to do further device-specific work *after* IOV has been enabled.
Quoted 4 lines
If the VF allocation fails in the PCI layer, then the SR-IOV core will invokes the callback again to notify the PF driver with zero VF count. The PF driver does not have to concern about this even the PCI layer code fails (and actually it's very rare).
Yes, this is good.
Quoted 8 lines
And I'm not sure why the PF driver wants to do further work *after* the VF is allocated. Does this mean PF driver have to set up some internal resources related to SR-IOV/VF? If yes, I suggest the PF driver do it before VF allocation. The design philosophy of SR-IOV/VF is that VF is treated as hot-plug device, which means it should be immediately usable by VF driver (e.g. VF driver is pre-loaded) after it appears in the PCI subsystem. If that is not the purpose, then PF driver should handle it not depending on the SR-IOV, right?
Yes, you are right. In fact I was assuming in this case that the PF driver might have to allocate VF specific resources before a PF <-> VF communication can be established but this can be done before the VF PCI device appears, so I was wrong with this. The current API is sufficient to handle all of this, so I am withdrawing my concern here ;-)
Quoted 20 lines
If you could elaborate your SR-IOV PF/VF h/w specific requirement, it would be help for me to answer this question :-)
Quoted 4 lines
2) Configuration of SR-IOV: the current API allows to enable/disable VFs from userspace via SYSFS. At the moment I am not quite clear what exactly is supposed to control these capabilities. This could be Linux tools or, on a virtualized system, hypervisor control tools.
This depends on user application, you know, which depends on the usage environment (i.e. native, KVM or Xen).
Quoted 9 lines
One thing I am missing though is an in-kernel API for this which I think might be useful. After all the PF driver controls the device, and, for example, when a device error occurs (e.g. a hardware failure which only the PF driver will be able to detect, not Linux), then the PF driver might have to de-allocate all resources, shut down VFs and reset the device, or something like that. In that case the PF driver needs to have a way to notify the Linux SR-IOV code about this and initiate cleaning up of VFs and their resources. At the moment, this would have to go through userspace, I believe, and I think that is
Quoted 7 lines
not
Quoted 2 lines
an optimal solution. Yu, do you have an opinion on how this would be realized?
Yes, the PF driver can use
pci_iov_unregisterto disable SR-IOV in case the fatal error occurs. This function also sends notification to user level through 'uevent' so user application can aware the change.
If pci_iov_unregister is accessible for kernel drivers than this is in fact all we need. Thanks for the clarification.
I think the patchset looks very good.
Quoted 68 lines
From: [email protected] [mailto:linux-pci- [email protected]] On Behalf Of Jesse Barnes
Sent: 16 December 2008 23:24
To: Yu Zhao
Cc: [email protected]; Chiang, Alexander; Helgaas, Bjorn; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 0/13 v7] PCI: Linux kernel SR-IOV supportQuoted 8 lines · Yu Zhao
Greetings,
Following patches are intended to support SR-IOV capability in the Linux kernel. With these patches, people can turn a PCI device with the capability into multiple ones from software perspective, which will benefit KVM and achieve other purposes such as QoS, security, and etc.
Quoted 7 lines
The Physical Function and Virtual Function drivers using the SR-IOV APIs will come soon!
Major changes from v6 to v7:
1, remove boot-time resource rebalancing support. (Greg KH) 2, emit uevent upon the PF driver is loaded. (Greg KH) 3, put SR-IOV callback function into the 'pci_driver'. (MatthewWilcox)
Quoted 2 lines
4, register SR-IOV service at the PF loading stage. 5, remove unnecessary APIs (pci_iov_enable/disable).
Thanks for your patience with this, Yu, I know it's been a long haul. :)
I applied 1-9 to my linux-next branch; and at least patch #10 needs a respin,
so can you re-do 10-13 as a new patch set?On re-reading the last thread, there was a lot of smoke, but very little fire
afaict. The main questions I saw were:1) do we need SR-IOV at all? why not just make each subsystem export devices to guests?
This is a bit of a red herring. Nothing about SR-IOV prevents us
from
making subsystems more v12n friendly. And since SR-IOV is a
hardware
feature supported by devices these days, we should make Linux
support it.2) should the PF/VF drivers be the same or not?
Again, the SR-IOV patchset and PCI spec don't dictate this. We're
free to
do what we want here.3) should VF devices be represented by
pci_devstructs? Yes. (This is an easy one :)4) can VF devices be used on the host?
Yet again, SR-IOV doesn't dictate this. Developers can make PF/VF
combo
drivers or split them, and export the resulting devices however
they want.
Some subsystem work may be needed to make this efficient, but SR-
IOV
itself is agnostic about it.
Quoted 3 lines
So overall I didn't see many objections to the actual code in the last post,
and the issues above certainly don't merit a NAK IMO...
I have two minor comments on this topic.
1) Currently the PF driver is called before the kernel initializes VFs and their resources, and the current API does not allow the PF driver to detect that easily if the allocation of the VFs and their resources has succeeded or not. It would be quite useful if the PF driver gets notified when the VFs have been created successfully as it might have to do further device-specific work *after* IOV has been enabled.
2) Configuration of SR-IOV: the current API allows to enable/disable VFs from userspace via SYSFS. At the moment I am not quite clear what exactly is supposed to control these capabilities. This could be Linux tools or, on a virtualized system, hypervisor control tools. One thing I am missing though is an in-kernel API for this which I think might be useful. After all the PF driver controls the device, and, for example, when a device error occurs (e.g. a hardware failure which only the PF driver will be able to detect, not Linux), then the PF driver might have to de-allocate all resources, shut down VFs and reset the device, or something like that. In that case the PF driver needs to have a way to notify the Linux SR-IOV code about this and initiate cleaning up of VFs and their resources. At the moment, this would have to go through userspace, I believe, and I think that is not an optimal solution. Yu, do you have an opinion on how this would be realized?
Anna
Quoted 1 line · Jesse Barnes
I applied 1-9 to my linux-next branch; and at least patch #10 needs a respin,
I still object to #2. We should have the flexibility to have 'struct resource's that are not in this array in the pci_dev. I would like to see the SR-IOV resources _not_ in this array (and indeed, I'd like to see PCI bridges keep their producer resources somewhere other than in this array). I accept that there are still some problems with this, but patch #2 moves us further from being able to achieve this goal, not closer.
Signature · 4 lines
Matthew Wilcox Intel Open Source Technology Centre "Bill, look, we understand that you're interested in selling us this operating system, but compare it to ours. We can't possibly take such a retrograde step."
Quoted 1 line · Jesse Barnes
I applied 1-9 to my linux-next branch; and at least patch #10 needs a respin,
I still object to #2. We should have the flexibility to have 'struct resource's that are not in this array in the pci_dev. I would like to see the SR-IOV resources _not_ in this array (and indeed, I'd like to see PCI bridges keep their producer resources somewhere other than in this array). I accept that there are still some problems with this, but patch #2 moves us further from being able to achieve this goal, not closer.
Signature · 4 lines
Matthew Wilcox Intel Open Source Technology Centre "Bill, look, we understand that you're interested in selling us this operating system, but compare it to ours. We can't possibly take such a retrograde step."
Quoted 11 lines · Matthew Wilcox
Quoted 2 lines · Jesse Barnes
I applied 1-9 to my linux-next branch; and at least patch #10 needs a respin,
I still object to #2. We should have the flexibility to have 'struct resource's that are not in this array in the
pci_dev. I would like to see the SR-IOV resources _not_ in this array (and indeed, I'd like to see PCI bridges keep their producer resources somewhere other than in this array). I accept that there are still some problems with this, but patch #2 moves us further from being able to achieve this goal, not closer.
Yeah, I can see what you mean here... but on the other hand it makes the existing code a bit clearer (no extra args), and really it doesn't push us *that* much further from non-pci_dev tied resources. Any patches in that direction will just get a few lines bigger, that's all.
But I agree that eventually we may want to have non-pci_dev resource lists, especially if we start adding advanced host bridge drivers or something.
Signature · 1 line
Jesse Barnes, Intel Open Source Technology Center
Quoted 11 lines · Matthew Wilcox
Quoted 2 lines · Jesse Barnes
I applied 1-9 to my linux-next branch; and at least patch #10 needs a respin,
I still object to #2. We should have the flexibility to have 'struct resource's that are not in this array in the
pci_dev. I would like to see the SR-IOV resources _not_ in this array (and indeed, I'd like to see PCI bridges keep their producer resources somewhere other than in this array). I accept that there are still some problems with this, but patch #2 moves us further from being able to achieve this goal, not closer.
Yeah, I can see what you mean here... but on the other hand it makes the existing code a bit clearer (no extra args), and really it doesn't push us *that* much further from non-pci_dev tied resources. Any patches in that direction will just get a few lines bigger, that's all.
But I agree that eventually we may want to have non-pci_dev resource lists, especially if we start adding advanced host bridge drivers or something.
Signature · 1 line
Jesse Barnes, Intel Open Source Technology Center
Quoted 8 lines · Matthew Wilcox
Quoted 1 line · Jesse Barnes
I applied 1-9 to my linux-next branch; and at least patch #10 needs a respin,
I still object to #2. We should have the flexibility to have 'struct resource's that are not in this array in the
pci_dev. I would like to see the SR-IOV resources _not_ in this array (and indeed, I'd like to see PCI bridges keep their producer resources somewhere other than in this array). I accept that there are still some problems with this, but
I understand your concern, and agree that using the array as resource manager is not the best way. But for now it's not possible as you know. We need a better resource manager for PCI subsystem to manage the various resources (traditional, device specific, bus related), which is another independent work from SR-IOV change.
Quoted 2 lines
patch #2 moves us further from being able to achieve this goal, not closer.
The array is obvious straightforward and can be easily replaced with a more advanced resource manager in the future. So I don't think we going
further from or closer to the goal.
Thanks,
Yu
Quoted 8 lines · Matthew Wilcox
Quoted 1 line · Jesse Barnes
I applied 1-9 to my linux-next branch; and at least patch #10 needs a respin,
I still object to #2. We should have the flexibility to have 'struct resource's that are not in this array in the
pci_dev. I would like to see the SR-IOV resources _not_ in this array (and indeed, I'd like to see PCI bridges keep their producer resources somewhere other than in this array). I accept that there are still some problems with this, but
I understand your concern, and agree that using the array as resource manager is not the best way. But for now it's not possible as you know. We need a better resource manager for PCI subsystem to manage the various resources (traditional, device specific, bus related), which is another independent work from SR-IOV change.
Quoted 2 lines
patch #2 moves us further from being able to achieve this goal, not closer.
The array is obvious straightforward and can be easily replaced with a more advanced resource manager in the future. So I don't think we going
further from or closer to the goal.
Thanks,
Yu
Quoted 11 lines · Yu Zhao
Greetings,
Following patches are intended to support SR-IOV capability in the Linux kernel. With these patches, people can turn a PCI device with the capability into multiple ones from software perspective, which will benefit KVM and achieve other purposes such as QoS, security, and etc.
The Physical Function and Virtual Function drivers using the SR-IOV APIs will come soon!
Quoted 6 lines
Major changes from v6 to v7:
1, remove boot-time resource rebalancing support. (Greg KH) 2, emit uevent upon the PF driver is loaded. (Greg KH)
3, put SR-IOV callback function into the 'pci_driver'. (Matthew Wilcox) 4, register SR-IOV service at the PF loading stage.
5, remove unnecessary APIs (pci_iov_enable/disable).
Thanks for your patience with this, Yu, I know it's been a long haul. :)
I applied 1-9 to my linux-next branch; and at least patch #10 needs a respin, so can you re-do 10-13 as a new patch set?
On re-reading the last thread, there was a lot of smoke, but very little fire afaict. The main questions I saw were:
1) do we need SR-IOV at all? why not just make each subsystem export devices to guests?
This is a bit of a red herring. Nothing about SR-IOV prevents us from making subsystems more v12n friendly. And since SR-IOV is a hardware feature supported by devices these days, we should make Linux support it.
2) should the PF/VF drivers be the same or not?
Again, the SR-IOV patchset and PCI spec don't dictate this. We're free to do what we want here.
3) should VF devices be represented by pci_dev structs? Yes. (This is an easy one :)
4) can VF devices be used on the host?
Yet again, SR-IOV doesn't dictate this. Developers can make PF/VF combo drivers or split them, and export the resulting devices however they want. Some subsystem work may be needed to make this efficient, but SR-IOV itself is agnostic about it.
So overall I didn't see many objections to the actual code in the last post, and the issues above certainly don't merit a NAK IMO...
Given a respin of 10-13 I think it's reasonable to merge this into 2.6.29, but I'd be much happier about it if we got some driver code along with it, so as not to have an unused interface sitting around for who knows how many releases. Is that reasonable? Do you know if any of the corresponding PF/VF driver bits are ready yet?
Thanks,
Signature · 1 line
Jesse Barnes, Intel Open Source Technology Center