PCI: Linux kernel SR-IOV support
Subsystems (5)
v6 cover PCI: Linux kernel SR-IOV support
Greetings,
Following patches are intended to support SR-IOV capability in the Linux kernel. With these patches, people can turn a PCI device with the capability into multiple ones from software perspective, which will benefit KVM and achieve other purposes such as QoS, security, and etc.
Changes from v5 to v6:
1, update ABI document to include SR-IOV sysfs entries (Greg KH) 2, fix two coding style problems (Ingo Molnar)
---
[PATCH 1/16 v6] PCI: remove unnecessary arg of pci_update_resource() [PATCH 2/16 v6] PCI: define PCI resource names in an 'enum' [PATCH 3/16 v6] PCI: export __pci_read_base
[PATCH 4/16 v6] PCI: make pci_alloc_child_bus() be able to handle NULL bridge [PATCH 5/16 v6] PCI: add a wrapper for resource_alignment() [PATCH 6/16 v6] PCI: add a new function to map BAR offset
[PATCH 7/16 v6] PCI: cleanup pcibios_allocate_resources()
[PATCH 8/16 v6] PCI: add boot options to reassign resources [PATCH 9/16 v6] PCI: add boot option to align MMIO resources [PATCH 10/16 v6] PCI: cleanup pci_bus_add_devices()
[PATCH 11/16 v6] PCI: split a new function from pci_bus_add_devices() [PATCH 12/16 v6] PCI: support the SR-IOV capability
[PATCH 13/16 v6] PCI: reserve bus range for SR-IOV device
[PATCH 14/16 v6] PCI: document for SR-IOV user and developer [PATCH 15/16 v6] PCI: document the SR-IOV sysfs entries
[PATCH 16/16 v6] PCI: document the new PCI boot parameters
---
Single Root I/O Virtualization (SR-IOV) capability defined by PCI-SIG is intended to enable multiple system software to share PCI hardware resources. PCI device that supports this capability can be extended to one Physical Functions plus multiple Virtual Functions. Physical Function, which could be considered as the "real" PCI device, reflects the hardware instance and manages all physical resources. Virtual Functions are associated with a Physical Function and shares physical resources with the Physical Function.Software can control allocation of Virtual Functions via registers encapsulated in the capability structure.
SR-IOV specification can be found at
http://www.pcisig.com/members/downloads/specifications/iov/sr-iov1.0_11Sep07.pdf
Devices that support SR-IOV are available from following vendors:
http://download.intel.com/design/network/ProdBrf/320025.pdf
http://www.netxen.com/products/chipsolutions/NX3031.html
http://www.neterion.com/products/x3100.html
PATCH v6 1/16 PCI: remove unnecessary arg of pci_update_resource()
This cleanup removes unnecessary argument 'struct resource *res' in pci_update_resource(), so it takes same arguments as other companion functions (pci_assign_resource(), etc.).
--- drivers/pci/pci.c | 4 ++-- drivers/pci/setup-res.c | 7 ++++--- include/linux/pci.h | 2 +- 3 files changed, 7 insertions(+), 6 deletions(-)
drivers/pci/pci.c +2 -2
@@ -376,8 +376,8 @@ pci_restore_bars(struct pci_dev *dev)return ; } -for (i =0 ; i < numres; i ++) -pci_update_resource (dev, &dev->resource[i], i); +for (i =0 ; i < numres; i++) +pci_update_resource (dev, i); }static struct pci_platform_pm_ops *pci_platform_pm;
drivers/pci/setup-res.c +4 -3
@@ -26,11 +26,12 @@#include "pci.h" -void pci_update_resource (struct pci_dev *dev,struct resource *res,int resno) +void pci_update_resource (struct pci_dev *dev,int resno) {struct pci_bus_region region;u32 new, check, mask;int reg; +struct resource *res = dev->resource + resno;/* * Ignore resources for unimplemented BARs and unused resource slots
@@ -162,7 +163,7 @@ int pci_assign_resource(struct pci_dev *dev, int resno) }else { res->flags &= ~IORESOURCE_STARTALIGN ;if (resno <PCI_BRIDGE_RESOURCES ) -pci_update_resource (dev, res, resno); +pci_update_resource (dev, resno); }return ret;
@@ -197,7 +198,7 @@ int pci_assign_resource_fixed(struct pci_dev *dev, int resno)dev_err (&dev->dev,"BAR %d: can't allocate %s resource %pR\n" , resno, res->flags &IORESOURCE_IO ?"I/O" :"mem" , res); }else if (resno <PCI_BRIDGE_RESOURCES ) { -pci_update_resource (dev, res, resno); +pci_update_resource (dev, resno); }return ret;
include/linux/pci.h +1 -1
@@ -626,7 +626,7 @@ int pcix_get_mmrbc(struct pci_dev *dev);int pcix_set_mmrbc (struct pci_dev *dev,int mmrbc);int pcie_get_readrq (struct pci_dev *dev);int pcie_set_readrq (struct pci_dev *dev,int rq); -void pci_update_resource (struct pci_dev *dev,struct resource *res,int resno); +void pci_update_resource (struct pci_dev *dev,int resno);int __must_check pci_assign_resource (struct pci_dev *dev,int i);int pci_select_bars (struct pci_dev *dev,unsigned long flags);
Signature · 1 line
1.5.6.4
PATCH v6 1/16 PCI: remove unnecessary arg of pci_update_resource()
In reply to Yu ZhaoThis cleanup removes unnecessary argument 'struct resource *res' in pci_update_resource(), so it takes same arguments as other companion functions (pci_assign_resource(), etc.).
--- drivers/pci/pci.c | 4 ++-- drivers/pci/setup-res.c | 7 ++++--- include/linux/pci.h | 2 +- 3 files changed, 7 insertions(+), 6 deletions(-)
drivers/pci/pci.c +2 -2
@@ -376,8 +376,8 @@ pci_restore_bars(struct pci_dev *dev)return ; } -for (i =0 ; i < numres; i ++) -pci_update_resource (dev, &dev->resource[i], i); +for (i =0 ; i < numres; i++) +pci_update_resource (dev, i); }static struct pci_platform_pm_ops *pci_platform_pm;
drivers/pci/setup-res.c +4 -3
@@ -26,11 +26,12 @@#include "pci.h" -void pci_update_resource (struct pci_dev *dev,struct resource *res,int resno) +void pci_update_resource (struct pci_dev *dev,int resno) {struct pci_bus_region region;u32 new, check, mask;int reg; +struct resource *res = dev->resource + resno;/* * Ignore resources for unimplemented BARs and unused resource slots
@@ -162,7 +163,7 @@ int pci_assign_resource(struct pci_dev *dev, int resno) }else { res->flags &= ~IORESOURCE_STARTALIGN ;if (resno <PCI_BRIDGE_RESOURCES ) -pci_update_resource (dev, res, resno); +pci_update_resource (dev, resno); }return ret;
@@ -197,7 +198,7 @@ int pci_assign_resource_fixed(struct pci_dev *dev, int resno)dev_err (&dev->dev,"BAR %d: can't allocate %s resource %pR\n" , resno, res->flags &IORESOURCE_IO ?"I/O" :"mem" , res); }else if (resno <PCI_BRIDGE_RESOURCES ) { -pci_update_resource (dev, res, resno); +pci_update_resource (dev, resno); }return ret;
include/linux/pci.h +1 -1
@@ -626,7 +626,7 @@ int pcix_get_mmrbc(struct pci_dev *dev);int pcix_set_mmrbc (struct pci_dev *dev,int mmrbc);int pcie_get_readrq (struct pci_dev *dev);int pcie_set_readrq (struct pci_dev *dev,int rq); -void pci_update_resource (struct pci_dev *dev,struct resource *res,int resno); +void pci_update_resource (struct pci_dev *dev,int resno);int __must_check pci_assign_resource (struct pci_dev *dev,int i);int pci_select_bars (struct pci_dev *dev,unsigned long flags);
Signature · 1 line
1.5.6.4
PATCH v6 2/16 PCI: define PCI resource names in an 'enum'
In reply to Yu ZhaoThis patch moves all definitions of the PCI resource names to an 'enum', and also replaces some hard-coded resource variables with symbol names. This change eases introduction of device specific resources.
--- drivers/pci/pci-sysfs.c | 4 +++- drivers/pci/pci.c | 19 ++----------------- drivers/pci/probe.c | 2 +- drivers/pci/proc.c | 7 ++++--- include/linux/pci.h | 37 ++++++++++++++++++++++++------------- 5 files changed, 34 insertions(+), 35 deletions(-)
drivers/pci/pci-sysfs.c +3 -1
@@ -101,11 +101,13 @@ resource_show(struct device * dev, struct device_attribute *attr, char * buf)struct pci_dev * pci_dev =to_pci_dev (dev);char * str = buf;int i; -int max =7 ; +int max;resource_size_t start, end;if (pci_dev->subordinate) max =DEVICE_COUNT_RESOURCE ; +else + max =PCI_BRIDGE_RESOURCES ;for (i =0 ; i < max; i++) {struct resource *res = &pci_dev->resource[i];
drivers/pci/pci.c +2 -17
@@ -359,24 +359,9 @@ pci_find_parent_resource(const struct pci_dev *dev, struct resource *res)static void pci_restore_bars (struct pci_dev *dev) { -int i, numres; - -switch (dev->hdr_type) { -case PCI_HEADER_TYPE_NORMAL : - numres =6 ; -break ; -case PCI_HEADER_TYPE_BRIDGE : - numres =2 ; -break ; -case PCI_HEADER_TYPE_CARDBUS : - numres =1 ; -break ; -default : -/* Should never get here, but just in case... */ -return ; - } +int i; -for (i =0 ; i < numres; i++) +for (i =0 ; i <PCI_BRIDGE_RESOURCES ; i++)pci_update_resource (dev, i); }
drivers/pci/probe.c +1 -1
@@ -426,7 +426,7 @@ static struct pci_bus *pci_alloc_child_bus(struct pci_bus *parent, child->subordinate =0xff ;/* Set up default resource pointers and names.. */ -for (i =0 ; i <4 ; i++) { +for (i =0 ; i <PCI_BRIDGE_RES_NUM ; i++) { child->resource[i] = &bridge->resource[PCI_BRIDGE_RESOURCES +i]; child->resource[i]->name = child->name; }
drivers/pci/proc.c +4 -3
@@ -352,15 +352,16 @@ static int show_device(struct seq_file *m, void *v) dev->vendor, dev->device, dev->irq); -/* Here should be 7 and not PCI_NUM_RESOURCES as we need to preserve compatibility */ -for (i=0 ; i<7 ; i++) { + +/* only print standard and ROM resources to preserve compatibility */ +for (i =0 ; i <=PCI_ROM_RESOURCE ; i++) {resource_size_t start, end;pci_resource_to_user (dev, i, &dev->resource[i], &start, &end);seq_printf (m,"\t%16llx" , (unsigned long long )(start | (dev->resource[i].flags &PCI_REGION_FLAG_MASK ))); } -for (i=0 ; i<7 ; i++) { +for (i =0 ; i <=PCI_ROM_RESOURCE ; i++) {resource_size_t start, end;pci_resource_to_user (dev, i, &dev->resource[i], &start, &end); seq_printf(m, "\t%16llx",
include/linux/pci.h +24 -13
@@ -76,7 +76,30 @@ enum pci_mmap_state {#define PCI_DMA_FROMDEVICE 2#define PCI_DMA_NONE 3 -#define DEVICE_COUNT_RESOURCE 12 +/* +* For PCI devices, the region numbers are assigned this way: +*/ +enum { +/* #0-5: standard PCI regions */ +PCI_STD_RESOURCES , +PCI_STD_RESOURCES_END =5 , + +/* #6: expansion ROM */ +PCI_ROM_RESOURCE , + +/* address space assigned to buses behind the bridge */ +#ifndef PCI_BRIDGE_RES_NUM +#define PCI_BRIDGE_RES_NUM 4 +#endif +PCI_BRIDGE_RESOURCES , +PCI_BRIDGE_RES_END =PCI_BRIDGE_RESOURCES +PCI_BRIDGE_RES_NUM -1 , + +/* total resources associated with a PCI device */ +PCI_NUM_RESOURCES , + +/* preserve this for compatibility */ +DEVICE_COUNT_RESOURCE +};typedef int __bitwise pci_power_t ;
@@ -262,18 +285,6 @@ static inline void pci_add_saved_cap(struct pci_dev *pci_dev,hlist_add_head (&new_cap->next, &pci_dev->saved_cap_space); } -/* -* For PCI devices, the region numbers are assigned this way: -* -* 0-5 standard PCI regions -* 6 expansion ROM -* 7-10 bridges: address space assigned to buses behind the bridge -*/ - -#define PCI_ROM_RESOURCE 6 -#define PCI_BRIDGE_RESOURCES 7 -#define PCI_NUM_RESOURCES 11 -#ifndef PCI_BUS_NUM_RESOURCES #define PCI_BUS_NUM_RESOURCES 16#endif
Signature · 1 line
1.5.6.4
PATCH v6 2/16 PCI: define PCI resource names in an 'enum'
In reply to Yu ZhaoThis patch moves all definitions of the PCI resource names to an 'enum', and also replaces some hard-coded resource variables with symbol names. This change eases introduction of device specific resources.
--- drivers/pci/pci-sysfs.c | 4 +++- drivers/pci/pci.c | 19 ++----------------- drivers/pci/probe.c | 2 +- drivers/pci/proc.c | 7 ++++--- include/linux/pci.h | 37 ++++++++++++++++++++++++------------- 5 files changed, 34 insertions(+), 35 deletions(-)
drivers/pci/pci-sysfs.c +3 -1
@@ -101,11 +101,13 @@ resource_show(struct device * dev, struct device_attribute *attr, char * buf)struct pci_dev * pci_dev =to_pci_dev (dev);char * str = buf;int i; -int max =7 ; +int max;resource_size_t start, end;if (pci_dev->subordinate) max =DEVICE_COUNT_RESOURCE ; +else + max =PCI_BRIDGE_RESOURCES ;for (i =0 ; i < max; i++) {struct resource *res = &pci_dev->resource[i];
drivers/pci/pci.c +2 -17
@@ -359,24 +359,9 @@ pci_find_parent_resource(const struct pci_dev *dev, struct resource *res)static void pci_restore_bars (struct pci_dev *dev) { -int i, numres; - -switch (dev->hdr_type) { -case PCI_HEADER_TYPE_NORMAL : - numres =6 ; -break ; -case PCI_HEADER_TYPE_BRIDGE : - numres =2 ; -break ; -case PCI_HEADER_TYPE_CARDBUS : - numres =1 ; -break ; -default : -/* Should never get here, but just in case... */ -return ; - } +int i; -for (i =0 ; i < numres; i++) +for (i =0 ; i <PCI_BRIDGE_RESOURCES ; i++)pci_update_resource (dev, i); }
drivers/pci/probe.c +1 -1
@@ -426,7 +426,7 @@ static struct pci_bus *pci_alloc_child_bus(struct pci_bus *parent, child->subordinate =0xff ;/* Set up default resource pointers and names.. */ -for (i =0 ; i <4 ; i++) { +for (i =0 ; i <PCI_BRIDGE_RES_NUM ; i++) { child->resource[i] = &bridge->resource[PCI_BRIDGE_RESOURCES +i]; child->resource[i]->name = child->name; }
drivers/pci/proc.c +4 -3
@@ -352,15 +352,16 @@ static int show_device(struct seq_file *m, void *v) dev->vendor, dev->device, dev->irq); -/* Here should be 7 and not PCI_NUM_RESOURCES as we need to preserve compatibility */ -for (i=0 ; i<7 ; i++) { + +/* only print standard and ROM resources to preserve compatibility */ +for (i =0 ; i <=PCI_ROM_RESOURCE ; i++) {resource_size_t start, end;pci_resource_to_user (dev, i, &dev->resource[i], &start, &end);seq_printf (m,"\t%16llx" , (unsigned long long )(start | (dev->resource[i].flags &PCI_REGION_FLAG_MASK ))); } -for (i=0 ; i<7 ; i++) { +for (i =0 ; i <=PCI_ROM_RESOURCE ; i++) {resource_size_t start, end;pci_resource_to_user (dev, i, &dev->resource[i], &start, &end); seq_printf(m, "\t%16llx",
include/linux/pci.h +24 -13
@@ -76,7 +76,30 @@ enum pci_mmap_state {#define PCI_DMA_FROMDEVICE 2#define PCI_DMA_NONE 3 -#define DEVICE_COUNT_RESOURCE 12 +/* +* For PCI devices, the region numbers are assigned this way: +*/ +enum { +/* #0-5: standard PCI regions */ +PCI_STD_RESOURCES , +PCI_STD_RESOURCES_END =5 , + +/* #6: expansion ROM */ +PCI_ROM_RESOURCE , + +/* address space assigned to buses behind the bridge */ +#ifndef PCI_BRIDGE_RES_NUM +#define PCI_BRIDGE_RES_NUM 4 +#endif +PCI_BRIDGE_RESOURCES , +PCI_BRIDGE_RES_END =PCI_BRIDGE_RESOURCES +PCI_BRIDGE_RES_NUM -1 , + +/* total resources associated with a PCI device */ +PCI_NUM_RESOURCES , + +/* preserve this for compatibility */ +DEVICE_COUNT_RESOURCE +};typedef int __bitwise pci_power_t ;
@@ -262,18 +285,6 @@ static inline void pci_add_saved_cap(struct pci_dev *pci_dev,hlist_add_head (&new_cap->next, &pci_dev->saved_cap_space); } -/* -* For PCI devices, the region numbers are assigned this way: -* -* 0-5 standard PCI regions -* 6 expansion ROM -* 7-10 bridges: address space assigned to buses behind the bridge -*/ - -#define PCI_ROM_RESOURCE 6 -#define PCI_BRIDGE_RESOURCES 7 -#define PCI_NUM_RESOURCES 11 -#ifndef PCI_BUS_NUM_RESOURCES #define PCI_BUS_NUM_RESOURCES 16#endif
Signature · 1 line
1.5.6.4
Re: [PATCH 2/16 v6] PCI: define PCI resource names in an 'enum'
Quoted 3 lines · Yu Zhao
This patch moves all definitions of the PCI resource names to an 'enum', and also replaces some hard-coded resource variables with symbol names. This change eases introduction of device specific resources.
Thanks for removing a bunch of magic numbers from the code.
Quoted 15 lines
static void
pci_restore_bars(struct pci_dev *dev)
{drivers/pci/pci.c +0
-12-int i, numres; - -switch (dev->hdr_type) { -case PCI_HEADER_TYPE_NORMAL : - numres =6 ; -break ; -case PCI_HEADER_TYPE_BRIDGE : - numres =2 ; -break ; -case PCI_HEADER_TYPE_CARDBUS : - numres =1 ; -break ;
drivers/pci/pci.c pci_find_parent_resource(const struct pci_dev *dev, struct resource *res) · line 366
Quoted 10 lines
drivers/pci/pci.c +2
-5-default : -/* Should never get here, but just in case... */ -return ; - } +int i; -for (i =0 ; i < numres; i++) +for (i =0 ; i <PCI_BRIDGE_RESOURCES ; i++)pci_update_resource (dev, i); }
The behavior of this function used to depend on dev->hdr_type. Now we don't look at hdr_type at all, so we do the same thing for all devices.
For example, for a CardBus device, we used to call pci_update_resource() only for BAR 0; now we call it for BARs 0-6.
Maybe this is safe, but I can't tell from the patch, so I think you should explain *why* it's safe in the changelog.
Quoted 7 lines
include/linux/pci.h +7
-0+/* +* For PCI devices, the region numbers are assigned this way: +*/ +enum { +/* #0-5: standard PCI regions */ +PCI_STD_RESOURCES , +PCI_STD_RESOURCES_END =5 ,
include/linux/pci.h enum pci_mmap_state { · line 95
Quoted 10 lines
include/linux/pci.h +10
-0+ +/* #6: expansion ROM */ + PCI_ROM_RESOURCE, + +/* address space assigned to buses behind the bridge */ +#ifndef PCI_BRIDGE_RES_NUM +#define PCI_BRIDGE_RES_NUM 4 +#endif +PCI_BRIDGE_RESOURCES , + PCI_BRIDGE_RES_END = PCI_BRIDGE_RESOURCES + PCI_BRIDGE_RES_NUM - 1,
Since you used "PCI_STD_RESOURCES_END" above, maybe you should use "PCI_BRIDGE_RESOURCES_END" instead of "PCI_BRIDGE_RES_END".
Bjorn
Re: [PATCH 2/16 v6] PCI: define PCI resource names in an 'enum'
In reply to Yu ZhaoQuoted 3 lines · Yu Zhao
This patch moves all definitions of the PCI resource names to an 'enum', and also replaces some hard-coded resource variables with symbol names. This change eases introduction of device specific resources.
Thanks for removing a bunch of magic numbers from the code.
Quoted 15 lines
static void
pci_restore_bars(struct pci_dev *dev)
{drivers/pci/pci.c +0
-12-int i, numres; - -switch (dev->hdr_type) { -case PCI_HEADER_TYPE_NORMAL : - numres =6 ; -break ; -case PCI_HEADER_TYPE_BRIDGE : - numres =2 ; -break ; -case PCI_HEADER_TYPE_CARDBUS : - numres =1 ; -break ;
drivers/pci/pci.c pci_find_parent_resource(const struct pci_dev *dev, struct resource *res) · line 366
Quoted 10 lines
drivers/pci/pci.c +2
-5-default : -/* Should never get here, but just in case... */ -return ; - } +int i; -for (i =0 ; i < numres; i++) +for (i =0 ; i <PCI_BRIDGE_RESOURCES ; i++)pci_update_resource (dev, i); }
The behavior of this function used to depend on dev->hdr_type. Now we don't look at hdr_type at all, so we do the same thing for all devices.
For example, for a CardBus device, we used to call pci_update_resource() only for BAR 0; now we call it for BARs 0-6.
Maybe this is safe, but I can't tell from the patch, so I think you should explain *why* it's safe in the changelog.
Quoted 7 lines
include/linux/pci.h +7
-0+/* +* For PCI devices, the region numbers are assigned this way: +*/ +enum { +/* #0-5: standard PCI regions */ +PCI_STD_RESOURCES , +PCI_STD_RESOURCES_END =5 ,
include/linux/pci.h enum pci_mmap_state { · line 95
Quoted 10 lines
include/linux/pci.h +10
-0+ +/* #6: expansion ROM */ + PCI_ROM_RESOURCE, + +/* address space assigned to buses behind the bridge */ +#ifndef PCI_BRIDGE_RES_NUM +#define PCI_BRIDGE_RES_NUM 4 +#endif +PCI_BRIDGE_RESOURCES , + PCI_BRIDGE_RES_END = PCI_BRIDGE_RESOURCES + PCI_BRIDGE_RES_NUM - 1,
Since you used "PCI_STD_RESOURCES_END" above, maybe you should use "PCI_BRIDGE_RESOURCES_END" instead of "PCI_BRIDGE_RES_END".
Bjorn
Re: [PATCH 2/16 v6] PCI: define PCI resource names in an 'enum'
Quoted 37 lines · Bjorn Helgaas
Quoted 3 lines · Yu Zhao
This patch moves all definitions of the PCI resource names to an 'enum', and also replaces some hard-coded resource variables with symbol names. This change eases introduction of device specific resources.
Thanks for removing a bunch of magic numbers from the code.
Quoted 15 lines
static void
pci_restore_bars(struct pci_dev *dev)
{drivers/pci/pci.c +0
-12-int i, numres; - -switch (dev->hdr_type) { -case PCI_HEADER_TYPE_NORMAL : - numres =6 ; -break ; -case PCI_HEADER_TYPE_BRIDGE : - numres =2 ; -break ; -case PCI_HEADER_TYPE_CARDBUS : - numres =1 ; -break ;drivers/pci/pci.c pci_find_parent_resource(const struct pci_dev *dev, struct resource *res) · line 366
Quoted 10 lines
drivers/pci/pci.c +2
-5-default : -/* Should never get here, but just in case... */ -return ; - } +int i; -for (i =0 ; i < numres; i++) +for (i =0 ; i <PCI_BRIDGE_RESOURCES ; i++)pci_update_resource (dev, i); }The behavior of this function used to depend on
dev->hdr_type. Now we don't look athdr_typeat all, so we do the same thing for all devices.
Quoted 5 lines
For example, for a CardBus device, we used to call
pci_update_resource()only for BAR 0; now we call it for BARs 0-6.Maybe this is safe, but I can't tell from the patch, so I think you should explain *why* it's safe in the changelog.
It's safe because pci_update_resource() will ignore unused resources. E.g., for a Cardbus, only BAR 0 is used and its 'flags' is set, then pci_update_resource() only updates it. BAR 1-6 are ignored since their 'flags' are 0.
I'll put more explanation in the changelog.
Thanks,
Yu
Re: [PATCH 2/16 v6] PCI: define PCI resource names in an 'enum'
Quoted 44 lines · Yu Zhao
Quoted 37 lines · Bjorn Helgaas
Quoted 3 lines · Yu Zhao
This patch moves all definitions of the PCI resource names to an 'enum', and also replaces some hard-coded resource variables with symbol names. This change eases introduction of device specific resources.
Thanks for removing a bunch of magic numbers from the code.
Quoted 15 lines
static void
pci_restore_bars(struct pci_dev *dev)
{drivers/pci/pci.c +0
-12-int i, numres; - -switch (dev->hdr_type) { -case PCI_HEADER_TYPE_NORMAL : - numres =6 ; -break ; -case PCI_HEADER_TYPE_BRIDGE : - numres =2 ; -break ; -case PCI_HEADER_TYPE_CARDBUS : - numres =1 ; -break ;drivers/pci/pci.c pci_find_parent_resource(const struct pci_dev *dev, struct resource *res) · line 366
Quoted 10 lines
drivers/pci/pci.c +2
-5-default : -/* Should never get here, but just in case... */ -return ; - } +int i; -for (i =0 ; i < numres; i++) +for (i =0 ; i <PCI_BRIDGE_RESOURCES ; i++)pci_update_resource (dev, i); }The behavior of this function used to depend on
dev->hdr_type. Now we don't look athdr_typeat all, so we do the same thing for all devices.Quoted 5 lines
For example, for a CardBus device, we used to call
pci_update_resource()only for BAR 0; now we call it for BARs 0-6.Maybe this is safe, but I can't tell from the patch, so I think you should explain *why* it's safe in the changelog.
Quoted 6 lines
It's safe because
pci_update_resource()will ignore unused resources. E.g., for a Cardbus, only BAR 0 is used and its 'flags' is set, thenpci_update_resource()only updates it. BAR 1-6 are ignored since their 'flags' are 0.I'll put more explanation in the changelog.
This is a logically separate change from merely substituting enum names for magic numbers, so you might even consider splitting it into a separate patch. Better bisection and all that, you know :-)
Bjorn
Re: [PATCH 2/16 v6] PCI: define PCI resource names in an 'enum'
Quoted 47 lines · Bjorn Helgaas
Quoted 46 lines · Yu Zhao
Quoted 35 lines · Bjorn Helgaas
Quoted 3 lines · Yu Zhao
This patch moves all definitions of the PCI resource names to an 'enum', and also replaces some hard-coded resource variables with symbol names. This change eases introduction of device specific resources.
Thanks for removing a bunch of magic numbers from the code.
Quoted 15 lines
static void
pci_restore_bars(struct pci_dev *dev)
{drivers/pci/pci.c +0
-12-int i, numres; - -switch (dev->hdr_type) { -case PCI_HEADER_TYPE_NORMAL : - numres =6 ; -break ; -case PCI_HEADER_TYPE_BRIDGE : - numres =2 ; -break ; -case PCI_HEADER_TYPE_CARDBUS : - numres =1 ; -break ;drivers/pci/pci.c pci_find_parent_resource(const struct pci_dev *dev, struct resource *res) · line 366
Quoted 10 lines
drivers/pci/pci.c +2
-5-default : -/* Should never get here, but just in case... */ -return ; - } +int i; -for (i =0 ; i < numres; i++) +for (i =0 ; i <PCI_BRIDGE_RESOURCES ; i++)pci_update_resource (dev, i); }The behavior of this function used to depend on
dev->hdr_type. Now we don't look athdr_typeat all, so we do the same thing for all devices.Quoted 5 lines
For example, for a CardBus device, we used to call
pci_update_resource()only for BAR 0; now we call it for BARs 0-6.Maybe this is safe, but I can't tell from the patch, so I think you should explain *why* it's safe in the changelog.
It's safe because
pci_update_resource()will ignore unused resources. E.g., for a Cardbus, only BAR 0 is used and its 'flags' is set, thenpci_update_resource()only updates it. BAR 1-6 are ignored since their 'flags' are 0.
Quoted 5 lines
Quoted 1 line
I'll put more explanation in the changelog.
This is a logically separate change from merely substituting enum names for magic numbers, so you might even consider splitting it into a separate patch. Better bisection and all that, you know :-)
Will do.
Thanks,
Yu
Re: [PATCH 2/16 v6] PCI: define PCI resource names in an 'enum'
In reply to Yu ZhaoQuoted 44 lines · Yu Zhao
Quoted 37 lines · Bjorn Helgaas
Quoted 3 lines · Yu Zhao
This patch moves all definitions of the PCI resource names to an 'enum', and also replaces some hard-coded resource variables with symbol names. This change eases introduction of device specific resources.
Thanks for removing a bunch of magic numbers from the code.
Quoted 15 lines
static void
pci_restore_bars(struct pci_dev *dev)
{drivers/pci/pci.c +0
-12-int i, numres; - -switch (dev->hdr_type) { -case PCI_HEADER_TYPE_NORMAL : - numres =6 ; -break ; -case PCI_HEADER_TYPE_BRIDGE : - numres =2 ; -break ; -case PCI_HEADER_TYPE_CARDBUS : - numres =1 ; -break ;drivers/pci/pci.c pci_find_parent_resource(const struct pci_dev *dev, struct resource *res) · line 366
Quoted 10 lines
drivers/pci/pci.c +2
-5-default : -/* Should never get here, but just in case... */ -return ; - } +int i; -for (i =0 ; i < numres; i++) +for (i =0 ; i <PCI_BRIDGE_RESOURCES ; i++)pci_update_resource (dev, i); }The behavior of this function used to depend on
dev->hdr_type. Now we don't look athdr_typeat all, so we do the same thing for all devices.Quoted 5 lines
For example, for a CardBus device, we used to call
pci_update_resource()only for BAR 0; now we call it for BARs 0-6.Maybe this is safe, but I can't tell from the patch, so I think you should explain *why* it's safe in the changelog.
Quoted 6 lines
It's safe because
pci_update_resource()will ignore unused resources. E.g., for a Cardbus, only BAR 0 is used and its 'flags' is set, thenpci_update_resource()only updates it. BAR 1-6 are ignored since their 'flags' are 0.I'll put more explanation in the changelog.
This is a logically separate change from merely substituting enum names for magic numbers, so you might even consider splitting it into a separate patch. Better bisection and all that, you know :-)
Bjorn
Re: [PATCH 2/16 v6] PCI: define PCI resource names in an 'enum'
In reply to Yu ZhaoQuoted 123 lines · Yu Zhao
This patch moves all definitions of the PCI resource names to an 'enum', and also replaces some hard-coded resource variables with symbol names. This change eases introduction of device specific resources.
Cc Alex Chiang <[email protected]>Cc Grant Grundler <[email protected]>Cc Greg KH <[email protected]>Cc Ingo Molnar <[email protected]>Cc Jesse Barnes <[email protected]>Cc Matthew Wilcox <[email protected]>Cc Randy Dunlap <[email protected]>Cc Roland Dreier <[email protected]>Signed-off-by Yu Zhao <[email protected]>--- drivers/pci/pci-sysfs.c | 4 +++- drivers/pci/pci.c | 19 ++----------------- drivers/pci/probe.c | 2 +- drivers/pci/proc.c | 7 ++++--- include/linux/pci.h | 37 ++++++++++++++++++++++++------------- 5 files changed, 34 insertions(+), 35 deletions(-)drivers/pci/pci-sysfs.c +3
-1@@ -101,11 +101,13 @@ resource_show(struct device * dev, struct device_attribute *attr, char * buf)struct pci_dev * pci_dev =to_pci_dev (dev);char * str = buf;int i; -int max =7 ; +int max;resource_size_t start, end;if (pci_dev->subordinate) max =DEVICE_COUNT_RESOURCE ; +else + max =PCI_BRIDGE_RESOURCES ;for (i =0 ; i < max; i++) {struct resource *res = &pci_dev->resource[i];drivers/pci/pci.c +2
-17@@ -359,24 +359,9 @@ pci_find_parent_resource(const struct pci_dev *dev, struct resource *res)static void pci_restore_bars (struct pci_dev *dev) { -int i, numres; - -switch (dev->hdr_type) { -case PCI_HEADER_TYPE_NORMAL : - numres =6 ; -break ; -case PCI_HEADER_TYPE_BRIDGE : - numres =2 ; -break ; -case PCI_HEADER_TYPE_CARDBUS : - numres =1 ; -break ; -default : -/* Should never get here, but just in case... */ -return ; - } +int i; -for (i =0 ; i < numres; i++) +for (i =0 ; i <PCI_BRIDGE_RESOURCES ; i++)pci_update_resource (dev, i); }drivers/pci/probe.c +1
-1@@ -426,7 +426,7 @@ static struct pci_bus *pci_alloc_child_bus(struct pci_bus *parent, child->subordinate =0xff ;/* Set up default resource pointers and names.. */ -for (i =0 ; i <4 ; i++) { +for (i =0 ; i <PCI_BRIDGE_RES_NUM ; i++) { child->resource[i] = &bridge->resource[PCI_BRIDGE_RESOURCES +i]; child->resource[i]->name = child->name; }drivers/pci/proc.c +4
-3@@ -352,15 +352,16 @@ static int show_device(struct seq_file *m, void *v) dev->vendor, dev->device, dev->irq); -/* Here should be 7 and not PCI_NUM_RESOURCES as we need to preserve compatibility */ -for (i=0 ; i<7 ; i++) { + +/* only print standard and ROM resources to preserve compatibility */ +for (i =0 ; i <=PCI_ROM_RESOURCE ; i++) {resource_size_t start, end;pci_resource_to_user (dev, i, &dev->resource[i], &start, &end);seq_printf (m,"\t%16llx" , (unsigned long long )(start | (dev->resource[i].flags &PCI_REGION_FLAG_MASK ))); } -for (i=0 ; i<7 ; i++) { +for (i =0 ; i <=PCI_ROM_RESOURCE ; i++) {resource_size_t start, end;pci_resource_to_user (dev, i, &dev->resource[i], &start, &end); seq_printf(m, "\t%16llx",include/linux/pci.h +5
-1@@ -76,7 +76,30 @@ enum pci_mmap_state {#define PCI_DMA_FROMDEVICE 2#define PCI_DMA_NONE 3 -#define DEVICE_COUNT_RESOURCE 12 +/* +* For PCI devices, the region numbers are assigned this way: +*/ +enum { +/* #0-5: standard PCI regions */
include/linux/pci.h enum pci_mmap_state { · line 93
Quoted 10 lines
include/linux/pci.h +10
-0+PCI_STD_RESOURCES , +PCI_STD_RESOURCES_END =5 , + +/* #6: expansion ROM */ +PCI_ROM_RESOURCE , + +/* address space assigned to buses behind the bridge */ +#ifndef PCI_BRIDGE_RES_NUM +#define PCI_BRIDGE_RES_NUM 4 +#endif
Is there any intention to ever set PCI_BRIDGE_RES_NUM to any value other than 4? I'm confused about why it is protected by #ifndef as I can't find it declared anywhere else.
Quoted 31 lines
include/linux/pci.h +9
-12+PCI_BRIDGE_RESOURCES , +PCI_BRIDGE_RES_END =PCI_BRIDGE_RESOURCES +PCI_BRIDGE_RES_NUM -1 , + +/* total resources associated with a PCI device */ +PCI_NUM_RESOURCES , + +/* preserve this for compatibility */ + DEVICE_COUNT_RESOURCE +};typedef int __bitwise pci_power_t ;@@ -262,18 +285,6 @@ static inline void pci_add_saved_cap(struct pci_dev *pci_dev,hlist_add_head (&new_cap->next, &pci_dev->saved_cap_space); } -/* -* For PCI devices, the region numbers are assigned this way: -* -* 0-5 standard PCI regions -* 6 expansion ROM -* 7-10 bridges: address space assigned to buses behind the bridge -*/ - -#define PCI_ROM_RESOURCE 6 -#define PCI_BRIDGE_RESOURCES 7 -#define PCI_NUM_RESOURCES 11 -#ifndef PCI_BUS_NUM_RESOURCES #define PCI_BUS_NUM_RESOURCES 16#endif
Signature · 3 lines
Simon Horman VA Linux Systems Japan K.K., Sydney, Australia Satellite Office H: www.vergenet.net/~horms/ W: www.valinux.co.jp/en
PATCH v6 3/16 PCI: export __pci_read_base
In reply to Yu ZhaoExport __pci_read_base() so it can be used by whole PCI subsystem.
--- drivers/pci/pci.h | 9 +++++++++ drivers/pci/probe.c | 20 +++++++++----------- 2 files changed, 18 insertions(+), 11 deletions(-)
drivers/pci/pci.h +9 -0
@@ -157,6 +157,15 @@ struct pci_slot_attribute { };#define to_pci_slot_attr (s) container_of(s, struct pci_slot_attribute, attr) +enum pci_bar_type { + pci_bar_unknown,/* Standard PCI BAR probe */ + pci_bar_io,/* An io port BAR */ + pci_bar_mem32,/* A 32-bit memory BAR */ + pci_bar_mem64,/* A 64-bit memory BAR */ +}; + +extern int __pci_read_base (struct pci_dev *dev,enum pci_bar_type type, +struct resource *res,unsigned int reg);extern void pci_enable_ari (struct pci_dev *dev);/** * pci_ari_enabled - query ARI forwarding status
drivers/pci/probe.c +9 -11
@@ -135,13 +135,6 @@ static u64 pci_size(u64 base, u64 maxbase, u64 mask)return size; } -enum pci_bar_type { - pci_bar_unknown,/* Standard PCI BAR probe */ - pci_bar_io,/* An io port BAR */ - pci_bar_mem32,/* A 32-bit memory BAR */ - pci_bar_mem64,/* A 64-bit memory BAR */ -}; -static inline enum pci_bar_type decode_bar (struct resource *res,u32 bar) {if ((bar &PCI_BASE_ADDRESS_SPACE ) ==PCI_BASE_ADDRESS_SPACE_IO ) {
@@ -156,11 +149,16 @@ static inline enum pci_bar_type decode_bar(struct resource *res, u32 bar)return pci_bar_mem32; } -/* -* If the type is not unknown, we assume that the lowest bit is 'enable'. -* Returns 1 if the BAR was 64-bit and 0 if it was 32-bit. +/** +* pci_read_base - read a PCI BAR +* @dev: the PCI device +* @type: type of the BAR +* @res: resource buffer to be filled in +* @pos: BAR position in the config space +* +* Returns 1 if the BAR is 64-bit, or 0 if 32-bit. */ -static int __pci_read_base (struct pci_dev *dev,enum pci_bar_type type, +int __pci_read_base (struct pci_dev *dev,enum pci_bar_type type,struct resource *res,unsigned int pos) {u32 l, sz, mask;
Signature · 1 line
1.5.6.4
PATCH v6 3/16 PCI: export __pci_read_base
In reply to Yu ZhaoExport __pci_read_base() so it can be used by whole PCI subsystem.
--- drivers/pci/pci.h | 9 +++++++++ drivers/pci/probe.c | 20 +++++++++----------- 2 files changed, 18 insertions(+), 11 deletions(-)
drivers/pci/pci.h +9 -0
@@ -157,6 +157,15 @@ struct pci_slot_attribute { };#define to_pci_slot_attr (s) container_of(s, struct pci_slot_attribute, attr) +enum pci_bar_type { + pci_bar_unknown,/* Standard PCI BAR probe */ + pci_bar_io,/* An io port BAR */ + pci_bar_mem32,/* A 32-bit memory BAR */ + pci_bar_mem64,/* A 64-bit memory BAR */ +}; + +extern int __pci_read_base (struct pci_dev *dev,enum pci_bar_type type, +struct resource *res,unsigned int reg);extern void pci_enable_ari (struct pci_dev *dev);/** * pci_ari_enabled - query ARI forwarding status
drivers/pci/probe.c +9 -11
@@ -135,13 +135,6 @@ static u64 pci_size(u64 base, u64 maxbase, u64 mask)return size; } -enum pci_bar_type { - pci_bar_unknown,/* Standard PCI BAR probe */ - pci_bar_io,/* An io port BAR */ - pci_bar_mem32,/* A 32-bit memory BAR */ - pci_bar_mem64,/* A 64-bit memory BAR */ -}; -static inline enum pci_bar_type decode_bar (struct resource *res,u32 bar) {if ((bar &PCI_BASE_ADDRESS_SPACE ) ==PCI_BASE_ADDRESS_SPACE_IO ) {
@@ -156,11 +149,16 @@ static inline enum pci_bar_type decode_bar(struct resource *res, u32 bar)return pci_bar_mem32; } -/* -* If the type is not unknown, we assume that the lowest bit is 'enable'. -* Returns 1 if the BAR was 64-bit and 0 if it was 32-bit. +/** +* pci_read_base - read a PCI BAR +* @dev: the PCI device +* @type: type of the BAR +* @res: resource buffer to be filled in +* @pos: BAR position in the config space +* +* Returns 1 if the BAR is 64-bit, or 0 if 32-bit. */ -static int __pci_read_base (struct pci_dev *dev,enum pci_bar_type type, +int __pci_read_base (struct pci_dev *dev,enum pci_bar_type type,struct resource *res,unsigned int pos) {u32 l, sz, mask;
Signature · 1 line
1.5.6.4
PATCH v6 4/16 PCI: make pci_alloc_child_bus() be able to handle NULL bridge
In reply to Yu ZhaoMake pci_alloc_child_bus() be able to allocate buses without bridge devices. Some SR-IOV devices can occupy more than one bus number, but there is no explicit bridges because that have internal routing mechanism.
--- drivers/pci/probe.c | 7 +++++-- 1 files changed, 5 insertions(+), 2 deletions(-)
drivers/pci/probe.c +5 -2
@@ -401,12 +401,10 @@ static struct pci_bus *pci_alloc_child_bus(struct pci_bus *parent,if (!child)return NULL ; - child->self = bridge; child->parent = parent; child->ops = parent->ops; child->sysdata = parent->sysdata; child->bus_flags = parent->bus_flags; - child->bridge =get_device (&bridge->dev);/* initialize some portions of the bus device, but don't register it * now as the parent is not properly set up yet. This device will get
@@ -423,6 +421,11 @@ static struct pci_bus *pci_alloc_child_bus(struct pci_bus *parent, child->primary = parent->secondary; child->subordinate =0xff ; +if (!bridge) +return child; + + child->self = bridge; + child->bridge =get_device (&bridge->dev);/* Set up default resource pointers and names.. */ for (i =0 ; i <PCI_BRIDGE_RES_NUM ; i++) { child->resource[i] = &bridge->resource[PCI_BRIDGE_RESOURCES +i];
Signature · 1 line
1.5.6.4
PATCH v6 4/16 PCI: make pci_alloc_child_bus() be able to handle NULL bridge
In reply to Yu ZhaoMake pci_alloc_child_bus() be able to allocate buses without bridge devices. Some SR-IOV devices can occupy more than one bus number, but there is no explicit bridges because that have internal routing mechanism.
--- drivers/pci/probe.c | 7 +++++-- 1 files changed, 5 insertions(+), 2 deletions(-)
drivers/pci/probe.c +5 -2
@@ -401,12 +401,10 @@ static struct pci_bus *pci_alloc_child_bus(struct pci_bus *parent,if (!child)return NULL ; - child->self = bridge; child->parent = parent; child->ops = parent->ops; child->sysdata = parent->sysdata; child->bus_flags = parent->bus_flags; - child->bridge =get_device (&bridge->dev);/* initialize some portions of the bus device, but don't register it * now as the parent is not properly set up yet. This device will get
@@ -423,6 +421,11 @@ static struct pci_bus *pci_alloc_child_bus(struct pci_bus *parent, child->primary = parent->secondary; child->subordinate =0xff ; +if (!bridge) +return child; + + child->self = bridge; + child->bridge =get_device (&bridge->dev);/* Set up default resource pointers and names.. */ for (i =0 ; i <PCI_BRIDGE_RES_NUM ; i++) { child->resource[i] = &bridge->resource[PCI_BRIDGE_RESOURCES +i];
Signature · 1 line
1.5.6.4
PATCH v6 5/16 PCI: add a wrapper for resource_alignment()
In reply to Yu ZhaoAdd a wrapper for resource_alignment() so it can handle device specific resource alignment.
--- drivers/pci/pci.c | 20 ++++++++++++++++++++ drivers/pci/pci.h | 1 + drivers/pci/setup-bus.c | 4 ++-- drivers/pci/setup-res.c | 7 ++++--- 4 files changed, 27 insertions(+), 5 deletions(-)
drivers/pci/pci.c +20 -0
@@ -1904,6 +1904,26 @@ int pci_select_bars(struct pci_dev *dev, unsigned long flags)return bars; } +/** +* pci_resource_alignment - get a PCI BAR resource alignment +* @dev: the PCI device +* @resno: the resource number +* +* Returns alignment size on success, or 0 on error. +*/ +int pci_resource_alignment (struct pci_dev *dev,int resno) +{ +resource_size_t align; +struct resource *res = dev->resource + resno; + + align =resource_alignment (res); +if (align) +return align; + +dev_err (&dev->dev,"alignment: invalid resource #%d\n" , resno); +return 0 ; +} +static void __devinit pci_no_domains (void ) {#ifdef CONFIG_PCI_DOMAINS
drivers/pci/pci.h +1 -0
@@ -166,6 +166,7 @@ enum pci_bar_type {extern int __pci_read_base (struct pci_dev *dev,enum pci_bar_type type,struct resource *res,unsigned int reg); +extern int pci_resource_alignment (struct pci_dev *dev,int resno);extern void pci_enable_ari (struct pci_dev *dev);/** * pci_ari_enabled - query ARI forwarding status
drivers/pci/setup-bus.c +2 -2
@@ -25,6 +25,7 @@#include <linux/ioport.h> #include <linux/cache.h> #include <linux/slab.h> +#include "pci.h" static void pbus_assign_resources_sorted (struct pci_bus *bus)
@@ -351,8 +352,7 @@ static int pbus_size_mem(struct pci_bus *bus, unsigned long mask, unsigned longif (r->parent || (r->flags & mask) != type)continue ; r_size =resource_size (r); -/* For bridges size != alignment */ - align =resource_alignment (r); + align =pci_resource_alignment (dev, i); order =__ffs (align) -20 ;if (order >11 ) { dev_warn(&dev->dev, "BAR %d bad alignment %llx: "
drivers/pci/setup-res.c +4 -3
@@ -133,7 +133,7 @@ int pci_assign_resource(struct pci_dev *dev, int resno) size =resource_size (res); min = (res->flags &IORESOURCE_IO ) ?PCIBIOS_MIN_IO :PCIBIOS_MIN_MEM ; - align =resource_alignment (res); + align =pci_resource_alignment (dev, resno);if (!align) { dev_err(&dev->dev, "BAR %d: can't allocate resource (bogus " "alignment) %pR flags %#lx\n",
@@ -224,7 +224,7 @@ void pdev_sort_resources(struct pci_dev *dev, struct resource_list *head)if (!(r->flags) || r->parent)continue ; - r_align =resource_alignment (r); + r_align =pci_resource_alignment (dev, i);if (!r_align) { dev_warn(&dev->dev, "BAR %d: bogus alignment " "%pR flags %#lx\n",
@@ -236,7 +236,8 @@ void pdev_sort_resources(struct pci_dev *dev, struct resource_list *head)struct resource_list *ln = list->next;if (ln) - align =resource_alignment (ln->res); + align =pci_resource_alignment (ln->dev, + ln->res - ln->dev->resource);if (r_align > align) { tmp =kmalloc (sizeof (*tmp),GFP_KERNEL );
Signature · 1 line
1.5.6.4
PATCH v6 5/16 PCI: add a wrapper for resource_alignment()
In reply to Yu ZhaoAdd a wrapper for resource_alignment() so it can handle device specific resource alignment.
--- drivers/pci/pci.c | 20 ++++++++++++++++++++ drivers/pci/pci.h | 1 + drivers/pci/setup-bus.c | 4 ++-- drivers/pci/setup-res.c | 7 ++++--- 4 files changed, 27 insertions(+), 5 deletions(-)
drivers/pci/pci.c +20 -0
@@ -1904,6 +1904,26 @@ int pci_select_bars(struct pci_dev *dev, unsigned long flags)return bars; } +/** +* pci_resource_alignment - get a PCI BAR resource alignment +* @dev: the PCI device +* @resno: the resource number +* +* Returns alignment size on success, or 0 on error. +*/ +int pci_resource_alignment (struct pci_dev *dev,int resno) +{ +resource_size_t align; +struct resource *res = dev->resource + resno; + + align =resource_alignment (res); +if (align) +return align; + +dev_err (&dev->dev,"alignment: invalid resource #%d\n" , resno); +return 0 ; +} +static void __devinit pci_no_domains (void ) {#ifdef CONFIG_PCI_DOMAINS
drivers/pci/pci.h +1 -0
@@ -166,6 +166,7 @@ enum pci_bar_type {extern int __pci_read_base (struct pci_dev *dev,enum pci_bar_type type,struct resource *res,unsigned int reg); +extern int pci_resource_alignment (struct pci_dev *dev,int resno);extern void pci_enable_ari (struct pci_dev *dev);/** * pci_ari_enabled - query ARI forwarding status
drivers/pci/setup-bus.c +2 -2
@@ -25,6 +25,7 @@#include <linux/ioport.h> #include <linux/cache.h> #include <linux/slab.h> +#include "pci.h" static void pbus_assign_resources_sorted (struct pci_bus *bus)
@@ -351,8 +352,7 @@ static int pbus_size_mem(struct pci_bus *bus, unsigned long mask, unsigned longif (r->parent || (r->flags & mask) != type)continue ; r_size =resource_size (r); -/* For bridges size != alignment */ - align =resource_alignment (r); + align =pci_resource_alignment (dev, i); order =__ffs (align) -20 ;if (order >11 ) { dev_warn(&dev->dev, "BAR %d bad alignment %llx: "
drivers/pci/setup-res.c +4 -3
@@ -133,7 +133,7 @@ int pci_assign_resource(struct pci_dev *dev, int resno) size =resource_size (res); min = (res->flags &IORESOURCE_IO ) ?PCIBIOS_MIN_IO :PCIBIOS_MIN_MEM ; - align =resource_alignment (res); + align =pci_resource_alignment (dev, resno);if (!align) { dev_err(&dev->dev, "BAR %d: can't allocate resource (bogus " "alignment) %pR flags %#lx\n",
@@ -224,7 +224,7 @@ void pdev_sort_resources(struct pci_dev *dev, struct resource_list *head)if (!(r->flags) || r->parent)continue ; - r_align =resource_alignment (r); + r_align =pci_resource_alignment (dev, i);if (!r_align) { dev_warn(&dev->dev, "BAR %d: bogus alignment " "%pR flags %#lx\n",
@@ -236,7 +236,8 @@ void pdev_sort_resources(struct pci_dev *dev, struct resource_list *head)struct resource_list *ln = list->next;if (ln) - align =resource_alignment (ln->res); + align =pci_resource_alignment (ln->dev, + ln->res - ln->dev->resource);if (r_align > align) { tmp =kmalloc (sizeof (*tmp),GFP_KERNEL );
Signature · 1 line
1.5.6.4
PATCH v6 6/16 PCI: add a new function to map BAR offset
In reply to Yu ZhaoAdd a function to map resource number to corresponding register so people can get the offset and type of device specific BARs.
--- drivers/pci/pci.c | 22 ++++++++++++++++++++++ drivers/pci/pci.h | 2 ++ drivers/pci/setup-res.c | 13 +++++-------- 3 files changed, 29 insertions(+), 8 deletions(-)
drivers/pci/pci.c +22 -0
@@ -1924,6 +1924,28 @@ int pci_resource_alignment(struct pci_dev *dev, int resno)return 0 ; } +/** +* pci_resource_bar - get position of the BAR associated with a resource +* @dev: the PCI device +* @resno: the resource number +* @type: the BAR type to be filled in +* +* Returns BAR position in config space, or 0 if the BAR is invalid. +*/ +int pci_resource_bar (struct pci_dev *dev,int resno,enum pci_bar_type *type) +{ +if (resno <PCI_ROM_RESOURCE ) { + *type = pci_bar_unknown; +return PCI_BASE_ADDRESS_0 +4 * resno; + }else if (resno ==PCI_ROM_RESOURCE ) { + *type = pci_bar_mem32; +return dev->rom_base_reg; + } + +dev_err (&dev->dev,"BAR: invalid resource #%d\n" , resno); +return 0 ; +} +static void __devinit pci_no_domains (void ) {#ifdef CONFIG_PCI_DOMAINS
drivers/pci/pci.h +2 -0
@@ -167,6 +167,8 @@ enum pci_bar_type {extern int __pci_read_base (struct pci_dev *dev,enum pci_bar_type type,struct resource *res,unsigned int reg);extern int pci_resource_alignment (struct pci_dev *dev,int resno); +extern int pci_resource_bar (struct pci_dev *dev,int resno, +enum pci_bar_type *type);extern void pci_enable_ari (struct pci_dev *dev);/** * pci_ari_enabled - query ARI forwarding status
drivers/pci/setup-res.c +5 -8
@@ -31,6 +31,7 @@ void pci_update_resource(struct pci_dev *dev, int resno)struct pci_bus_region region;u32 new, check, mask;int reg; +enum pci_bar_type type;struct resource *res = dev->resource + resno;/*
@@ -62,17 +63,13 @@ void pci_update_resource(struct pci_dev *dev, int resno)else mask = (u32 )PCI_BASE_ADDRESS_MEM_MASK ; -if (resno <6 ) { - reg =PCI_BASE_ADDRESS_0 +4 * resno; - }else if (resno ==PCI_ROM_RESOURCE ) { + reg =pci_resource_bar (dev, resno, &type); +if (!reg) +return ; +if (type != pci_bar_unknown) {if (!(res->flags &IORESOURCE_ROM_ENABLE ))return ; new |=PCI_ROM_ADDRESS_ENABLE ; - reg = dev->rom_base_reg; - }else { -/* Hmm, non-standard resource. */ - -return ;/* kill uninitialised var warning */ }pci_write_config_dword (dev, reg, new);
Signature · 1 line
1.5.6.4
PATCH v6 6/16 PCI: add a new function to map BAR offset
In reply to Yu ZhaoAdd a function to map resource number to corresponding register so people can get the offset and type of device specific BARs.
--- drivers/pci/pci.c | 22 ++++++++++++++++++++++ drivers/pci/pci.h | 2 ++ drivers/pci/setup-res.c | 13 +++++-------- 3 files changed, 29 insertions(+), 8 deletions(-)
drivers/pci/pci.c +22 -0
@@ -1924,6 +1924,28 @@ int pci_resource_alignment(struct pci_dev *dev, int resno)return 0 ; } +/** +* pci_resource_bar - get position of the BAR associated with a resource +* @dev: the PCI device +* @resno: the resource number +* @type: the BAR type to be filled in +* +* Returns BAR position in config space, or 0 if the BAR is invalid. +*/ +int pci_resource_bar (struct pci_dev *dev,int resno,enum pci_bar_type *type) +{ +if (resno <PCI_ROM_RESOURCE ) { + *type = pci_bar_unknown; +return PCI_BASE_ADDRESS_0 +4 * resno; + }else if (resno ==PCI_ROM_RESOURCE ) { + *type = pci_bar_mem32; +return dev->rom_base_reg; + } + +dev_err (&dev->dev,"BAR: invalid resource #%d\n" , resno); +return 0 ; +} +static void __devinit pci_no_domains (void ) {#ifdef CONFIG_PCI_DOMAINS
drivers/pci/pci.h +2 -0
@@ -167,6 +167,8 @@ enum pci_bar_type {extern int __pci_read_base (struct pci_dev *dev,enum pci_bar_type type,struct resource *res,unsigned int reg);extern int pci_resource_alignment (struct pci_dev *dev,int resno); +extern int pci_resource_bar (struct pci_dev *dev,int resno, +enum pci_bar_type *type);extern void pci_enable_ari (struct pci_dev *dev);/** * pci_ari_enabled - query ARI forwarding status
drivers/pci/setup-res.c +5 -8
@@ -31,6 +31,7 @@ void pci_update_resource(struct pci_dev *dev, int resno)struct pci_bus_region region;u32 new, check, mask;int reg; +enum pci_bar_type type;struct resource *res = dev->resource + resno;/*
@@ -62,17 +63,13 @@ void pci_update_resource(struct pci_dev *dev, int resno)else mask = (u32 )PCI_BASE_ADDRESS_MEM_MASK ; -if (resno <6 ) { - reg =PCI_BASE_ADDRESS_0 +4 * resno; - }else if (resno ==PCI_ROM_RESOURCE ) { + reg =pci_resource_bar (dev, resno, &type); +if (!reg) +return ; +if (type != pci_bar_unknown) {if (!(res->flags &IORESOURCE_ROM_ENABLE ))return ; new |=PCI_ROM_ADDRESS_ENABLE ; - reg = dev->rom_base_reg; - }else { -/* Hmm, non-standard resource. */ - -return ;/* kill uninitialised var warning */ }pci_write_config_dword (dev, reg, new);
Signature · 1 line
1.5.6.4
PATCH v6 7/16 PCI: cleanup pcibios_allocate_resources()
In reply to Yu ZhaoThis cleanup makes pcibios_allocate_resources() easier to read.
--- arch/x86/pci/i386.c | 28 ++++++++++++++-------------- 1 files changed, 14 insertions(+), 14 deletions(-)
arch/x86/pci/i386.c +14 -14
@@ -147,7 +147,7 @@ static void __init pcibios_allocate_bus_resources(struct list_head *bus_list)static void __init pcibios_allocate_resources (int pass) {struct pci_dev *dev =NULL ; -int idx, disabled; +int idx, enabled;u16 command;struct resource *r, *pr;
@@ -160,22 +160,22 @@ static void __init pcibios_allocate_resources(int pass)if (!r->start)/* Address not assigned at all */ continue ;if (r->flags &IORESOURCE_IO ) - disabled = !(command &PCI_COMMAND_IO ); + enabled = command &PCI_COMMAND_IO ;else - disabled = !(command &PCI_COMMAND_MEMORY ); -if (pass == disabled) { -dev_dbg (&dev->dev,"resource %#08llx-%#08llx (f=%lx, d=%d, p=%d)\n" , + enabled = command &PCI_COMMAND_MEMORY ; +if (pass == enabled) +continue ; +dev_dbg (&dev->dev,"resource %#08llx-%#08llx (f=%lx, d=%d, p=%d)\n" , (unsigned long long ) r->start, (unsigned long long ) r->end, - r->flags, disabled, pass); - pr =pci_find_parent_resource (dev, r); -if (!pr ||request_resource (pr, r) <0 ) { -dev_err (&dev->dev,"BAR %d: can't allocate resource\n" , idx); -/* We'll assign a new address later */ - r->end -= r->start; - r->start =0 ; - } - } + r->flags, enabled, pass); + pr =pci_find_parent_resource (dev, r); +if (pr && !request_resource (pr, r)) +continue ; +dev_err (&dev->dev,"BAR %d: can't allocate resource\n" , idx); +/* We'll assign a new address later */ + r->end -= r->start; + r->start =0 ; }if (!pass) { r = &dev->resource[PCI_ROM_RESOURCE ];
Signature · 1 line
1.5.6.4
PATCH v6 7/16 PCI: cleanup pcibios_allocate_resources()
In reply to Yu ZhaoThis cleanup makes pcibios_allocate_resources() easier to read.
--- arch/x86/pci/i386.c | 28 ++++++++++++++-------------- 1 files changed, 14 insertions(+), 14 deletions(-)
arch/x86/pci/i386.c +14 -14
@@ -147,7 +147,7 @@ static void __init pcibios_allocate_bus_resources(struct list_head *bus_list)static void __init pcibios_allocate_resources (int pass) {struct pci_dev *dev =NULL ; -int idx, disabled; +int idx, enabled;u16 command;struct resource *r, *pr;
@@ -160,22 +160,22 @@ static void __init pcibios_allocate_resources(int pass)if (!r->start)/* Address not assigned at all */ continue ;if (r->flags &IORESOURCE_IO ) - disabled = !(command &PCI_COMMAND_IO ); + enabled = command &PCI_COMMAND_IO ;else - disabled = !(command &PCI_COMMAND_MEMORY ); -if (pass == disabled) { -dev_dbg (&dev->dev,"resource %#08llx-%#08llx (f=%lx, d=%d, p=%d)\n" , + enabled = command &PCI_COMMAND_MEMORY ; +if (pass == enabled) +continue ; +dev_dbg (&dev->dev,"resource %#08llx-%#08llx (f=%lx, d=%d, p=%d)\n" , (unsigned long long ) r->start, (unsigned long long ) r->end, - r->flags, disabled, pass); - pr =pci_find_parent_resource (dev, r); -if (!pr ||request_resource (pr, r) <0 ) { -dev_err (&dev->dev,"BAR %d: can't allocate resource\n" , idx); -/* We'll assign a new address later */ - r->end -= r->start; - r->start =0 ; - } - } + r->flags, enabled, pass); + pr =pci_find_parent_resource (dev, r); +if (pr && !request_resource (pr, r)) +continue ; +dev_err (&dev->dev,"BAR %d: can't allocate resource\n" , idx); +/* We'll assign a new address later */ + r->end -= r->start; + r->start =0 ; }if (!pass) { r = &dev->resource[PCI_ROM_RESOURCE ];
Signature · 1 line
1.5.6.4
Re: [PATCH 7/16 v6] PCI: cleanup pcibios_allocate_resources()
Quoted 32 lines · Yu Zhao
This cleanup makes
pcibios_allocate_resources()easier to read.Cc Alex Chiang <[email protected]>Cc Grant Grundler <[email protected]>Cc Greg KH <[email protected]>Cc Ingo Molnar <[email protected]>Cc Jesse Barnes <[email protected]>Cc Matthew Wilcox <[email protected]>Cc Randy Dunlap <[email protected]>Cc Roland Dreier <[email protected]>Signed-off-by Yu Zhao <[email protected]>--- arch/x86/pci/i386.c | 28 ++++++++++++++-------------- 1 files changed, 14 insertions(+), 14 deletions(-)arch/x86/pci/i386.c +1
-1@@ -147,7 +147,7 @@ static void __init pcibios_allocate_bus_resources(struct list_head *bus_list)static void __init pcibios_allocate_resources (int pass) {struct pci_dev *dev =NULL ; -int idx, disabled; +int idx, enabled;u16 command;struct resource *r, *pr;@@ -160,22 +160,22 @@ static void __init pcibios_allocate_resources(int pass)if (!r->start)/* Address not assigned at all */ continue ;
arch/x86/pci/i386.c static void __init pcibios_allocate_resources(int pass) · line 167
Quoted 10 lines
arch/x86/pci/i386.c +4
-4if (r->flags &IORESOURCE_IO ) - disabled = !(command &PCI_COMMAND_IO ); + enabled = command &PCI_COMMAND_IO ;else - disabled = !(command &PCI_COMMAND_MEMORY ); -if (pass == disabled) { - dev_dbg(&dev->dev, "resource %#08llx-%#08llx (f=%lx, d=%d, p=%d)\n", + enabled = command &PCI_COMMAND_MEMORY ; +if (pass == enabled) +continue ;
it seems you change the flow here for MMIO
because PCI_COMMAND_MEMORY is 2.
YH
Re: [PATCH 7/16 v6] PCI: cleanup pcibios_allocate_resources()
Quoted 31 lines · Yinghai Lu
Quoted 16 lines · Yu Zhao
arch/x86/pci/i386.c +1
-1@@ -147,7 +147,7 @@ static void __init pcibios_allocate_bus_resources(struct list_head *bus_list)static void __init pcibios_allocate_resources (int pass) {struct pci_dev *dev =NULL ; -int idx, disabled; +int idx, enabled;u16 command;struct resource *r, *pr;@@ -160,22 +160,22 @@ static void __init pcibios_allocate_resources(int pass)if (!r->start)/* Address not assigned at all */ continue ;arch/x86/pci/i386.c static void __init pcibios_allocate_resources(int pass) · line 167
Quoted 10 lines
arch/x86/pci/i386.c +4
-4if (r->flags &IORESOURCE_IO ) - disabled = !(command &PCI_COMMAND_IO ); + enabled = command &PCI_COMMAND_IO ;else - disabled = !(command &PCI_COMMAND_MEMORY ); -if (pass == disabled) { - dev_dbg(&dev->dev, "resource %#08llx-%#08llx (f=%lx, d=%d, p=%d)\n", + enabled = command &PCI_COMMAND_MEMORY ; +if (pass == enabled) +continue ;it seems you change the flow here for MMIO
becausePCI_COMMAND_MEMORYis 2.
Quoted 1 line
YH
Nice finding! Will change it back to 'disable' next version.
Thanks,
Yu
Re: [PATCH 7/16 v6] PCI: cleanup pcibios_allocate_resources()
In reply to Yinghai LuQuoted 31 lines · Yinghai Lu
Quoted 16 lines · Yu Zhao
arch/x86/pci/i386.c +1
-1@@ -147,7 +147,7 @@ static void __init pcibios_allocate_bus_resources(struct list_head *bus_list)static void __init pcibios_allocate_resources (int pass) {struct pci_dev *dev =NULL ; -int idx, disabled; +int idx, enabled;u16 command;struct resource *r, *pr;@@ -160,22 +160,22 @@ static void __init pcibios_allocate_resources(int pass)if (!r->start)/* Address not assigned at all */ continue ;arch/x86/pci/i386.c static void __init pcibios_allocate_resources(int pass) · line 167
Quoted 10 lines
arch/x86/pci/i386.c +4
-4if (r->flags &IORESOURCE_IO ) - disabled = !(command &PCI_COMMAND_IO ); + enabled = command &PCI_COMMAND_IO ;else - disabled = !(command &PCI_COMMAND_MEMORY ); -if (pass == disabled) { - dev_dbg(&dev->dev, "resource %#08llx-%#08llx (f=%lx, d=%d, p=%d)\n", + enabled = command &PCI_COMMAND_MEMORY ; +if (pass == enabled) +continue ;it seems you change the flow here for MMIO
becausePCI_COMMAND_MEMORYis 2.
Quoted 1 line
YH
Nice finding! Will change it back to 'disable' next version.
Thanks,
Yu
Re: [PATCH 7/16 v6] PCI: cleanup pcibios_allocate_resources()
In reply to Yu ZhaoQuoted 32 lines · Yu Zhao
This cleanup makes
pcibios_allocate_resources()easier to read.Cc Alex Chiang <[email protected]>Cc Grant Grundler <[email protected]>Cc Greg KH <[email protected]>Cc Ingo Molnar <[email protected]>Cc Jesse Barnes <[email protected]>Cc Matthew Wilcox <[email protected]>Cc Randy Dunlap <[email protected]>Cc Roland Dreier <[email protected]>Signed-off-by Yu Zhao <[email protected]>--- arch/x86/pci/i386.c | 28 ++++++++++++++-------------- 1 files changed, 14 insertions(+), 14 deletions(-)arch/x86/pci/i386.c +1
-1@@ -147,7 +147,7 @@ static void __init pcibios_allocate_bus_resources(struct list_head *bus_list)static void __init pcibios_allocate_resources (int pass) {struct pci_dev *dev =NULL ; -int idx, disabled; +int idx, enabled;u16 command;struct resource *r, *pr;@@ -160,22 +160,22 @@ static void __init pcibios_allocate_resources(int pass)if (!r->start)/* Address not assigned at all */ continue ;
arch/x86/pci/i386.c static void __init pcibios_allocate_resources(int pass) · line 167
Quoted 10 lines
arch/x86/pci/i386.c +4
-4if (r->flags &IORESOURCE_IO ) - disabled = !(command &PCI_COMMAND_IO ); + enabled = command &PCI_COMMAND_IO ;else - disabled = !(command &PCI_COMMAND_MEMORY ); -if (pass == disabled) { - dev_dbg(&dev->dev, "resource %#08llx-%#08llx (f=%lx, d=%d, p=%d)\n", + enabled = command &PCI_COMMAND_MEMORY ; +if (pass == enabled) +continue ;
it seems you change the flow here for MMIO
because PCI_COMMAND_MEMORY is 2.
YH
PATCH v6 8/16 PCI: add boot options to reassign resources
In reply to Yu ZhaoThis patch adds boot options so user can reassign device resources of all devices under a bus.
The boot options can be used as:
pci=assign-mmio=0000:01,assign-pio=0000:02
'[dddd:]bb' is the domain and bus number.
--- arch/x86/pci/common.c | 73 +++++++++++++++++++++++++++++++++++++++++++++++++ arch/x86/pci/i386.c | 10 ++++--- arch/x86/pci/pci.h | 3 ++ 3 files changed, 82 insertions(+), 4 deletions(-)
arch/x86/pci/common.c +73 -0
@@ -137,6 +137,72 @@ static void __devinit pcibios_fixup_device_resources(struct pci_dev *dev) } } +static char *pci_assign_pio; +static char *pci_assign_mmio; + +static int pcibios_bus_resource_needs_fixup (struct pci_bus *bus) +{ +int i; +int type =0 ; +int domain, busnr; + +if (!bus->self) +return 0 ; + +for (i =0 ; i <2 ; i++) { +char *str = i ? pci_assign_pio : pci_assign_mmio; + +while (str && *str) { +if (sscanf (str,"%04x:%02x" , &domain, &busnr) !=2 ) { +if (sscanf (str,"%02x" , &busnr) !=1 ) +break ; + domain =0 ; + } + +if (pci_domain_nr (bus) == domain && + bus->number == busnr) { + type |= i ?IORESOURCE_IO :IORESOURCE_MEM ; +break ; + } + + str =strchr (str,';' ); +if (str) + str++; + } + } + +return type; +} + +static void __devinit pcibios_fixup_bus_resources (struct pci_bus *bus) +{ +int i; +int type =pcibios_bus_resource_needs_fixup (bus); + +if (!type) +return ; + +for (i =0 ; i <PCI_BUS_NUM_RESOURCES ; i++) { +struct resource *res = bus->resource[i]; + +if (!res) +continue ; +if (res->flags & type) + res->flags =0 ; + } +} + +int pcibios_resource_needs_fixup (struct pci_dev *dev,int resno) +{ +struct pci_bus *bus; + +for (bus = dev->bus; bus && bus != pci_root_bus; bus = bus->parent) +if (pcibios_bus_resource_needs_fixup (bus)) +return 1 ; + +return 0 ; +} +/* * Called after each bus is probed, but before its children * are examined.
@@ -147,6 +213,7 @@ void __devinit pcibios_fixup_bus(struct pci_bus *b)struct pci_dev *dev;pci_read_bridge_bases (b); +pcibios_fixup_bus_resources (b);list_for_each_entry (dev, &b->devices, bus_list)pcibios_fixup_device_resources (dev); }
@@ -519,6 +586,12 @@ char * __devinit pcibios_setup(char *str) }else if (!strcmp (str,"skip_isa_align" )) { pci_probe |=PCI_CAN_SKIP_ISA_ALIGN ;return NULL ; + }else if (!strncmp (str,"assign-pio=" ,11 )) { + pci_assign_pio = str +11 ; +return NULL ; + }else if (!strncmp (str,"assign-mmio=" ,12 )) { + pci_assign_mmio = str +12 ; +return NULL ; }return str; }
arch/x86/pci/i386.c +6 -4
@@ -169,10 +169,12 @@ static void __init pcibios_allocate_resources(int pass) (unsigned long long ) r->start, (unsigned long long ) r->end, r->flags, enabled, pass); - pr =pci_find_parent_resource (dev, r); -if (pr && !request_resource (pr, r)) -continue ; -dev_err (&dev->dev,"BAR %d: can't allocate resource\n" , idx); +if (!pcibios_resource_needs_fixup (dev, idx)) { + pr =pci_find_parent_resource (dev, r); +if (pr && !request_resource (pr, r)) +continue ; +dev_err (&dev->dev,"BAR %d: can't allocate resource\n" , idx); + }/* We'll assign a new address later */ r->end -= r->start; r->start =0 ;
arch/x86/pci/pci.h +3 -0
@@ -117,6 +117,9 @@ extern int __init pcibios_init(void);extern int __init pci_mmcfg_arch_init (void );extern void __init pci_mmcfg_arch_free (void ); +/* pci-common.c */ +extern int pcibios_resource_needs_fixup (struct pci_dev *dev,int resno); +/* * AMD Fam10h CPUs are buggy, and cannot access MMIO config space * on their northbrige except through the * %eax register. As such, you MUST
Signature · 1 line
1.5.6.4
PATCH v6 8/16 PCI: add boot options to reassign resources
In reply to Yu ZhaoThis patch adds boot options so user can reassign device resources of all devices under a bus.
The boot options can be used as:
pci=assign-mmio=0000:01,assign-pio=0000:02
'[dddd:]bb' is the domain and bus number.
--- arch/x86/pci/common.c | 73 +++++++++++++++++++++++++++++++++++++++++++++++++ arch/x86/pci/i386.c | 10 ++++--- arch/x86/pci/pci.h | 3 ++ 3 files changed, 82 insertions(+), 4 deletions(-)
arch/x86/pci/common.c +73 -0
@@ -137,6 +137,72 @@ static void __devinit pcibios_fixup_device_resources(struct pci_dev *dev) } } +static char *pci_assign_pio; +static char *pci_assign_mmio; + +static int pcibios_bus_resource_needs_fixup (struct pci_bus *bus) +{ +int i; +int type =0 ; +int domain, busnr; + +if (!bus->self) +return 0 ; + +for (i =0 ; i <2 ; i++) { +char *str = i ? pci_assign_pio : pci_assign_mmio; + +while (str && *str) { +if (sscanf (str,"%04x:%02x" , &domain, &busnr) !=2 ) { +if (sscanf (str,"%02x" , &busnr) !=1 ) +break ; + domain =0 ; + } + +if (pci_domain_nr (bus) == domain && + bus->number == busnr) { + type |= i ?IORESOURCE_IO :IORESOURCE_MEM ; +break ; + } + + str =strchr (str,';' ); +if (str) + str++; + } + } + +return type; +} + +static void __devinit pcibios_fixup_bus_resources (struct pci_bus *bus) +{ +int i; +int type =pcibios_bus_resource_needs_fixup (bus); + +if (!type) +return ; + +for (i =0 ; i <PCI_BUS_NUM_RESOURCES ; i++) { +struct resource *res = bus->resource[i]; + +if (!res) +continue ; +if (res->flags & type) + res->flags =0 ; + } +} + +int pcibios_resource_needs_fixup (struct pci_dev *dev,int resno) +{ +struct pci_bus *bus; + +for (bus = dev->bus; bus && bus != pci_root_bus; bus = bus->parent) +if (pcibios_bus_resource_needs_fixup (bus)) +return 1 ; + +return 0 ; +} +/* * Called after each bus is probed, but before its children * are examined.
@@ -147,6 +213,7 @@ void __devinit pcibios_fixup_bus(struct pci_bus *b)struct pci_dev *dev;pci_read_bridge_bases (b); +pcibios_fixup_bus_resources (b);list_for_each_entry (dev, &b->devices, bus_list)pcibios_fixup_device_resources (dev); }
@@ -519,6 +586,12 @@ char * __devinit pcibios_setup(char *str) }else if (!strcmp (str,"skip_isa_align" )) { pci_probe |=PCI_CAN_SKIP_ISA_ALIGN ;return NULL ; + }else if (!strncmp (str,"assign-pio=" ,11 )) { + pci_assign_pio = str +11 ; +return NULL ; + }else if (!strncmp (str,"assign-mmio=" ,12 )) { + pci_assign_mmio = str +12 ; +return NULL ; }return str; }
arch/x86/pci/i386.c +6 -4
@@ -169,10 +169,12 @@ static void __init pcibios_allocate_resources(int pass) (unsigned long long ) r->start, (unsigned long long ) r->end, r->flags, enabled, pass); - pr =pci_find_parent_resource (dev, r); -if (pr && !request_resource (pr, r)) -continue ; -dev_err (&dev->dev,"BAR %d: can't allocate resource\n" , idx); +if (!pcibios_resource_needs_fixup (dev, idx)) { + pr =pci_find_parent_resource (dev, r); +if (pr && !request_resource (pr, r)) +continue ; +dev_err (&dev->dev,"BAR %d: can't allocate resource\n" , idx); + }/* We'll assign a new address later */ r->end -= r->start; r->start =0 ;
arch/x86/pci/pci.h +3 -0
@@ -117,6 +117,9 @@ extern int __init pcibios_init(void);extern int __init pci_mmcfg_arch_init (void );extern void __init pci_mmcfg_arch_free (void ); +/* pci-common.c */ +extern int pcibios_resource_needs_fixup (struct pci_dev *dev,int resno); +/* * AMD Fam10h CPUs are buggy, and cannot access MMIO config space * on their northbrige except through the * %eax register. As such, you MUST
Signature · 1 line
1.5.6.4
Re: [PATCH 8/16 v6] PCI: add boot options to reassign resources
Quoted 6 lines · Yu Zhao
This patch adds boot options so user can reassign device resources of all devices under a bus.
The boot options can be used as:
pci=assign-mmio=0000:01,assign-pio=0000:02'[dddd:]bb' is the domain and bus number.
I think this example is incorrect because you look for ";" to separate options, not ",".
Bjorn
Quoted 149 lines
Cc Alex Chiang <[email protected]>Cc Grant Grundler <[email protected]>Cc Greg KH <[email protected]>Cc Ingo Molnar <[email protected]>Cc Jesse Barnes <[email protected]>Cc Matthew Wilcox <[email protected]>Cc Randy Dunlap <[email protected]>Cc Roland Dreier <[email protected]>Signed-off-by Yu Zhao <[email protected]>--- arch/x86/pci/common.c | 73 +++++++++++++++++++++++++++++++++++++++++++++++++ arch/x86/pci/i386.c | 10 ++++--- arch/x86/pci/pci.h | 3 ++ 3 files changed, 82 insertions(+), 4 deletions(-)arch/x86/pci/common.c +73
-0@@ -137,6 +137,72 @@ static void __devinit pcibios_fixup_device_resources(struct pci_dev *dev) } } +static char *pci_assign_pio; +static char *pci_assign_mmio; + +static int pcibios_bus_resource_needs_fixup (struct pci_bus *bus) +{ +int i; +int type =0 ; +int domain, busnr; + +if (!bus->self) +return 0 ; + +for (i =0 ; i <2 ; i++) { +char *str = i ? pci_assign_pio : pci_assign_mmio; + +while (str && *str) { +if (sscanf (str,"%04x:%02x" , &domain, &busnr) !=2 ) { +if (sscanf (str,"%02x" , &busnr) !=1 ) +break ; + domain =0 ; + } + +if (pci_domain_nr (bus) == domain && + bus->number == busnr) { + type |= i ?IORESOURCE_IO :IORESOURCE_MEM ; +break ; + } + + str =strchr (str,';' ); +if (str) + str++; + } + } + +return type; +} + +static void __devinit pcibios_fixup_bus_resources (struct pci_bus *bus) +{ +int i; +int type =pcibios_bus_resource_needs_fixup (bus); + +if (!type) +return ; + +for (i =0 ; i <PCI_BUS_NUM_RESOURCES ; i++) { +struct resource *res = bus->resource[i]; + +if (!res) +continue ; +if (res->flags & type) + res->flags =0 ; + } +} + +int pcibios_resource_needs_fixup (struct pci_dev *dev,int resno) +{ +struct pci_bus *bus; + +for (bus = dev->bus; bus && bus != pci_root_bus; bus = bus->parent) +if (pcibios_bus_resource_needs_fixup (bus)) +return 1 ; + +return 0 ; +} +/* * Called after each bus is probed, but before its children * are examined. @@ -147,6 +213,7 @@ void __devinit pcibios_fixup_bus(struct pci_bus *b)struct pci_dev *dev;pci_read_bridge_bases (b); +pcibios_fixup_bus_resources (b);list_for_each_entry (dev, &b->devices, bus_list)pcibios_fixup_device_resources (dev); }@@ -519,6 +586,12 @@ char * __devinit pcibios_setup(char *str) }else if (!strcmp (str,"skip_isa_align" )) { pci_probe |=PCI_CAN_SKIP_ISA_ALIGN ;return NULL ; + }else if (!strncmp (str,"assign-pio=" ,11 )) { + pci_assign_pio = str +11 ; +return NULL ; + }else if (!strncmp (str,"assign-mmio=" ,12 )) { + pci_assign_mmio = str +12 ; +return NULL ; }return str; }arch/x86/pci/i386.c +6
-4@@ -169,10 +169,12 @@ static void __init pcibios_allocate_resources(int pass) (unsigned long long ) r->start, (unsigned long long ) r->end, r->flags, enabled, pass); - pr =pci_find_parent_resource (dev, r); -if (pr && !request_resource (pr, r)) -continue ; -dev_err (&dev->dev,"BAR %d: can't allocate resource\n" , idx); +if (!pcibios_resource_needs_fixup (dev, idx)) { + pr =pci_find_parent_resource (dev, r); +if (pr && !request_resource (pr, r)) +continue ; +dev_err (&dev->dev,"BAR %d: can't allocate resource\n" , idx); + }/* We'll assign a new address later */ r->end -= r->start; r->start =0 ;arch/x86/pci/pci.h +3
-0@@ -117,6 +117,9 @@ extern int __init pcibios_init(void);extern int __init pci_mmcfg_arch_init (void );extern void __init pci_mmcfg_arch_free (void ); +/* pci-common.c */ +extern int pcibios_resource_needs_fixup (struct pci_dev *dev,int resno); +/* * AMD Fam10h CPUs are buggy, and cannot access MMIO config space * on their northbrige except through the * %eax register. As such, you MUST
Re: [PATCH 8/16 v6] PCI: add boot options to reassign resources
Quoted 10 lines · Bjorn Helgaas
Quoted 6 lines · Yu Zhao
This patch adds boot options so user can reassign device resources of all devices under a bus.
The boot options can be used as:
pci=assign-mmio=0000:01,assign-pio=0000:02'[dddd:]bb' is the domain and bus number.
I think this example is incorrect because you look for ";" to separate options, not ",".
The semicolon is used to separate multiple parameters for assign-mmio and assign-pio. E.g., 'pci=assign-mmio=0000:01;0001:02;0004:03'. And the comma separates different parameters for 'pci='.
Re: [PATCH 8/16 v6] PCI: add boot options to reassign resources
In reply to Bjorn HelgaasQuoted 10 lines · Bjorn Helgaas
Quoted 6 lines · Yu Zhao
This patch adds boot options so user can reassign device resources of all devices under a bus.
The boot options can be used as:
pci=assign-mmio=0000:01,assign-pio=0000:02'[dddd:]bb' is the domain and bus number.
I think this example is incorrect because you look for ";" to separate options, not ",".
The semicolon is used to separate multiple parameters for assign-mmio and assign-pio. E.g., 'pci=assign-mmio=0000:01;0001:02;0004:03'. And the comma separates different parameters for 'pci='.
Re: [PATCH 8/16 v6] PCI: add boot options to reassign resources
In reply to Yu ZhaoQuoted 6 lines · Yu Zhao
This patch adds boot options so user can reassign device resources of all devices under a bus.
The boot options can be used as:
pci=assign-mmio=0000:01,assign-pio=0000:02'[dddd:]bb' is the domain and bus number.
I think this example is incorrect because you look for ";" to separate options, not ",".
Bjorn
Quoted 149 lines
Cc Alex Chiang <[email protected]>Cc Grant Grundler <[email protected]>Cc Greg KH <[email protected]>Cc Ingo Molnar <[email protected]>Cc Jesse Barnes <[email protected]>Cc Matthew Wilcox <[email protected]>Cc Randy Dunlap <[email protected]>Cc Roland Dreier <[email protected]>Signed-off-by Yu Zhao <[email protected]>--- arch/x86/pci/common.c | 73 +++++++++++++++++++++++++++++++++++++++++++++++++ arch/x86/pci/i386.c | 10 ++++--- arch/x86/pci/pci.h | 3 ++ 3 files changed, 82 insertions(+), 4 deletions(-)arch/x86/pci/common.c +73
-0@@ -137,6 +137,72 @@ static void __devinit pcibios_fixup_device_resources(struct pci_dev *dev) } } +static char *pci_assign_pio; +static char *pci_assign_mmio; + +static int pcibios_bus_resource_needs_fixup (struct pci_bus *bus) +{ +int i; +int type =0 ; +int domain, busnr; + +if (!bus->self) +return 0 ; + +for (i =0 ; i <2 ; i++) { +char *str = i ? pci_assign_pio : pci_assign_mmio; + +while (str && *str) { +if (sscanf (str,"%04x:%02x" , &domain, &busnr) !=2 ) { +if (sscanf (str,"%02x" , &busnr) !=1 ) +break ; + domain =0 ; + } + +if (pci_domain_nr (bus) == domain && + bus->number == busnr) { + type |= i ?IORESOURCE_IO :IORESOURCE_MEM ; +break ; + } + + str =strchr (str,';' ); +if (str) + str++; + } + } + +return type; +} + +static void __devinit pcibios_fixup_bus_resources (struct pci_bus *bus) +{ +int i; +int type =pcibios_bus_resource_needs_fixup (bus); + +if (!type) +return ; + +for (i =0 ; i <PCI_BUS_NUM_RESOURCES ; i++) { +struct resource *res = bus->resource[i]; + +if (!res) +continue ; +if (res->flags & type) + res->flags =0 ; + } +} + +int pcibios_resource_needs_fixup (struct pci_dev *dev,int resno) +{ +struct pci_bus *bus; + +for (bus = dev->bus; bus && bus != pci_root_bus; bus = bus->parent) +if (pcibios_bus_resource_needs_fixup (bus)) +return 1 ; + +return 0 ; +} +/* * Called after each bus is probed, but before its children * are examined. @@ -147,6 +213,7 @@ void __devinit pcibios_fixup_bus(struct pci_bus *b)struct pci_dev *dev;pci_read_bridge_bases (b); +pcibios_fixup_bus_resources (b);list_for_each_entry (dev, &b->devices, bus_list)pcibios_fixup_device_resources (dev); }@@ -519,6 +586,12 @@ char * __devinit pcibios_setup(char *str) }else if (!strcmp (str,"skip_isa_align" )) { pci_probe |=PCI_CAN_SKIP_ISA_ALIGN ;return NULL ; + }else if (!strncmp (str,"assign-pio=" ,11 )) { + pci_assign_pio = str +11 ; +return NULL ; + }else if (!strncmp (str,"assign-mmio=" ,12 )) { + pci_assign_mmio = str +12 ; +return NULL ; }return str; }arch/x86/pci/i386.c +6
-4@@ -169,10 +169,12 @@ static void __init pcibios_allocate_resources(int pass) (unsigned long long ) r->start, (unsigned long long ) r->end, r->flags, enabled, pass); - pr =pci_find_parent_resource (dev, r); -if (pr && !request_resource (pr, r)) -continue ; -dev_err (&dev->dev,"BAR %d: can't allocate resource\n" , idx); +if (!pcibios_resource_needs_fixup (dev, idx)) { + pr =pci_find_parent_resource (dev, r); +if (pr && !request_resource (pr, r)) +continue ; +dev_err (&dev->dev,"BAR %d: can't allocate resource\n" , idx); + }/* We'll assign a new address later */ r->end -= r->start; r->start =0 ;arch/x86/pci/pci.h +3
-0@@ -117,6 +117,9 @@ extern int __init pcibios_init(void);extern int __init pci_mmcfg_arch_init (void );extern void __init pci_mmcfg_arch_free (void ); +/* pci-common.c */ +extern int pcibios_resource_needs_fixup (struct pci_dev *dev,int resno); +/* * AMD Fam10h CPUs are buggy, and cannot access MMIO config space * on their northbrige except through the * %eax register. As such, you MUST
PATCH v6 9/16 PCI: add boot option to align MMIO resources
In reply to Yu ZhaoThis patch adds boot option to align MMIO resource for a device. The alignment is a bigger value between the PAGE_SIZE and the resource size.
The boot option can be used as:
pci=align-mmio=0000:01:02.3
'[0000:]01:02.3' is the domain, bus, device and function number of the device.
--- arch/x86/pci/common.c | 37 +++++++++++++++++++++++++++++++++++++ drivers/pci/pci.c | 20 ++++++++++++++++++-- include/linux/pci.h | 1 + 3 files changed, 56 insertions(+), 2 deletions(-)
arch/x86/pci/common.c +37 -0
@@ -139,6 +139,7 @@ static void __devinit pcibios_fixup_device_resources(struct pci_dev *dev)static char *pci_assign_pio;static char *pci_assign_mmio; +static char *pci_align_mmio;static int pcibios_bus_resource_needs_fixup (struct pci_bus *bus) {
@@ -192,6 +193,36 @@ static void __devinit pcibios_fixup_bus_resources(struct pci_bus *bus) } } +int pcibios_resource_alignment (struct pci_dev *dev,int resno) +{ +int domain, busnr, slot, func; +char *str = pci_align_mmio; + +if (dev->resource[resno].flags &IORESOURCE_IO ) +return 0 ; + +while (str && *str) { +if (sscanf (str,"%04x:%02x:%02x.%d" , + &domain, &busnr, &slot, &func) !=4 ) { +if (sscanf (str,"%02x:%02x.%d" , + &busnr, &slot, &func) !=3 ) +break ; + domain =0 ; + } + +if (pci_domain_nr (dev->bus) == domain && + dev->bus->number == busnr && + dev->devfn ==PCI_DEVFN (slot, func)) +return PAGE_SIZE ; + + str =strchr (str,';' ); +if (str) + str++; + } + +return 0 ; +} +int pcibios_resource_needs_fixup (struct pci_dev *dev,int resno) {struct pci_bus *bus;
@@ -200,6 +231,9 @@ int pcibios_resource_needs_fixup(struct pci_dev *dev, int resno)if (pcibios_bus_resource_needs_fixup (bus))return 1 ; +if (pcibios_resource_alignment (dev, resno)) +return 1 ; +return 0 ; }
@@ -592,6 +626,9 @@ char * __devinit pcibios_setup(char *str) }else if (!strncmp (str,"assign-mmio=" ,12 )) { pci_assign_mmio = str +12 ;return NULL ; + }else if (!strncmp (str,"align-mmio=" ,11 )) { + pci_align_mmio = str +11 ; +return NULL ; }return str; }
drivers/pci/pci.c +18 -2
@@ -1015,6 +1015,20 @@ int __attribute__ ((weak)) pcibios_set_pcie_reset_state(struct pci_dev *dev, }/** +* pcibios_resource_alignment - get resource alignment requirement +* @dev: the PCI device +* @resno: resource number +* +* Queries the resource alignment from PCI low level code. Returns positive +* if there is alignment requirement of the resource, or 0 otherwise. +*/ +int __attribute__ ((weak))pcibios_resource_alignment (struct pci_dev *dev, +int resno) +{ +return 0 ; +} + +/** * pci_set_pcie_reset_state - set reset state for device dev * @dev: the PCI-E device reset * @state: Reset state to enter into
@@ -1913,12 +1927,14 @@ int pci_select_bars(struct pci_dev *dev, unsigned long flags)*/ int pci_resource_alignment (struct pci_dev *dev,int resno) { -resource_size_t align; +resource_size_t align, bios_align;struct resource *res = dev->resource + resno; + bios_align =pcibios_resource_alignment (dev, resno); + align =resource_alignment (res);if (align) -return align; +return align > bios_align ? align : bios_align;dev_err (&dev->dev,"alignment: invalid resource #%d\n" , resno);return 0 ;
include/linux/pci.h +1 -0
@@ -1121,6 +1121,7 @@ int pcibios_add_platform_entries(struct pci_dev *dev);void pcibios_disable_device (struct pci_dev *dev);int pcibios_set_pcie_reset_state (struct pci_dev *dev,enum pcie_reset_state state); +int pcibios_resource_alignment (struct pci_dev *dev,int resno);#ifdef CONFIG_PCI_MMCONFIG extern void __init pci_mmcfg_early_init (void );
Signature · 1 line
1.5.6.4
Re: [PATCH 9/16 v6] PCI: add boot option to align MMIO resources
Quoted 3 lines · Yu Zhao
This patch adds boot option to align MMIO resource for a device. The alignment is a bigger value between the
PAGE_SIZEand the resource size.
It looks like this forces alignment on PAGE_SIZE, not "a bigger value between the PAGE_SIZE and the resource size." Can you clarify the changelog to specify exactly what alignment this option forces?
Quoted 4 lines
The boot option can be used as:
pci=align-mmio=0000:01:02.3'[0000:]01:02.3' is the domain, bus, device and function number of the device.
I think you also support using multiple "align-mmio=DDDD:BB:dd.f" options separated by ";", but I had to read the code to figure that out. Can you give an example of this in the changelog and the kernel-parameters.txt patch?
Bjorn
Quoted 139 lines
Cc Alex Chiang <[email protected]>Cc Grant Grundler <[email protected]>Cc Greg KH <[email protected]>Cc Ingo Molnar <[email protected]>Cc Jesse Barnes <[email protected]>Cc Matthew Wilcox <[email protected]>Cc Randy Dunlap <[email protected]>Cc Roland Dreier <[email protected]>Signed-off-by Yu Zhao <[email protected]>--- arch/x86/pci/common.c | 37 +++++++++++++++++++++++++++++++++++++ drivers/pci/pci.c | 20 ++++++++++++++++++-- include/linux/pci.h | 1 + 3 files changed, 56 insertions(+), 2 deletions(-)arch/x86/pci/common.c +37
-0@@ -139,6 +139,7 @@ static void __devinit pcibios_fixup_device_resources(struct pci_dev *dev)static char *pci_assign_pio;static char *pci_assign_mmio; +static char *pci_align_mmio;static int pcibios_bus_resource_needs_fixup (struct pci_bus *bus) {@@ -192,6 +193,36 @@ static void __devinit pcibios_fixup_bus_resources(struct pci_bus *bus) } } +int pcibios_resource_alignment (struct pci_dev *dev,int resno) +{ +int domain, busnr, slot, func; +char *str = pci_align_mmio; + +if (dev->resource[resno].flags &IORESOURCE_IO ) +return 0 ; + +while (str && *str) { +if (sscanf (str,"%04x:%02x:%02x.%d" , + &domain, &busnr, &slot, &func) !=4 ) { +if (sscanf (str,"%02x:%02x.%d" , + &busnr, &slot, &func) !=3 ) +break ; + domain =0 ; + } + +if (pci_domain_nr (dev->bus) == domain && + dev->bus->number == busnr && + dev->devfn ==PCI_DEVFN (slot, func)) +return PAGE_SIZE ; + + str =strchr (str,';' ); +if (str) + str++; + } + +return 0 ; +} +int pcibios_resource_needs_fixup (struct pci_dev *dev,int resno) {struct pci_bus *bus;@@ -200,6 +231,9 @@ int pcibios_resource_needs_fixup(struct pci_dev *dev, int resno)if (pcibios_bus_resource_needs_fixup (bus))return 1 ; +if (pcibios_resource_alignment (dev, resno)) +return 1 ; +return 0 ; }@@ -592,6 +626,9 @@ char * __devinit pcibios_setup(char *str) }else if (!strncmp (str,"assign-mmio=" ,12 )) { pci_assign_mmio = str +12 ;return NULL ; + }else if (!strncmp (str,"align-mmio=" ,11 )) { + pci_align_mmio = str +11 ; +return NULL ; }return str; }drivers/pci/pci.c +18
-2@@ -1015,6 +1015,20 @@ int __attribute__ ((weak)) pcibios_set_pcie_reset_state(struct pci_dev *dev, }/** +* pcibios_resource_alignment - get resource alignment requirement +* @dev: the PCI device +* @resno: resource number +* +* Queries the resource alignment from PCI low level code. Returns positive +* if there is alignment requirement of the resource, or 0 otherwise. +*/ +int __attribute__ ((weak))pcibios_resource_alignment (struct pci_dev *dev, +int resno) +{ +return 0 ; +} + +/** * pci_set_pcie_reset_state - set reset state for device dev * @dev: the PCI-E device reset * @state: Reset state to enter into @@ -1913,12 +1927,14 @@ int pci_select_bars(struct pci_dev *dev, unsigned long flags)*/ int pci_resource_alignment (struct pci_dev *dev,int resno) { -resource_size_t align; +resource_size_t align, bios_align;struct resource *res = dev->resource + resno; + bios_align =pcibios_resource_alignment (dev, resno); + align =resource_alignment (res);if (align) -return align; +return align > bios_align ? align : bios_align;dev_err (&dev->dev,"alignment: invalid resource #%d\n" , resno);return 0 ;include/linux/pci.h +1
-0@@ -1121,6 +1121,7 @@ int pcibios_add_platform_entries(struct pci_dev *dev);void pcibios_disable_device (struct pci_dev *dev);int pcibios_set_pcie_reset_state (struct pci_dev *dev,enum pcie_reset_state state); +int pcibios_resource_alignment (struct pci_dev *dev,int resno);#ifdef CONFIG_PCI_MMCONFIG extern void __init pci_mmcfg_early_init (void );
Re: [PATCH 9/16 v6] PCI: add boot option to align MMIO resources
Quoted 9 lines · Bjorn Helgaas
Quoted 3 lines · Yu Zhao
This patch adds boot option to align MMIO resource for a device. The alignment is a bigger value between the
PAGE_SIZEand the resource size.It looks like this forces alignment on
PAGE_SIZE, not "a bigger value between thePAGE_SIZEand the resource size." Can you clarify the changelog to specify exactly what alignment this option forces?
I guess following would explain your question.
Quoted 15 lines
Quoted 15 lines
int
pci_resource_alignment(struct pci_dev *dev, int resno){drivers/pci/pci.c +4
-2-resource_size_t align; +resource_size_t align, bios_align;struct resource *res = dev->resource + resno; + bios_align =pcibios_resource_alignment (dev, resno); + align =resource_alignment (res);if (align) -return align; +return align > bios_align ? align : bios_align;dev_err (&dev->dev,"alignment: invalid resource #%d\n" , resno);return 0 ;
Re: [PATCH 9/16 v6] PCI: add boot option to align MMIO resources
In reply to Bjorn HelgaasQuoted 9 lines · Bjorn Helgaas
Quoted 3 lines · Yu Zhao
This patch adds boot option to align MMIO resource for a device. The alignment is a bigger value between the
PAGE_SIZEand the resource size.It looks like this forces alignment on
PAGE_SIZE, not "a bigger value between thePAGE_SIZEand the resource size." Can you clarify the changelog to specify exactly what alignment this option forces?
I guess following would explain your question.
Quoted 15 lines
Quoted 15 lines
int
pci_resource_alignment(struct pci_dev *dev, int resno){drivers/pci/pci.c +4
-2-resource_size_t align; +resource_size_t align, bios_align;struct resource *res = dev->resource + resno; + bios_align =pcibios_resource_alignment (dev, resno); + align =resource_alignment (res);if (align) -return align; +return align > bios_align ? align : bios_align;dev_err (&dev->dev,"alignment: invalid resource #%d\n" , resno);return 0 ;
Re: [PATCH 9/16 v6] PCI: add boot option to align MMIO resources
In reply to Yu ZhaoQuoted 3 lines · Yu Zhao
This patch adds boot option to align MMIO resource for a device. The alignment is a bigger value between the
PAGE_SIZEand the resource size.
It looks like this forces alignment on PAGE_SIZE, not "a bigger value between the PAGE_SIZE and the resource size." Can you clarify the changelog to specify exactly what alignment this option forces?
Quoted 4 lines
The boot option can be used as:
pci=align-mmio=0000:01:02.3'[0000:]01:02.3' is the domain, bus, device and function number of the device.
I think you also support using multiple "align-mmio=DDDD:BB:dd.f" options separated by ";", but I had to read the code to figure that out. Can you give an example of this in the changelog and the kernel-parameters.txt patch?
Bjorn
Quoted 139 lines
Cc Alex Chiang <[email protected]>Cc Grant Grundler <[email protected]>Cc Greg KH <[email protected]>Cc Ingo Molnar <[email protected]>Cc Jesse Barnes <[email protected]>Cc Matthew Wilcox <[email protected]>Cc Randy Dunlap <[email protected]>Cc Roland Dreier <[email protected]>Signed-off-by Yu Zhao <[email protected]>--- arch/x86/pci/common.c | 37 +++++++++++++++++++++++++++++++++++++ drivers/pci/pci.c | 20 ++++++++++++++++++-- include/linux/pci.h | 1 + 3 files changed, 56 insertions(+), 2 deletions(-)arch/x86/pci/common.c +37
-0@@ -139,6 +139,7 @@ static void __devinit pcibios_fixup_device_resources(struct pci_dev *dev)static char *pci_assign_pio;static char *pci_assign_mmio; +static char *pci_align_mmio;static int pcibios_bus_resource_needs_fixup (struct pci_bus *bus) {@@ -192,6 +193,36 @@ static void __devinit pcibios_fixup_bus_resources(struct pci_bus *bus) } } +int pcibios_resource_alignment (struct pci_dev *dev,int resno) +{ +int domain, busnr, slot, func; +char *str = pci_align_mmio; + +if (dev->resource[resno].flags &IORESOURCE_IO ) +return 0 ; + +while (str && *str) { +if (sscanf (str,"%04x:%02x:%02x.%d" , + &domain, &busnr, &slot, &func) !=4 ) { +if (sscanf (str,"%02x:%02x.%d" , + &busnr, &slot, &func) !=3 ) +break ; + domain =0 ; + } + +if (pci_domain_nr (dev->bus) == domain && + dev->bus->number == busnr && + dev->devfn ==PCI_DEVFN (slot, func)) +return PAGE_SIZE ; + + str =strchr (str,';' ); +if (str) + str++; + } + +return 0 ; +} +int pcibios_resource_needs_fixup (struct pci_dev *dev,int resno) {struct pci_bus *bus;@@ -200,6 +231,9 @@ int pcibios_resource_needs_fixup(struct pci_dev *dev, int resno)if (pcibios_bus_resource_needs_fixup (bus))return 1 ; +if (pcibios_resource_alignment (dev, resno)) +return 1 ; +return 0 ; }@@ -592,6 +626,9 @@ char * __devinit pcibios_setup(char *str) }else if (!strncmp (str,"assign-mmio=" ,12 )) { pci_assign_mmio = str +12 ;return NULL ; + }else if (!strncmp (str,"align-mmio=" ,11 )) { + pci_align_mmio = str +11 ; +return NULL ; }return str; }drivers/pci/pci.c +18
-2@@ -1015,6 +1015,20 @@ int __attribute__ ((weak)) pcibios_set_pcie_reset_state(struct pci_dev *dev, }/** +* pcibios_resource_alignment - get resource alignment requirement +* @dev: the PCI device +* @resno: resource number +* +* Queries the resource alignment from PCI low level code. Returns positive +* if there is alignment requirement of the resource, or 0 otherwise. +*/ +int __attribute__ ((weak))pcibios_resource_alignment (struct pci_dev *dev, +int resno) +{ +return 0 ; +} + +/** * pci_set_pcie_reset_state - set reset state for device dev * @dev: the PCI-E device reset * @state: Reset state to enter into @@ -1913,12 +1927,14 @@ int pci_select_bars(struct pci_dev *dev, unsigned long flags)*/ int pci_resource_alignment (struct pci_dev *dev,int resno) { -resource_size_t align; +resource_size_t align, bios_align;struct resource *res = dev->resource + resno; + bios_align =pcibios_resource_alignment (dev, resno); + align =resource_alignment (res);if (align) -return align; +return align > bios_align ? align : bios_align;dev_err (&dev->dev,"alignment: invalid resource #%d\n" , resno);return 0 ;include/linux/pci.h +1
-0@@ -1121,6 +1121,7 @@ int pcibios_add_platform_entries(struct pci_dev *dev);void pcibios_disable_device (struct pci_dev *dev);int pcibios_set_pcie_reset_state (struct pci_dev *dev,enum pcie_reset_state state); +int pcibios_resource_alignment (struct pci_dev *dev,int resno);#ifdef CONFIG_PCI_MMCONFIG extern void __init pci_mmcfg_early_init (void );
PATCH v6 9/16 PCI: add boot option to align MMIO resources
In reply to Yu ZhaoThis patch adds boot option to align MMIO resource for a device. The alignment is a bigger value between the PAGE_SIZE and the resource size.
The boot option can be used as:
pci=align-mmio=0000:01:02.3
'[0000:]01:02.3' is the domain, bus, device and function number of the device.
--- arch/x86/pci/common.c | 37 +++++++++++++++++++++++++++++++++++++ drivers/pci/pci.c | 20 ++++++++++++++++++-- include/linux/pci.h | 1 + 3 files changed, 56 insertions(+), 2 deletions(-)
arch/x86/pci/common.c +37 -0
@@ -139,6 +139,7 @@ static void __devinit pcibios_fixup_device_resources(struct pci_dev *dev)static char *pci_assign_pio;static char *pci_assign_mmio; +static char *pci_align_mmio;static int pcibios_bus_resource_needs_fixup (struct pci_bus *bus) {
@@ -192,6 +193,36 @@ static void __devinit pcibios_fixup_bus_resources(struct pci_bus *bus) } } +int pcibios_resource_alignment (struct pci_dev *dev,int resno) +{ +int domain, busnr, slot, func; +char *str = pci_align_mmio; + +if (dev->resource[resno].flags &IORESOURCE_IO ) +return 0 ; + +while (str && *str) { +if (sscanf (str,"%04x:%02x:%02x.%d" , + &domain, &busnr, &slot, &func) !=4 ) { +if (sscanf (str,"%02x:%02x.%d" , + &busnr, &slot, &func) !=3 ) +break ; + domain =0 ; + } + +if (pci_domain_nr (dev->bus) == domain && + dev->bus->number == busnr && + dev->devfn ==PCI_DEVFN (slot, func)) +return PAGE_SIZE ; + + str =strchr (str,';' ); +if (str) + str++; + } + +return 0 ; +} +int pcibios_resource_needs_fixup (struct pci_dev *dev,int resno) {struct pci_bus *bus;
@@ -200,6 +231,9 @@ int pcibios_resource_needs_fixup(struct pci_dev *dev, int resno)if (pcibios_bus_resource_needs_fixup (bus))return 1 ; +if (pcibios_resource_alignment (dev, resno)) +return 1 ; +return 0 ; }
@@ -592,6 +626,9 @@ char * __devinit pcibios_setup(char *str) }else if (!strncmp (str,"assign-mmio=" ,12 )) { pci_assign_mmio = str +12 ;return NULL ; + }else if (!strncmp (str,"align-mmio=" ,11 )) { + pci_align_mmio = str +11 ; +return NULL ; }return str; }
drivers/pci/pci.c +18 -2
@@ -1015,6 +1015,20 @@ int __attribute__ ((weak)) pcibios_set_pcie_reset_state(struct pci_dev *dev, }/** +* pcibios_resource_alignment - get resource alignment requirement +* @dev: the PCI device +* @resno: resource number +* +* Queries the resource alignment from PCI low level code. Returns positive +* if there is alignment requirement of the resource, or 0 otherwise. +*/ +int __attribute__ ((weak))pcibios_resource_alignment (struct pci_dev *dev, +int resno) +{ +return 0 ; +} + +/** * pci_set_pcie_reset_state - set reset state for device dev * @dev: the PCI-E device reset * @state: Reset state to enter into
@@ -1913,12 +1927,14 @@ int pci_select_bars(struct pci_dev *dev, unsigned long flags)*/ int pci_resource_alignment (struct pci_dev *dev,int resno) { -resource_size_t align; +resource_size_t align, bios_align;struct resource *res = dev->resource + resno; + bios_align =pcibios_resource_alignment (dev, resno); + align =resource_alignment (res);if (align) -return align; +return align > bios_align ? align : bios_align;dev_err (&dev->dev,"alignment: invalid resource #%d\n" , resno);return 0 ;
include/linux/pci.h +1 -0
@@ -1121,6 +1121,7 @@ int pcibios_add_platform_entries(struct pci_dev *dev);void pcibios_disable_device (struct pci_dev *dev);int pcibios_set_pcie_reset_state (struct pci_dev *dev,enum pcie_reset_state state); +int pcibios_resource_alignment (struct pci_dev *dev,int resno);#ifdef CONFIG_PCI_MMCONFIG extern void __init pci_mmcfg_early_init (void );
Signature · 1 line
1.5.6.4
PATCH v6 10/16 PCI: cleanup pci_bus_add_devices()
In reply to Yu ZhaoThis cleanup makes pci_bus_add_devices() easier to read.
--- drivers/pci/bus.c | 56 +++++++++++++++++++++++++------------------------ drivers/pci/remove.c | 2 + 2 files changed, 31 insertions(+), 27 deletions(-)
drivers/pci/bus.c +29 -27
@@ -71,7 +71,7 @@ pci_bus_alloc_resource(struct pci_bus *bus, struct resource *res, }/** -* add a single device +* pci_bus_add_device - add a single device * @dev: device to add * * This adds a single pci device to the global
@@ -105,7 +105,7 @@ int pci_bus_add_device(struct pci_dev *dev)void pci_bus_add_devices (struct pci_bus *bus) {struct pci_dev *dev; -struct pci_bus *child_bus; +struct pci_bus *child;int retval;list_for_each_entry (dev, &bus->devices, bus_list) {
@@ -120,39 +120,41 @@ void pci_bus_add_devices(struct pci_bus *bus)list_for_each_entry (dev, &bus->devices, bus_list) {BUG_ON (!dev->is_added); + child = dev->subordinate;/* * If there is an unattached subordinate bus, attach * it and then scan for unattached PCI devices. */ -if (dev->subordinate) { -if (list_empty (&dev->subordinate->node)) { -down_write (&pci_bus_sem); -list_add_tail (&dev->subordinate->node, - &dev->bus->children); -up_write (&pci_bus_sem); - } -pci_bus_add_devices (dev->subordinate); - -/* register the bus with sysfs as the parent is now -* properly registered. */ - child_bus = dev->subordinate; -if (child_bus->is_added) -continue ; - child_bus->dev.parent = child_bus->bridge; - retval =device_register (&child_bus->dev); -if (retval) -dev_err (&dev->dev,"Error registering pci_bus," -" continuing...\n" ); -else { - child_bus->is_added =1 ; - retval =device_create_file (&child_bus->dev, - &dev_attr_cpuaffinity); - } +if (!child) +continue ; +if (list_empty (&child->node)) { +down_write (&pci_bus_sem); +list_add_tail (&child->node, + &dev->bus->children); +up_write (&pci_bus_sem); + } +pci_bus_add_devices (child); + +/* +* register the bus with sysfs as the parent is now +* properly registered. +*/ +if (child->is_added) +continue ; + child->dev.parent = child->bridge; + retval =device_register (&child->dev); +if (retval) +dev_err (&dev->dev,"Error registering pci_bus," +" continuing...\n" ); +else { + child->is_added =1 ; + retval =device_create_file (&child->dev, + &dev_attr_cpuaffinity);if (retval)dev_err (&dev->dev,"Error creating cpuaffinity" " file, continuing...\n" ); - retval =device_create_file (&child_bus->dev, + retval =device_create_file (&child->dev, &dev_attr_cpulistaffinity);if (retval) dev_err(&dev->dev,
drivers/pci/remove.c +2 -0
@@ -72,6 +72,8 @@ void pci_remove_bus(struct pci_bus *pci_bus)list_del (&pci_bus->node);up_write (&pci_bus_sem);pci_remove_legacy_files (pci_bus); +if (!pci_bus->is_added) +return ;device_remove_file (&pci_bus->dev, &dev_attr_cpuaffinity);device_remove_file (&pci_bus->dev, &dev_attr_cpulistaffinity);device_unregister (&pci_bus->dev);
Signature · 1 line
1.5.6.4
PATCH v6 10/16 PCI: cleanup pci_bus_add_devices()
In reply to Yu ZhaoThis cleanup makes pci_bus_add_devices() easier to read.
--- drivers/pci/bus.c | 56 +++++++++++++++++++++++++------------------------ drivers/pci/remove.c | 2 + 2 files changed, 31 insertions(+), 27 deletions(-)
drivers/pci/bus.c +29 -27
@@ -71,7 +71,7 @@ pci_bus_alloc_resource(struct pci_bus *bus, struct resource *res, }/** -* add a single device +* pci_bus_add_device - add a single device * @dev: device to add * * This adds a single pci device to the global
@@ -105,7 +105,7 @@ int pci_bus_add_device(struct pci_dev *dev)void pci_bus_add_devices (struct pci_bus *bus) {struct pci_dev *dev; -struct pci_bus *child_bus; +struct pci_bus *child;int retval;list_for_each_entry (dev, &bus->devices, bus_list) {
@@ -120,39 +120,41 @@ void pci_bus_add_devices(struct pci_bus *bus)list_for_each_entry (dev, &bus->devices, bus_list) {BUG_ON (!dev->is_added); + child = dev->subordinate;/* * If there is an unattached subordinate bus, attach * it and then scan for unattached PCI devices. */ -if (dev->subordinate) { -if (list_empty (&dev->subordinate->node)) { -down_write (&pci_bus_sem); -list_add_tail (&dev->subordinate->node, - &dev->bus->children); -up_write (&pci_bus_sem); - } -pci_bus_add_devices (dev->subordinate); - -/* register the bus with sysfs as the parent is now -* properly registered. */ - child_bus = dev->subordinate; -if (child_bus->is_added) -continue ; - child_bus->dev.parent = child_bus->bridge; - retval =device_register (&child_bus->dev); -if (retval) -dev_err (&dev->dev,"Error registering pci_bus," -" continuing...\n" ); -else { - child_bus->is_added =1 ; - retval =device_create_file (&child_bus->dev, - &dev_attr_cpuaffinity); - } +if (!child) +continue ; +if (list_empty (&child->node)) { +down_write (&pci_bus_sem); +list_add_tail (&child->node, + &dev->bus->children); +up_write (&pci_bus_sem); + } +pci_bus_add_devices (child); + +/* +* register the bus with sysfs as the parent is now +* properly registered. +*/ +if (child->is_added) +continue ; + child->dev.parent = child->bridge; + retval =device_register (&child->dev); +if (retval) +dev_err (&dev->dev,"Error registering pci_bus," +" continuing...\n" ); +else { + child->is_added =1 ; + retval =device_create_file (&child->dev, + &dev_attr_cpuaffinity);if (retval)dev_err (&dev->dev,"Error creating cpuaffinity" " file, continuing...\n" ); - retval =device_create_file (&child_bus->dev, + retval =device_create_file (&child->dev, &dev_attr_cpulistaffinity);if (retval) dev_err(&dev->dev,
drivers/pci/remove.c +2 -0
@@ -72,6 +72,8 @@ void pci_remove_bus(struct pci_bus *pci_bus)list_del (&pci_bus->node);up_write (&pci_bus_sem);pci_remove_legacy_files (pci_bus); +if (!pci_bus->is_added) +return ;device_remove_file (&pci_bus->dev, &dev_attr_cpuaffinity);device_remove_file (&pci_bus->dev, &dev_attr_cpulistaffinity);device_unregister (&pci_bus->dev);
Signature · 1 line
1.5.6.4
PATCH v6 11/16 PCI: split a new function from pci_bus_add_devices()
In reply to Yu ZhaoThis patch splits a new function from pci_bus_add_devices(). The new function can be used to register PCI bus to the device core and create its sysfs entries.
--- drivers/pci/bus.c | 47 ++++++++++++++++++++++++++++------------------- include/linux/pci.h | 1 + 2 files changed, 29 insertions(+), 19 deletions(-)
drivers/pci/bus.c +28 -19
@@ -91,6 +91,32 @@ int pci_bus_add_device(struct pci_dev *dev) }/** +* pci_bus_add_child - add a child bus +* @bus: bus to add +* +* This adds sysfs entries for a single bus +*/ +int pci_bus_add_child (struct pci_bus *bus) +{ +int retval; + +if (bus->bridge) + bus->dev.parent = bus->bridge; + + retval =device_register (&bus->dev); +if (retval) +return retval; + + bus->is_added =1 ; + + retval =device_create_file (&bus->dev, &dev_attr_cpuaffinity); +if (retval) +return retval; + +return device_create_file (&bus->dev, &dev_attr_cpulistaffinity); +} + +/** * pci_bus_add_devices - insert newly discovered PCI devices * @bus: bus to check for new devices *
@@ -141,26 +167,9 @@ void pci_bus_add_devices(struct pci_bus *bus)*/ if (child->is_added)continue ; - child->dev.parent = child->bridge; - retval =device_register (&child->dev); + retval =pci_bus_add_child (child);if (retval) -dev_err (&dev->dev,"Error registering pci_bus," -" continuing...\n" ); -else { - child->is_added =1 ; - retval =device_create_file (&child->dev, - &dev_attr_cpuaffinity); -if (retval) -dev_err (&dev->dev,"Error creating cpuaffinity" -" file, continuing...\n" ); - - retval =device_create_file (&child->dev, - &dev_attr_cpulistaffinity); -if (retval) -dev_err (&dev->dev, -"Error creating cpulistaffinity" -" file, continuing...\n" ); - } +dev_err (&dev->dev,"Error adding bus, continuing\n" ); } }
include/linux/pci.h +1 -0
@@ -528,6 +528,7 @@ struct pci_dev *pci_scan_single_device(struct pci_bus *bus, int devfn);void pci_device_add (struct pci_dev *dev,struct pci_bus *bus);unsigned int pci_scan_child_bus (struct pci_bus *bus);int __must_check pci_bus_add_device (struct pci_dev *dev); +int pci_bus_add_child (struct pci_bus *bus);void pci_read_bridge_bases (struct pci_bus *child);struct resource *pci_find_parent_resource (const struct pci_dev *dev,struct resource *res);
Signature · 1 line
1.5.6.4
PATCH v6 11/16 PCI: split a new function from pci_bus_add_devices()
In reply to Yu ZhaoThis patch splits a new function from pci_bus_add_devices(). The new function can be used to register PCI bus to the device core and create its sysfs entries.
--- drivers/pci/bus.c | 47 ++++++++++++++++++++++++++++------------------- include/linux/pci.h | 1 + 2 files changed, 29 insertions(+), 19 deletions(-)
drivers/pci/bus.c +28 -19
@@ -91,6 +91,32 @@ int pci_bus_add_device(struct pci_dev *dev) }/** +* pci_bus_add_child - add a child bus +* @bus: bus to add +* +* This adds sysfs entries for a single bus +*/ +int pci_bus_add_child (struct pci_bus *bus) +{ +int retval; + +if (bus->bridge) + bus->dev.parent = bus->bridge; + + retval =device_register (&bus->dev); +if (retval) +return retval; + + bus->is_added =1 ; + + retval =device_create_file (&bus->dev, &dev_attr_cpuaffinity); +if (retval) +return retval; + +return device_create_file (&bus->dev, &dev_attr_cpulistaffinity); +} + +/** * pci_bus_add_devices - insert newly discovered PCI devices * @bus: bus to check for new devices *
@@ -141,26 +167,9 @@ void pci_bus_add_devices(struct pci_bus *bus)*/ if (child->is_added)continue ; - child->dev.parent = child->bridge; - retval =device_register (&child->dev); + retval =pci_bus_add_child (child);if (retval) -dev_err (&dev->dev,"Error registering pci_bus," -" continuing...\n" ); -else { - child->is_added =1 ; - retval =device_create_file (&child->dev, - &dev_attr_cpuaffinity); -if (retval) -dev_err (&dev->dev,"Error creating cpuaffinity" -" file, continuing...\n" ); - - retval =device_create_file (&child->dev, - &dev_attr_cpulistaffinity); -if (retval) -dev_err (&dev->dev, -"Error creating cpulistaffinity" -" file, continuing...\n" ); - } +dev_err (&dev->dev,"Error adding bus, continuing\n" ); } }
include/linux/pci.h +1 -0
@@ -528,6 +528,7 @@ struct pci_dev *pci_scan_single_device(struct pci_bus *bus, int devfn);void pci_device_add (struct pci_dev *dev,struct pci_bus *bus);unsigned int pci_scan_child_bus (struct pci_bus *bus);int __must_check pci_bus_add_device (struct pci_dev *dev); +int pci_bus_add_child (struct pci_bus *bus);void pci_read_bridge_bases (struct pci_bus *child);struct resource *pci_find_parent_resource (const struct pci_dev *dev,struct resource *res);
Signature · 1 line
1.5.6.4
PATCH v6 12/16 PCI: support the SR-IOV capability
In reply to Yu ZhaoSupport Single Root I/O Virtualization (SR-IOV) capability.
--- drivers/pci/Kconfig | 12 + drivers/pci/Makefile | 2 + drivers/pci/iov.c | 592 ++++++++++++++++++++++++++++++++++++++++++++++ drivers/pci/pci-sysfs.c | 4 + drivers/pci/pci.c | 14 + drivers/pci/pci.h | 48 ++++ drivers/pci/probe.c | 4 + include/linux/pci.h | 39 +++ include/linux/pci_regs.h | 21 ++ 9 files changed, 736 insertions(+), 0 deletions(-) create mode 100644 drivers/pci/iov.c
drivers/pci/Kconfig +12 -0
@@ -50,3 +50,15 @@ config HT_IRQ This allows native hypertransport devices to use interrupts. If unsure say Y. + +config PCI_IOV + bool "PCI SR-IOV support" + depends on PCI + select PCI_MSI + default n + help + This option allows device drivers to enable Single Root I/O + Virtualization. Each Virtual Function's PCI configuration + space can be accessed using its own Bus, Device and Function + Number (Routing ID). Each Virtual Function also has PCI Memory + Space, which is used to map its own register set.
drivers/pci/Makefile +2 -0
@@ -55,3 +55,5 @@ obj-$(CONFIG_PCI_SYSCALL) += syscall.o ifeq ($(CONFIG_PCI_DEBUG),y) EXTRA_CFLAGS += -DDEBUG endif + +obj-$(CONFIG_PCI_IOV) += iov.o
drivers/pci/iov.c new file +592 -0
592 lines not on this page · raw message
drivers/pci/pci-sysfs.c +4 -0
@@ -847,6 +847,9 @@ static int pci_create_capabilities_sysfs(struct pci_dev *dev)/* Active State Power Management */ pcie_aspm_create_sysfs_dev_files (dev); +/* Single Root I/O Virtualization */ +pci_iov_create_sysfs (dev); +return 0 ; }
@@ -932,6 +935,7 @@ static void pci_remove_capabilities_sysfs(struct pci_dev *dev) }pcie_aspm_remove_sysfs_dev_files (dev); +pci_iov_remove_sysfs (dev); }/**
drivers/pci/pci.c +14 -0
@@ -1936,6 +1936,13 @@ int pci_resource_alignment(struct pci_dev *dev, int resno)if (align)return align > bios_align ? align : bios_align; +if (resno >PCI_ROM_RESOURCE && resno <PCI_BRIDGE_RESOURCES ) { +/* device specific resource */ + align =pci_iov_resource_align (dev, resno); +if (align) +return align > bios_align ? align : bios_align; + } +dev_err (&dev->dev,"alignment: invalid resource #%d\n" , resno);return 0 ; }
@@ -1950,12 +1957,19 @@ int pci_resource_alignment(struct pci_dev *dev, int resno)*/ int pci_resource_bar (struct pci_dev *dev,int resno,enum pci_bar_type *type) { +int reg; +if (resno <PCI_ROM_RESOURCE ) { *type = pci_bar_unknown;return PCI_BASE_ADDRESS_0 +4 * resno; }else if (resno ==PCI_ROM_RESOURCE ) { *type = pci_bar_mem32;return dev->rom_base_reg; + }else if (resno <PCI_BRIDGE_RESOURCES ) { +/* device specific resource */ + reg =pci_iov_resource_bar (dev, resno, type); +if (reg) +return reg; }dev_err (&dev->dev,"BAR: invalid resource #%d\n" , resno);
drivers/pci/pci.h +48 -0
52 lines not on this page · raw message
drivers/pci/probe.c +4 -0
@@ -779,6 +779,7 @@ static int pci_setup_device(struct pci_dev * dev)static void pci_release_capabilities (struct pci_dev *dev) {pci_vpd_release (dev); +pci_iov_release (dev); }/**
@@ -962,6 +963,9 @@ static void pci_init_capabilities(struct pci_dev *dev)/* Alternative Routing-ID Forwarding */ pci_enable_ari (dev); + +/* Single Root I/O Virtualization */ +pci_iov_init (dev); }void pci_device_add (struct pci_dev *dev,struct pci_bus *bus)
include/linux/pci.h +39 -0
@@ -87,6 +87,12 @@ enum {/* #6: expansion ROM */ PCI_ROM_RESOURCE, +/* device specific resources */ +#ifdef CONFIG_PCI_IOV + PCI_IOV_RESOURCES, + PCI_IOV_RESOURCES_END = PCI_IOV_RESOURCES + PCI_IOV_NUM_BAR - 1, +#endif +/* address space assigned to buses behind the bridge */ #ifndef PCI_BRIDGE_RES_NUM #define PCI_BRIDGE_RES_NUM 4
@@ -165,6 +171,7 @@ struct pci_cap_saved_state {struct pcie_link_state ;struct pci_vpd ; +struct pci_iov ;/* * The pci_dev structure is used to describe PCI devices.
@@ -253,6 +260,7 @@ struct pci_dev {struct list_head msi_list;#endif struct pci_vpd *vpd; +struct pci_iov *iov; };extern struct pci_dev *alloc_pci_dev (void );
@@ -1147,5 +1155,36 @@ static inline void * pci_ioremap_bar(struct pci_dev *pdev, int bar) }#endif +/* SR-IOV events masks */ +#define PCI_IOV_NUM_VIRTFN 0x0000FFFFU/* NumVFs to be set */ +/* SR-IOV events values */ +#define PCI_IOV_ENABLE 0x00010000U/* SR-IOV enable request */ +#define PCI_IOV_DISABLE 0x00020000U/* SR-IOV disable request */ +#define PCI_IOV_NUMVFS 0x00040000U/* SR-IOV disable request */ + +#ifdef CONFIG_PCI_IOV +extern int pci_iov_enable (struct pci_dev *dev,int numvfs); +extern void pci_iov_disable (struct pci_dev *dev); +extern int pci_iov_register (struct pci_dev *dev, +int (*callback)(struct pci_dev *dev,u32 event)); +extern void pci_iov_unregister (struct pci_dev *dev); +#else +static inline int pci_iov_enable (struct pci_dev *dev,int numvfs) +{ +return -EIO ; +} +static inline void pci_iov_disable (struct pci_dev *dev) +{ +} +static inline int pci_iov_register (struct pci_dev *dev, +int (*callback)(struct pci_dev *dev,u32 event)) +{ +return -EIO ; +} +static inline void pci_iov_unregister (struct pci_dev *dev) +{ +} +#endif /* CONFIG_PCI_IOV */ +#endif /* __KERNEL__ */ #endif /* LINUX_PCI_H */
include/linux/pci_regs.h +21 -0
@@ -363,6 +363,7 @@#define PCI_EXP_TYPE_UPSTREAM 0x5/* Upstream Port */ #define PCI_EXP_TYPE_DOWNSTREAM 0x6/* Downstream Port */ #define PCI_EXP_TYPE_PCI_BRIDGE 0x7/* PCI/PCI-X Bridge */ +#define PCI_EXP_TYPE_RC_END 0x9/* Root Complex Integrated Endpoint */ #define PCI_EXP_FLAGS_SLOT 0x0100/* Slot implemented */ #define PCI_EXP_FLAGS_IRQ 0x3e00/* Interrupt message number */ #define PCI_EXP_DEVCAP 4/* Device capabilities */
@@ -434,6 +435,7 @@#define PCI_EXT_CAP_ID_DSN 3#define PCI_EXT_CAP_ID_PWR 4#define PCI_EXT_CAP_ID_ARI 14 +#define PCI_EXT_CAP_ID_IOV 16/* Advanced Error Reporting */ #define PCI_ERR_UNCOR_STATUS 4/* Uncorrectable Error Status */
@@ -551,4 +553,23 @@#define PCI_ARI_CTRL_ACS 0x0002/* ACS Function Groups Enable */ #define PCI_ARI_CTRL_FG (x) (((x) >> 4) & 7)/* Function Group */ +/* Single Root I/O Virtualization */ +#define PCI_IOV_CAP 0x04/* SR-IOV Capabilities */ +#define PCI_IOV_CTRL 0x08/* SR-IOV Control */ +#define PCI_IOV_CTRL_VFE 0x01/* VF Enable */ +#define PCI_IOV_CTRL_MSE 0x08/* VF Memory Space Enable */ +#define PCI_IOV_CTRL_ARI 0x10/* ARI Capable Hierarchy */ +#define PCI_IOV_STATUS 0x0a/* SR-IOV Status */ +#define PCI_IOV_INITIAL_VF 0x0c/* Initial VFs */ +#define PCI_IOV_TOTAL_VF 0x0e/* Total VFs */ +#define PCI_IOV_NUM_VF 0x10/* Number of VFs */ +#define PCI_IOV_FUNC_LINK 0x12/* Function Dependency Link */ +#define PCI_IOV_VF_OFFSET 0x14/* First VF Offset */ +#define PCI_IOV_VF_STRIDE 0x16/* Following VF Stride */ +#define PCI_IOV_VF_DID 0x1a/* VF Device ID */ +#define PCI_IOV_SUP_PGSIZE 0x1c/* Supported Page Sizes */ +#define PCI_IOV_SYS_PGSIZE 0x20/* System Page Size */ +#define PCI_IOV_BAR_0 0x24/* VF BAR0 */ +#define PCI_IOV_NUM_BAR 6/* Number of VF BARs */ +#endif /* LINUX_PCI_REGS_H */
Signature · 1 line
1.5.6.4
PATCH v6 12/16 PCI: support the SR-IOV capability
In reply to Yu ZhaoSupport Single Root I/O Virtualization (SR-IOV) capability.
--- drivers/pci/Kconfig | 12 + drivers/pci/Makefile | 2 + drivers/pci/iov.c | 592 ++++++++++++++++++++++++++++++++++++++++++++++ drivers/pci/pci-sysfs.c | 4 + drivers/pci/pci.c | 14 + drivers/pci/pci.h | 48 ++++ drivers/pci/probe.c | 4 + include/linux/pci.h | 39 +++ include/linux/pci_regs.h | 21 ++ 9 files changed, 736 insertions(+), 0 deletions(-) create mode 100644 drivers/pci/iov.c
drivers/pci/Kconfig +12 -0
@@ -50,3 +50,15 @@ config HT_IRQ This allows native hypertransport devices to use interrupts. If unsure say Y. + +config PCI_IOV + bool "PCI SR-IOV support" + depends on PCI + select PCI_MSI + default n + help + This option allows device drivers to enable Single Root I/O + Virtualization. Each Virtual Function's PCI configuration + space can be accessed using its own Bus, Device and Function + Number (Routing ID). Each Virtual Function also has PCI Memory + Space, which is used to map its own register set.
drivers/pci/Makefile +2 -0
@@ -55,3 +55,5 @@ obj-$(CONFIG_PCI_SYSCALL) += syscall.o ifeq ($(CONFIG_PCI_DEBUG),y) EXTRA_CFLAGS += -DDEBUG endif + +obj-$(CONFIG_PCI_IOV) += iov.o
drivers/pci/iov.c new file +592 -0
592 lines not on this page · raw message
drivers/pci/pci-sysfs.c +4 -0
@@ -847,6 +847,9 @@ static int pci_create_capabilities_sysfs(struct pci_dev *dev)/* Active State Power Management */ pcie_aspm_create_sysfs_dev_files (dev); +/* Single Root I/O Virtualization */ +pci_iov_create_sysfs (dev); +return 0 ; }
@@ -932,6 +935,7 @@ static void pci_remove_capabilities_sysfs(struct pci_dev *dev) }pcie_aspm_remove_sysfs_dev_files (dev); +pci_iov_remove_sysfs (dev); }/**
drivers/pci/pci.c +14 -0
@@ -1936,6 +1936,13 @@ int pci_resource_alignment(struct pci_dev *dev, int resno)if (align)return align > bios_align ? align : bios_align; +if (resno >PCI_ROM_RESOURCE && resno <PCI_BRIDGE_RESOURCES ) { +/* device specific resource */ + align =pci_iov_resource_align (dev, resno); +if (align) +return align > bios_align ? align : bios_align; + } +dev_err (&dev->dev,"alignment: invalid resource #%d\n" , resno);return 0 ; }
@@ -1950,12 +1957,19 @@ int pci_resource_alignment(struct pci_dev *dev, int resno)*/ int pci_resource_bar (struct pci_dev *dev,int resno,enum pci_bar_type *type) { +int reg; +if (resno <PCI_ROM_RESOURCE ) { *type = pci_bar_unknown;return PCI_BASE_ADDRESS_0 +4 * resno; }else if (resno ==PCI_ROM_RESOURCE ) { *type = pci_bar_mem32;return dev->rom_base_reg; + }else if (resno <PCI_BRIDGE_RESOURCES ) { +/* device specific resource */ + reg =pci_iov_resource_bar (dev, resno, type); +if (reg) +return reg; }dev_err (&dev->dev,"BAR: invalid resource #%d\n" , resno);
drivers/pci/pci.h +48 -0
52 lines not on this page · raw message
drivers/pci/probe.c +4 -0
@@ -779,6 +779,7 @@ static int pci_setup_device(struct pci_dev * dev)static void pci_release_capabilities (struct pci_dev *dev) {pci_vpd_release (dev); +pci_iov_release (dev); }/**
@@ -962,6 +963,9 @@ static void pci_init_capabilities(struct pci_dev *dev)/* Alternative Routing-ID Forwarding */ pci_enable_ari (dev); + +/* Single Root I/O Virtualization */ +pci_iov_init (dev); }void pci_device_add (struct pci_dev *dev,struct pci_bus *bus)
include/linux/pci.h +39 -0
@@ -87,6 +87,12 @@ enum {/* #6: expansion ROM */ PCI_ROM_RESOURCE, +/* device specific resources */ +#ifdef CONFIG_PCI_IOV + PCI_IOV_RESOURCES, + PCI_IOV_RESOURCES_END = PCI_IOV_RESOURCES + PCI_IOV_NUM_BAR - 1, +#endif +/* address space assigned to buses behind the bridge */ #ifndef PCI_BRIDGE_RES_NUM #define PCI_BRIDGE_RES_NUM 4
@@ -165,6 +171,7 @@ struct pci_cap_saved_state {struct pcie_link_state ;struct pci_vpd ; +struct pci_iov ;/* * The pci_dev structure is used to describe PCI devices.
@@ -253,6 +260,7 @@ struct pci_dev {struct list_head msi_list;#endif struct pci_vpd *vpd; +struct pci_iov *iov; };extern struct pci_dev *alloc_pci_dev (void );
@@ -1147,5 +1155,36 @@ static inline void * pci_ioremap_bar(struct pci_dev *pdev, int bar) }#endif +/* SR-IOV events masks */ +#define PCI_IOV_NUM_VIRTFN 0x0000FFFFU/* NumVFs to be set */ +/* SR-IOV events values */ +#define PCI_IOV_ENABLE 0x00010000U/* SR-IOV enable request */ +#define PCI_IOV_DISABLE 0x00020000U/* SR-IOV disable request */ +#define PCI_IOV_NUMVFS 0x00040000U/* SR-IOV disable request */ + +#ifdef CONFIG_PCI_IOV +extern int pci_iov_enable (struct pci_dev *dev,int numvfs); +extern void pci_iov_disable (struct pci_dev *dev); +extern int pci_iov_register (struct pci_dev *dev, +int (*callback)(struct pci_dev *dev,u32 event)); +extern void pci_iov_unregister (struct pci_dev *dev); +#else +static inline int pci_iov_enable (struct pci_dev *dev,int numvfs) +{ +return -EIO ; +} +static inline void pci_iov_disable (struct pci_dev *dev) +{ +} +static inline int pci_iov_register (struct pci_dev *dev, +int (*callback)(struct pci_dev *dev,u32 event)) +{ +return -EIO ; +} +static inline void pci_iov_unregister (struct pci_dev *dev) +{ +} +#endif /* CONFIG_PCI_IOV */ +#endif /* __KERNEL__ */ #endif /* LINUX_PCI_H */
include/linux/pci_regs.h +21 -0
@@ -363,6 +363,7 @@#define PCI_EXP_TYPE_UPSTREAM 0x5/* Upstream Port */ #define PCI_EXP_TYPE_DOWNSTREAM 0x6/* Downstream Port */ #define PCI_EXP_TYPE_PCI_BRIDGE 0x7/* PCI/PCI-X Bridge */ +#define PCI_EXP_TYPE_RC_END 0x9/* Root Complex Integrated Endpoint */ #define PCI_EXP_FLAGS_SLOT 0x0100/* Slot implemented */ #define PCI_EXP_FLAGS_IRQ 0x3e00/* Interrupt message number */ #define PCI_EXP_DEVCAP 4/* Device capabilities */
@@ -434,6 +435,7 @@#define PCI_EXT_CAP_ID_DSN 3#define PCI_EXT_CAP_ID_PWR 4#define PCI_EXT_CAP_ID_ARI 14 +#define PCI_EXT_CAP_ID_IOV 16/* Advanced Error Reporting */ #define PCI_ERR_UNCOR_STATUS 4/* Uncorrectable Error Status */
@@ -551,4 +553,23 @@#define PCI_ARI_CTRL_ACS 0x0002/* ACS Function Groups Enable */ #define PCI_ARI_CTRL_FG (x) (((x) >> 4) & 7)/* Function Group */ +/* Single Root I/O Virtualization */ +#define PCI_IOV_CAP 0x04/* SR-IOV Capabilities */ +#define PCI_IOV_CTRL 0x08/* SR-IOV Control */ +#define PCI_IOV_CTRL_VFE 0x01/* VF Enable */ +#define PCI_IOV_CTRL_MSE 0x08/* VF Memory Space Enable */ +#define PCI_IOV_CTRL_ARI 0x10/* ARI Capable Hierarchy */ +#define PCI_IOV_STATUS 0x0a/* SR-IOV Status */ +#define PCI_IOV_INITIAL_VF 0x0c/* Initial VFs */ +#define PCI_IOV_TOTAL_VF 0x0e/* Total VFs */ +#define PCI_IOV_NUM_VF 0x10/* Number of VFs */ +#define PCI_IOV_FUNC_LINK 0x12/* Function Dependency Link */ +#define PCI_IOV_VF_OFFSET 0x14/* First VF Offset */ +#define PCI_IOV_VF_STRIDE 0x16/* Following VF Stride */ +#define PCI_IOV_VF_DID 0x1a/* VF Device ID */ +#define PCI_IOV_SUP_PGSIZE 0x1c/* Supported Page Sizes */ +#define PCI_IOV_SYS_PGSIZE 0x20/* System Page Size */ +#define PCI_IOV_BAR_0 0x24/* VF BAR0 */ +#define PCI_IOV_NUM_BAR 6/* Number of VF BARs */ +#endif /* LINUX_PCI_REGS_H */
Signature · 1 line
1.5.6.4
PATCH v6 13/16 PCI: reserve bus range for SR-IOV device
In reply to Yu ZhaoReserve bus range for SR-IOV at device scanning stage.
--- drivers/pci/iov.c | 24 ++++++++++++++++++++++++ drivers/pci/pci.h | 5 +++++ drivers/pci/probe.c | 3 +++ 3 files changed, 32 insertions(+), 0 deletions(-)
drivers/pci/iov.c +24 -0
@@ -498,6 +498,30 @@ int pci_iov_resource_bar(struct pci_dev *dev, int resno, }/** +* pci_iov_bus_range - find bus range used by SR-IOV capability +* @bus: the PCI bus +* +* Returns max number of buses (exclude current one) used by Virtual +* Functions. +*/ +int pci_iov_bus_range (struct pci_bus *bus) +{ +int max =0 ; +u8 busnr, devfn; +struct pci_dev *dev; + +list_for_each_entry (dev, &bus->devices, bus_list) { +if (!dev->iov) +continue ; +vf_rid (dev, dev->iov->totalvfs -1 , &busnr, &devfn); +if (busnr > max) + max = busnr; + } + +return max ? max - bus->number :0 ; +} + +/** * pci_iov_register - register SR-IOV service * @dev: the PCI device * @callback: callback function for SR-IOV events
drivers/pci/pci.h +5 -0
@@ -204,6 +204,7 @@ void pci_iov_remove_sysfs(struct pci_dev *dev);extern int pci_iov_resource_align (struct pci_dev *dev,int resno);extern int pci_iov_resource_bar (struct pci_dev *dev,int resno,enum pci_bar_type *type); +extern int pci_iov_bus_range (struct pci_bus *bus);#else static inline int pci_iov_init (struct pci_dev *dev) {
@@ -227,6 +228,10 @@ static inline int pci_iov_resource_bar(struct pci_dev *dev, int resno, {return 0 ; } +extern inline int pci_iov_bus_range (struct pci_bus *bus) +{ +return 0 ; +}#endif /* CONFIG_PCI_IOV */ #endif /* DRIVERS_PCI_H */
drivers/pci/probe.c +3 -0
@@ -1068,6 +1068,9 @@ unsigned int __devinit pci_scan_child_bus(struct pci_bus *bus)for (devfn =0 ; devfn <0x100 ; devfn +=8 )pci_scan_slot (bus, devfn); +/* Reserve buses for SR-IOV capability. */ + max +=pci_iov_bus_range (bus); +/* * After performing arch-dependent fixup of the bus, look behind * all PCI-to-PCI bridges on this bus.
Signature · 1 line
1.5.6.4
PATCH v6 13/16 PCI: reserve bus range for SR-IOV device
In reply to Yu ZhaoReserve bus range for SR-IOV at device scanning stage.
--- drivers/pci/iov.c | 24 ++++++++++++++++++++++++ drivers/pci/pci.h | 5 +++++ drivers/pci/probe.c | 3 +++ 3 files changed, 32 insertions(+), 0 deletions(-)
drivers/pci/iov.c +24 -0
@@ -498,6 +498,30 @@ int pci_iov_resource_bar(struct pci_dev *dev, int resno, }/** +* pci_iov_bus_range - find bus range used by SR-IOV capability +* @bus: the PCI bus +* +* Returns max number of buses (exclude current one) used by Virtual +* Functions. +*/ +int pci_iov_bus_range (struct pci_bus *bus) +{ +int max =0 ; +u8 busnr, devfn; +struct pci_dev *dev; + +list_for_each_entry (dev, &bus->devices, bus_list) { +if (!dev->iov) +continue ; +vf_rid (dev, dev->iov->totalvfs -1 , &busnr, &devfn); +if (busnr > max) + max = busnr; + } + +return max ? max - bus->number :0 ; +} + +/** * pci_iov_register - register SR-IOV service * @dev: the PCI device * @callback: callback function for SR-IOV events
drivers/pci/pci.h +5 -0
@@ -204,6 +204,7 @@ void pci_iov_remove_sysfs(struct pci_dev *dev);extern int pci_iov_resource_align (struct pci_dev *dev,int resno);extern int pci_iov_resource_bar (struct pci_dev *dev,int resno,enum pci_bar_type *type); +extern int pci_iov_bus_range (struct pci_bus *bus);#else static inline int pci_iov_init (struct pci_dev *dev) {
@@ -227,6 +228,10 @@ static inline int pci_iov_resource_bar(struct pci_dev *dev, int resno, {return 0 ; } +extern inline int pci_iov_bus_range (struct pci_bus *bus) +{ +return 0 ; +}#endif /* CONFIG_PCI_IOV */ #endif /* DRIVERS_PCI_H */
drivers/pci/probe.c +3 -0
@@ -1068,6 +1068,9 @@ unsigned int __devinit pci_scan_child_bus(struct pci_bus *bus)for (devfn =0 ; devfn <0x100 ; devfn +=8 )pci_scan_slot (bus, devfn); +/* Reserve buses for SR-IOV capability. */ + max +=pci_iov_bus_range (bus); +/* * After performing arch-dependent fixup of the bus, look behind * all PCI-to-PCI bridges on this bus.
Signature · 1 line
1.5.6.4
PATCH v6 14/16 PCI: document for SR-IOV user and developer
In reply to Yu ZhaoCreate HOW-TO for SR-IOV user and driver developer.
--- Documentation/DocBook/kernel-api.tmpl | 1 + Documentation/PCI/pci-iov-howto.txt | 181 +++++++++++++++++++++++++++++++++ 2 files changed, 182 insertions(+), 0 deletions(-) create mode 100644 Documentation/PCI/pci-iov-howto.txt
Documentation/DocBook/kernel-api.tmpl +1 -0
@@ -251,6 +251,7 @@ X!Edrivers/pci/hotplug.c --> !Edrivers/pci/probe.c !Edrivers/pci/rom.c +!Edrivers/pci/iov.c </sect1> <sect1><title>PCI Hotplug Support Library</title> !Edrivers/pci/hotplug/pci_hotplug_core.c
Documentation/PCI/pci-iov-howto.txt new file +181 -0
181 lines not on this page · raw message
Signature · 1 line
1.5.6.4
PATCH v6 14/16 PCI: document for SR-IOV user and developer
In reply to Yu ZhaoCreate HOW-TO for SR-IOV user and driver developer.
--- Documentation/DocBook/kernel-api.tmpl | 1 + Documentation/PCI/pci-iov-howto.txt | 181 +++++++++++++++++++++++++++++++++ 2 files changed, 182 insertions(+), 0 deletions(-) create mode 100644 Documentation/PCI/pci-iov-howto.txt
Documentation/DocBook/kernel-api.tmpl +1 -0
@@ -251,6 +251,7 @@ X!Edrivers/pci/hotplug.c --> !Edrivers/pci/probe.c !Edrivers/pci/rom.c +!Edrivers/pci/iov.c </sect1> <sect1><title>PCI Hotplug Support Library</title> !Edrivers/pci/hotplug/pci_hotplug_core.c
Documentation/PCI/pci-iov-howto.txt new file +181 -0
181 lines not on this page · raw message
Signature · 1 line
1.5.6.4
PATCH v6 15/16 PCI: document the SR-IOV sysfs entries
In reply to Yu ZhaoDocument the SR-IOV sysfs entries.
--- Documentation/ABI/testing/sysfs-bus-pci | 33 +++++++++++++++++++++++++++++++ 1 files changed, 33 insertions(+), 0 deletions(-)
Documentation/ABI/testing/sysfs-bus-pci +33 -0
@@ -9,3 +9,36 @@ Description: that some devices may have malformatted data. If the underlying VPD has a writable section then the corresponding section of this file will be writable. + +What: /sys/bus/pci/devices/.../iov/enable +Date: October 2008 +Contact: Yu Zhao <[email protected]> +Description: + This file appears when a device has the SR-IOV capability. + It holds the status of the capability, and could be written + (0/1) to disable and enable the capability if the PF driver + supports this operation. + +What: /sys/bus/pci/devices/.../iov/initialvfs +Date: October 2008 +Contact: Yu Zhao <[email protected]> +Description: + This file appears when a device has the SR-IOV capability. + It holds the number of initial Virtual Functions (read-only). + +What: /sys/bus/pci/devices/.../iov/totalvfs +Date: October 2008 +Contact: Yu Zhao <[email protected]> +Description: + This file appears when a device has the SR-IOV capability. + It holds the number of total Virtual Functions (read-only). + + +What: /sys/bus/pci/devices/.../iov/numvfs +Date: October 2008 +Contact: Yu Zhao <[email protected]> +Description: + This file appears when a device has the SR-IOV capability. + It holds the number of available Virtual Functions, and + could be written (1 ~ InitialVFs) to change the number if + the PF driver supports this operation.
Signature · 1 line
1.5.6.4
PATCH v6 15/16 PCI: document the SR-IOV sysfs entries
In reply to Yu ZhaoDocument the SR-IOV sysfs entries.
--- Documentation/ABI/testing/sysfs-bus-pci | 33 +++++++++++++++++++++++++++++++ 1 files changed, 33 insertions(+), 0 deletions(-)
Documentation/ABI/testing/sysfs-bus-pci +33 -0
@@ -9,3 +9,36 @@ Description: that some devices may have malformatted data. If the underlying VPD has a writable section then the corresponding section of this file will be writable. + +What: /sys/bus/pci/devices/.../iov/enable +Date: October 2008 +Contact: Yu Zhao <[email protected]> +Description: + This file appears when a device has the SR-IOV capability. + It holds the status of the capability, and could be written + (0/1) to disable and enable the capability if the PF driver + supports this operation. + +What: /sys/bus/pci/devices/.../iov/initialvfs +Date: October 2008 +Contact: Yu Zhao <[email protected]> +Description: + This file appears when a device has the SR-IOV capability. + It holds the number of initial Virtual Functions (read-only). + +What: /sys/bus/pci/devices/.../iov/totalvfs +Date: October 2008 +Contact: Yu Zhao <[email protected]> +Description: + This file appears when a device has the SR-IOV capability. + It holds the number of total Virtual Functions (read-only). + + +What: /sys/bus/pci/devices/.../iov/numvfs +Date: October 2008 +Contact: Yu Zhao <[email protected]> +Description: + This file appears when a device has the SR-IOV capability. + It holds the number of available Virtual Functions, and + could be written (1 ~ InitialVFs) to change the number if + the PF driver supports this operation.
Signature · 1 line
1.5.6.4
Re: [PATCH 15/16 v6] PCI: document the SR-IOV sysfs entries
Quoted 21 lines · Yu Zhao
Document the SR-IOV sysfs entries.
Cc Alex Chiang <[email protected]>Cc Grant Grundler <[email protected]>Cc Greg KH <[email protected]>Cc Ingo Molnar <[email protected]>Cc Jesse Barnes <[email protected]>Cc Matthew Wilcox <[email protected]>Cc Randy Dunlap <[email protected]>Cc Roland Dreier <[email protected]>Signed-off-by Yu Zhao <[email protected]>--- Documentation/ABI/testing/sysfs-bus-pci | 33 +++++++++++++++++++++++++++++++ 1 files changed, 33 insertions(+), 0 deletions(-)Documentation/ABI/testing/sysfs-bus-pci +0
-0
Documentation/ABI/testing/sysfs-bus-pci Description: · line 13
Quoted 5 lines
Documentation/ABI/testing/sysfs-bus-pci +2
-0@@ -9,3 +9,36 @@ Description: that some devices may have malformatted data. If the underlying VPD has a writable section then the corresponding section of this file will be writable. + +What: /sys/bus/pci/devices/.../iov/enable
Are you sure this is still the correct location with your change to struct device?
thanks,
greg k-h
Re: [PATCH 15/16 v6] PCI: document the SR-IOV sysfs entries
In reply to Yu ZhaoQuoted 21 lines · Yu Zhao
Document the SR-IOV sysfs entries.
Cc Alex Chiang <[email protected]>Cc Grant Grundler <[email protected]>Cc Greg KH <[email protected]>Cc Ingo Molnar <[email protected]>Cc Jesse Barnes <[email protected]>Cc Matthew Wilcox <[email protected]>Cc Randy Dunlap <[email protected]>Cc Roland Dreier <[email protected]>Signed-off-by Yu Zhao <[email protected]>--- Documentation/ABI/testing/sysfs-bus-pci | 33 +++++++++++++++++++++++++++++++ 1 files changed, 33 insertions(+), 0 deletions(-)Documentation/ABI/testing/sysfs-bus-pci +0
-0
Documentation/ABI/testing/sysfs-bus-pci Description: · line 13
Quoted 5 lines
Documentation/ABI/testing/sysfs-bus-pci +2
-0@@ -9,3 +9,36 @@ Description: that some devices may have malformatted data. If the underlying VPD has a writable section then the corresponding section of this file will be writable. + +What: /sys/bus/pci/devices/.../iov/enable
Are you sure this is still the correct location with your change to struct device?
thanks,
greg k-h
Re: [PATCH 15/16 v6] PCI: document the SR-IOV sysfs entries
Quoted 17 lines · Greg KH
Quoted 16 lines · Yu Zhao
Document the SR-IOV sysfs entries.
Cc Alex Chiang <[email protected]>Cc Grant Grundler <[email protected]>Cc Greg KH <[email protected]>Cc Ingo Molnar <[email protected]>Cc Jesse Barnes <[email protected]>Cc Matthew Wilcox <[email protected]>Cc Randy Dunlap <[email protected]>Cc Roland Dreier <[email protected]>Signed-off-by Yu Zhao <[email protected]>--- Documentation/ABI/testing/sysfs-bus-pci | 33 +++++++++++++++++++++++++++++++ 1 files changed, 33 insertions(+), 0 deletions(-)
Quoted 13 lines
Documentation/ABI/testing/sysfs-bus-pci Description: · line 13
Quoted 10 lines
Documentation/ABI/testing/sysfs-bus-pci +2
-0@@ -9,3 +9,36 @@ Description: that some devices may have malformatted data. If the underlying VPD has a writable section then the corresponding section of this file will be writable. + +What: /sys/bus/pci/devices/.../iov/enableAre you sure this is still the correct location with your change to struct device?
Nevermind, this is correct.
But the bigger problem is that userspace doesn't know when these attributes show up. So tools like udev and HAL and others can't look for them as they never get notified, and they don't even know if they should be looking for them or not.
Is there any way to tie these attributes to the "main" pci device so that they get created before the device is announced to the world? Doing that would solve this issue.
thanks,
greg k-h
Re: [PATCH 15/16 v6] PCI: document the SR-IOV sysfs entries
Quoted 25 lines · Greg KH
Quoted 4 lines · Greg KH
Quoted 3 lines · Yu Zhao
Documentation/ABI/testing/sysfs-bus-pci | 33 +++++++++++++++++++++++++++++++ 1 files changed, 33 insertions(+), 0 deletions(-)Quoted 12 lines
Documentation/ABI/testing/sysfs-bus-pci Description: · line 13
Quoted 10 lines
Documentation/ABI/testing/sysfs-bus-pci +2
-0@@ -9,3 +9,36 @@ Description: that some devices may have malformatted data. If the underlying VPD has a writable section then the corresponding section of this file will be writable. + +What: /sys/bus/pci/devices/.../iov/enableAre you sure this is still the correct location with your change to struct device?
Nevermind, this is correct.
But the bigger problem is that userspace doesn't know when these attributes show up. So tools like udev and HAL and others can't look for them as they never get notified, and they don't even know if they should be looking for them or not.
Quoted 7 lines
Is there any way to tie these attributes to the "main" pci device so that they get created before the device is announced to the world? Doing that would solve this issue.
thanks,
greg k-h
Currently PCI subsystem has /sys/.../{vendor,device,...} bundled to the main PCI device (I suppose this means the entries are created by 'device_add')
And after the PCI device is announced, /sys/.../{config,resourceX,rom,vpd,iov,...} get created depending on if these features are supported.
Making dynamic entries tie to the main PCI device would require PCI subsystem to allocate different 'bus_type' for the devices, right?
Regards,
Yu
Re: [PATCH 15/16 v6] PCI: document the SR-IOV sysfs entries
Quoted 31 lines · Yu
Quoted 19 lines · Greg KH
Quoted 5 lines · Greg KH
Quoted 4 lines · Yu Zhao
Documentation/ABI/testing/sysfs-bus-pci | 33 +++++++++++++++++++++++++++++++ 1 files changed, 33 insertions(+), 0 deletions(-)Quoted 13 lines
Documentation/ABI/testing/sysfs-bus-pci Description: · line 13
Quoted 11 lines
Documentation/ABI/testing/sysfs-bus-pci +2
-0@@ -9,3 +9,36 @@ Description: that some devices may have malformatted data. If the underlying VPD has a writable section then the corresponding section of this file will be writable. + +What: /sys/bus/pci/devices/.../iov/enableAre you sure this is still the correct location with your change to struct device?
Quoted 10 lines
Nevermind, this is correct.
But the bigger problem is that userspace doesn't know when these attributes show up. So tools like udev and HAL and others can't look for them as they never get notified, and they don't even know if they should be looking for them or not.
Is there any way to tie these attributes to the "main" pci device so that they get created before the device is announced to the world? Doing that would solve this issue.
thanks,
greg k-h
Quoted 7 lines
Currently PCI subsystem has
/sys/.../{vendor,device,...}bundled to the main PCI device (I suppose this means the entries are created by 'device_add')And after the PCI device is announced,
/sys/.../{config,resourceX,rom,vpd,iov,...}get created depending on if these features are supported.
And that's a bug. Let's not continue to make the same bug here as well.
Quoted 2 lines
Making dynamic entries tie to the main PCI device would require PCI subsystem to allocate different '
bus_type' for the devices, right?
No, it would just mean they need to be all added before the device is fully registered with the driver core.
thanks,
greg k-h
Re: [PATCH 15/16 v6] PCI: document the SR-IOV sysfs entries
In reply to Zhao, YuQuoted 31 lines · Yu
Quoted 19 lines · Greg KH
Quoted 5 lines · Greg KH
Quoted 4 lines · Yu Zhao
Documentation/ABI/testing/sysfs-bus-pci | 33 +++++++++++++++++++++++++++++++ 1 files changed, 33 insertions(+), 0 deletions(-)Quoted 13 lines
Documentation/ABI/testing/sysfs-bus-pci Description: · line 13
Quoted 11 lines
Documentation/ABI/testing/sysfs-bus-pci +2
-0@@ -9,3 +9,36 @@ Description: that some devices may have malformatted data. If the underlying VPD has a writable section then the corresponding section of this file will be writable. + +What: /sys/bus/pci/devices/.../iov/enableAre you sure this is still the correct location with your change to struct device?
Quoted 10 lines
Nevermind, this is correct.
But the bigger problem is that userspace doesn't know when these attributes show up. So tools like udev and HAL and others can't look for them as they never get notified, and they don't even know if they should be looking for them or not.
Is there any way to tie these attributes to the "main" pci device so that they get created before the device is announced to the world? Doing that would solve this issue.
thanks,
greg k-h
Quoted 7 lines
Currently PCI subsystem has
/sys/.../{vendor,device,...}bundled to the main PCI device (I suppose this means the entries are created by 'device_add')And after the PCI device is announced,
/sys/.../{config,resourceX,rom,vpd,iov,...}get created depending on if these features are supported.
And that's a bug. Let's not continue to make the same bug here as well.
Quoted 2 lines
Making dynamic entries tie to the main PCI device would require PCI subsystem to allocate different '
bus_type' for the devices, right?
No, it would just mean they need to be all added before the device is fully registered with the driver core.
thanks,
greg k-h
Re: [PATCH 15/16 v6] PCI: document the SR-IOV sysfs entries
Quoted 40 lines · Greg KH
Quoted 31 lines · Yu
Quoted 19 lines · Greg KH
Quoted 5 lines · Greg KH
Quoted 4 lines · Yu Zhao
Documentation/ABI/testing/sysfs-bus-pci | 33 +++++++++++++++++++++++++++++++ 1 files changed, 33 insertions(+), 0 deletions(-)Quoted 13 lines
Documentation/ABI/testing/sysfs-bus-pci Description: · line 13
Quoted 11 lines
Documentation/ABI/testing/sysfs-bus-pci +2
-0@@ -9,3 +9,36 @@ Description: that some devices may have malformatted data. If the underlying VPD has a writable section then the corresponding section of this file will be writable. + +What: /sys/bus/pci/devices/.../iov/enableAre you sure this is still the correct location with your change to struct device?
Quoted 10 lines
Nevermind, this is correct.
But the bigger problem is that userspace doesn't know when these attributes show up. So tools like udev and HAL and others can't look for them as they never get notified, and they don't even know if they should be looking for them or not.
Is there any way to tie these attributes to the "main" pci device so that they get created before the device is announced to the world? Doing that would solve this issue.
thanks,
greg k-hQuoted 7 lines
Currently PCI subsystem has
/sys/.../{vendor,device,...}bundled to the main PCI device (I suppose this means the entries are created by 'device_add')And after the PCI device is announced,
/sys/.../{config,resourceX,rom,vpd,iov,...}get created depending on if these features are supported.
Quoted 7 lines
And that's a bug. Let's not continue to make the same bug here as well.
Quoted 2 lines
Making dynamic entries tie to the main PCI device would require PCI subsystem to allocate different '
bus_type' for the devices, right?No, it would just mean they need to be all added before the device is fully registered with the driver core.
I looked into the PCI and driver core code again, but didn't figured out how to do it.
A 'pci_dev' is added by pci_bus_add_device() via device_add(), which creates sysfs entries according to 'dev_attrs' in the 'pci_bus_type'. If we want those dynamic entries to appear before the uevent is triggered, we have to bundle them into the 'dev_attrs'. Is this right way for the dynamic entries? Or I missed something?
Thanks,
Yu
Re: [PATCH 15/16 v6] PCI: document the SR-IOV sysfs entries
In reply to Greg KHQuoted 40 lines · Greg KH
Quoted 31 lines · Yu
Quoted 19 lines · Greg KH
Quoted 5 lines · Greg KH
Quoted 4 lines · Yu Zhao
Documentation/ABI/testing/sysfs-bus-pci | 33 +++++++++++++++++++++++++++++++ 1 files changed, 33 insertions(+), 0 deletions(-)Quoted 13 lines
Documentation/ABI/testing/sysfs-bus-pci Description: · line 13
Quoted 11 lines
Documentation/ABI/testing/sysfs-bus-pci +2
-0@@ -9,3 +9,36 @@ Description: that some devices may have malformatted data. If the underlying VPD has a writable section then the corresponding section of this file will be writable. + +What: /sys/bus/pci/devices/.../iov/enableAre you sure this is still the correct location with your change to struct device?
Quoted 10 lines
Nevermind, this is correct.
But the bigger problem is that userspace doesn't know when these attributes show up. So tools like udev and HAL and others can't look for them as they never get notified, and they don't even know if they should be looking for them or not.
Is there any way to tie these attributes to the "main" pci device so that they get created before the device is announced to the world? Doing that would solve this issue.
thanks,
greg k-hQuoted 7 lines
Currently PCI subsystem has
/sys/.../{vendor,device,...}bundled to the main PCI device (I suppose this means the entries are created by 'device_add')And after the PCI device is announced,
/sys/.../{config,resourceX,rom,vpd,iov,...}get created depending on if these features are supported.
Quoted 7 lines
And that's a bug. Let's not continue to make the same bug here as well.
Quoted 2 lines
Making dynamic entries tie to the main PCI device would require PCI subsystem to allocate different '
bus_type' for the devices, right?No, it would just mean they need to be all added before the device is fully registered with the driver core.
I looked into the PCI and driver core code again, but didn't figured out how to do it.
A 'pci_dev' is added by pci_bus_add_device() via device_add(), which creates sysfs entries according to 'dev_attrs' in the 'pci_bus_type'. If we want those dynamic entries to appear before the uevent is triggered, we have to bundle them into the 'dev_attrs'. Is this right way for the dynamic entries? Or I missed something?
Thanks,
Yu
Re: [PATCH 15/16 v6] PCI: document the SR-IOV sysfs entries
Quoted 49 lines · Yu Zhao
Quoted 40 lines · Greg KH
Quoted 31 lines · Yu
Quoted 19 lines · Greg KH
Quoted 5 lines · Greg KH
Quoted 4 lines · Yu Zhao
Documentation/ABI/testing/sysfs-bus-pci | 33 +++++++++++++++++++++++++++++++ 1 files changed, 33 insertions(+), 0 deletions(-)Quoted 13 lines
Documentation/ABI/testing/sysfs-bus-pci Description: · line 13
Quoted 11 lines
Documentation/ABI/testing/sysfs-bus-pci +2
-0@@ -9,3 +9,36 @@ Description: that some devices may have malformatted data. If the underlying VPD has a writable section then the corresponding section of this file will be writable. + +What: /sys/bus/pci/devices/.../iov/enableAre you sure this is still the correct location with your change to struct device?
Quoted 10 lines
Nevermind, this is correct.
But the bigger problem is that userspace doesn't know when these attributes show up. So tools like udev and HAL and others can't look for them as they never get notified, and they don't even know if they should be looking for them or not.
Is there any way to tie these attributes to the "main" pci device so that they get created before the device is announced to the world? Doing that would solve this issue.
thanks,
greg k-hQuoted 7 lines
Currently PCI subsystem has
/sys/.../{vendor,device,...}bundled to the main PCI device (I suppose this means the entries are created by 'device_add')And after the PCI device is announced,
/sys/.../{config,resourceX,rom,vpd,iov,...}get created depending on if these features are supported.Quoted 7 lines
And that's a bug. Let's not continue to make the same bug here as well.
Quoted 2 lines
Making dynamic entries tie to the main PCI device would require PCI subsystem to allocate different '
bus_type' for the devices, right?No, it would just mean they need to be all added before the device is fully registered with the driver core.
Quoted 8 lines
I looked into the PCI and driver core code again, but didn't figured out how to do it.
A '
pci_dev' is added bypci_bus_add_device()viadevice_add(), which creates sysfs entries according to 'dev_attrs' in the 'pci_bus_type'. If we want those dynamic entries to appear before the uevent is triggered, we have to bundle them into the 'dev_attrs'. Is this right way for the dynamic entries? Or I missed something?
Yes, that is correct.
Or you can add attributes before device_add() is called to the device, which is probably much easier to do, right?
There are also "conditional" attributes, which get only displayed if some kind of condition is met, I think you want to use those.
thanks,
greg k-h
Re: [PATCH 15/16 v6] PCI: document the SR-IOV sysfs entries
Quoted 28 lines · Greg KH
Quoted 18 lines · Yu Zhao
Quoted 9 lines · Greg KH
Quoted 7 lines · Yu
Currently PCI subsystem has
/sys/.../{vendor,device,...}bundled to the main PCI device (I suppose this means the entries are created by 'device_add')And after the PCI device is announced,
/sys/.../{config,resourceX,rom,vpd,iov,...}get created depending on if these features are supported.Quoted 7 lines
And that's a bug. Let's not continue to make the same bug here as well.
Quoted 2 lines
Making dynamic entries tie to the main PCI device would require PCI subsystem to allocate different '
bus_type' for the devices, right?No, it would just mean they need to be all added before the device is fully registered with the driver core.
Quoted 8 lines
I looked into the PCI and driver core code again, but didn't figured out how to do it.
A '
pci_dev' is added bypci_bus_add_device()viadevice_add(), which creates sysfs entries according to 'dev_attrs' in the 'pci_bus_type'. If we want those dynamic entries to appear before the uevent is triggered, we have to bundle them into the 'dev_attrs'. Is this right way for the dynamic entries? Or I missed something?
Quoted 7 lines
Yes, that is correct.
Or you can add attributes before
device_add()is called to the device, which is probably much easier to do, right?There are also "conditional" attributes, which get only displayed if some kind of condition is met, I think you want to use those.
The problem is the sysfs directory of the PCI device is created by the kobject_add() in the device_add() as follows. And the static entries bundled with the 'pci_bus_type' are created by the bus_add_device(). Between the kobject_add() and the kobject_uevent(), we don't have any other choice to add the dynamic entries.
In device_add():
error = kobject_add(&dev->kobj, dev->kobj.parent, "%s", dev->bus_id); ... error = bus_add_device(dev); ... kobject_uevent(&dev->kobj, KOBJ_ADD);
So looks like the only way is to make the dynamic entries bundled with the 'pci_bus_type', which means they would become static no matter the device supports the entries (i.e. corresponding capabilities) or not.
Thanks,
Yu
Re: [PATCH 15/16 v6] PCI: document the SR-IOV sysfs entries
Quoted 52 lines · Yu Zhao
Quoted 28 lines · Greg KH
Quoted 18 lines · Yu Zhao
Quoted 9 lines · Greg KH
Quoted 7 lines · Yu
Currently PCI subsystem has
/sys/.../{vendor,device,...}bundled to the main PCI device (I suppose this means the entries are created by 'device_add')And after the PCI device is announced,
/sys/.../{config,resourceX,rom,vpd,iov,...}get created depending on if these features are supported.Quoted 7 lines
And that's a bug. Let's not continue to make the same bug here as well.
Quoted 2 lines
Making dynamic entries tie to the main PCI device would require PCI subsystem to allocate different '
bus_type' for the devices, right?No, it would just mean they need to be all added before the device is fully registered with the driver core.
Quoted 8 lines
I looked into the PCI and driver core code again, but didn't figured out how to do it.
A '
pci_dev' is added bypci_bus_add_device()viadevice_add(), which creates sysfs entries according to 'dev_attrs' in the 'pci_bus_type'. If we want those dynamic entries to appear before the uevent is triggered, we have to bundle them into the 'dev_attrs'. Is this right way for the dynamic entries? Or I missed something?Quoted 7 lines
Yes, that is correct.
Or you can add attributes before
device_add()is called to the device, which is probably much easier to do, right?There are also "conditional" attributes, which get only displayed if some kind of condition is met, I think you want to use those.
The problem is the sysfs directory of the PCI device is created by the
kobject_add()in thedevice_add()as follows. And the static entries bundled with the 'pci_bus_type' are created by thebus_add_device(). Between thekobject_add()and thekobject_uevent(), we don't have any other choice to add the dynamic entries.In
device_add():error = kobject_add(&dev->kobj, dev->kobj.parent, "%s", dev->bus_id); ... error = bus_add_device(dev); ... kobject_uevent(&dev->kobj, KOBJ_ADD);
Quoted 3 lines
So looks like the only way is to make the dynamic entries bundled with the '
pci_bus_type', which means they would become static no matter the device supports the entries (i.e. corresponding capabilities) or not.
No, this can work, other busses do this. There are "conditional" attributes that only get enabled if specific things happen, and you can add attributes before device_add() is called. See the scsi code for examples of both of these options.
thanks,
greg k-h
Re: [PATCH 15/16 v6] PCI: document the SR-IOV sysfs entries
In reply to Yu ZhaoQuoted 52 lines · Yu Zhao
Quoted 28 lines · Greg KH
Quoted 18 lines · Yu Zhao
Quoted 9 lines · Greg KH
Quoted 7 lines · Yu
Currently PCI subsystem has
/sys/.../{vendor,device,...}bundled to the main PCI device (I suppose this means the entries are created by 'device_add')And after the PCI device is announced,
/sys/.../{config,resourceX,rom,vpd,iov,...}get created depending on if these features are supported.Quoted 7 lines
And that's a bug. Let's not continue to make the same bug here as well.
Quoted 2 lines
Making dynamic entries tie to the main PCI device would require PCI subsystem to allocate different '
bus_type' for the devices, right?No, it would just mean they need to be all added before the device is fully registered with the driver core.
Quoted 8 lines
I looked into the PCI and driver core code again, but didn't figured out how to do it.
A '
pci_dev' is added bypci_bus_add_device()viadevice_add(), which creates sysfs entries according to 'dev_attrs' in the 'pci_bus_type'. If we want those dynamic entries to appear before the uevent is triggered, we have to bundle them into the 'dev_attrs'. Is this right way for the dynamic entries? Or I missed something?Quoted 7 lines
Yes, that is correct.
Or you can add attributes before
device_add()is called to the device, which is probably much easier to do, right?There are also "conditional" attributes, which get only displayed if some kind of condition is met, I think you want to use those.
The problem is the sysfs directory of the PCI device is created by the
kobject_add()in thedevice_add()as follows. And the static entries bundled with the 'pci_bus_type' are created by thebus_add_device(). Between thekobject_add()and thekobject_uevent(), we don't have any other choice to add the dynamic entries.In
device_add():error = kobject_add(&dev->kobj, dev->kobj.parent, "%s", dev->bus_id); ... error = bus_add_device(dev); ... kobject_uevent(&dev->kobj, KOBJ_ADD);
Quoted 3 lines
So looks like the only way is to make the dynamic entries bundled with the '
pci_bus_type', which means they would become static no matter the device supports the entries (i.e. corresponding capabilities) or not.
No, this can work, other busses do this. There are "conditional" attributes that only get enabled if specific things happen, and you can add attributes before device_add() is called. See the scsi code for examples of both of these options.
thanks,
greg k-h
Re: [PATCH 15/16 v6] PCI: document the SR-IOV sysfs entries
In reply to Yu ZhaoQuoted 20 lines · Yu Zhao
Quoted 2 lines · Greg KH
There are also "conditional" attributes, which get only displayed if some kind of condition is met, I think you want to use those.
The problem is the sysfs directory of the PCI device is created by the
kobject_add()in thedevice_add()as follows. And the static entries bundled with the 'pci_bus_type' are created by thebus_add_device(). Between thekobject_add()and thekobject_uevent(), we don't have any other choice to add the dynamic entries.In
device_add():error = kobject_add(&dev->kobj, dev->kobj.parent, "%s", dev->bus_id); ... error = bus_add_device(dev); ... kobject_uevent(&dev->kobj, KOBJ_ADD);
Quoted 3 lines
So looks like the only way is to make the dynamic entries bundled with the '
pci_bus_type', which means they would become static no matter the device supports the entries (i.e. corresponding capabilities) or not.
There is device_add_attrs() which is just called between the calls you mention above. Like Greg said, it can add groups, and groups have an is_visible() callback, which can be used to conditionally create attributes out of of predefined list.
Kay
Re: [PATCH 15/16 v6] PCI: document the SR-IOV sysfs entries
In reply to Yu ZhaoQuoted 20 lines · Yu Zhao
Quoted 2 lines · Greg KH
There are also "conditional" attributes, which get only displayed if some kind of condition is met, I think you want to use those.
The problem is the sysfs directory of the PCI device is created by the
kobject_add()in thedevice_add()as follows. And the static entries bundled with the 'pci_bus_type' are created by thebus_add_device(). Between thekobject_add()and thekobject_uevent(), we don't have any other choice to add the dynamic entries.In
device_add():error = kobject_add(&dev->kobj, dev->kobj.parent, "%s", dev->bus_id); ... error = bus_add_device(dev); ... kobject_uevent(&dev->kobj, KOBJ_ADD);
Quoted 3 lines
So looks like the only way is to make the dynamic entries bundled with the '
pci_bus_type', which means they would become static no matter the device supports the entries (i.e. corresponding capabilities) or not.
There is device_add_attrs() which is just called between the calls you mention above. Like Greg said, it can add groups, and groups have an is_visible() callback, which can be used to conditionally create attributes out of of predefined list.
Kay
Re: [PATCH 15/16 v6] PCI: document the SR-IOV sysfs entries
In reply to Greg KHQuoted 28 lines · Greg KH
Quoted 18 lines · Yu Zhao
Quoted 9 lines · Greg KH
Quoted 7 lines · Yu
Currently PCI subsystem has
/sys/.../{vendor,device,...}bundled to the main PCI device (I suppose this means the entries are created by 'device_add')And after the PCI device is announced,
/sys/.../{config,resourceX,rom,vpd,iov,...}get created depending on if these features are supported.Quoted 7 lines
And that's a bug. Let's not continue to make the same bug here as well.
Quoted 2 lines
Making dynamic entries tie to the main PCI device would require PCI subsystem to allocate different '
bus_type' for the devices, right?No, it would just mean they need to be all added before the device is fully registered with the driver core.
Quoted 8 lines
I looked into the PCI and driver core code again, but didn't figured out how to do it.
A '
pci_dev' is added bypci_bus_add_device()viadevice_add(), which creates sysfs entries according to 'dev_attrs' in the 'pci_bus_type'. If we want those dynamic entries to appear before the uevent is triggered, we have to bundle them into the 'dev_attrs'. Is this right way for the dynamic entries? Or I missed something?
Quoted 7 lines
Yes, that is correct.
Or you can add attributes before
device_add()is called to the device, which is probably much easier to do, right?There are also "conditional" attributes, which get only displayed if some kind of condition is met, I think you want to use those.
The problem is the sysfs directory of the PCI device is created by the kobject_add() in the device_add() as follows. And the static entries bundled with the 'pci_bus_type' are created by the bus_add_device(). Between the kobject_add() and the kobject_uevent(), we don't have any other choice to add the dynamic entries.
In device_add():
error = kobject_add(&dev->kobj, dev->kobj.parent, "%s", dev->bus_id); ... error = bus_add_device(dev); ... kobject_uevent(&dev->kobj, KOBJ_ADD);
So looks like the only way is to make the dynamic entries bundled with the 'pci_bus_type', which means they would become static no matter the device supports the entries (i.e. corresponding capabilities) or not.
Thanks,
Yu
Re: [PATCH 15/16 v6] PCI: document the SR-IOV sysfs entries
In reply to Greg KHQuoted 25 lines · Greg KH
Quoted 4 lines · Greg KH
Quoted 3 lines · Yu Zhao
Documentation/ABI/testing/sysfs-bus-pci | 33 +++++++++++++++++++++++++++++++ 1 files changed, 33 insertions(+), 0 deletions(-)Quoted 12 lines
Documentation/ABI/testing/sysfs-bus-pci Description: · line 13
Quoted 10 lines
Documentation/ABI/testing/sysfs-bus-pci +2
-0@@ -9,3 +9,36 @@ Description: that some devices may have malformatted data. If the underlying VPD has a writable section then the corresponding section of this file will be writable. + +What: /sys/bus/pci/devices/.../iov/enableAre you sure this is still the correct location with your change to struct device?
Nevermind, this is correct.
But the bigger problem is that userspace doesn't know when these attributes show up. So tools like udev and HAL and others can't look for them as they never get notified, and they don't even know if they should be looking for them or not.
Quoted 7 lines
Is there any way to tie these attributes to the "main" pci device so that they get created before the device is announced to the world? Doing that would solve this issue.
thanks,
greg k-h
Currently PCI subsystem has /sys/.../{vendor,device,...} bundled to the main PCI device (I suppose this means the entries are created by 'device_add')
And after the PCI device is announced, /sys/.../{config,resourceX,rom,vpd,iov,...} get created depending on if these features are supported.
Making dynamic entries tie to the main PCI device would require PCI subsystem to allocate different 'bus_type' for the devices, right?
Regards,
Yu
Re: [PATCH 15/16 v6] PCI: document the SR-IOV sysfs entries
In reply to Greg KHQuoted 17 lines · Greg KH
Quoted 16 lines · Yu Zhao
Document the SR-IOV sysfs entries.
Cc Alex Chiang <[email protected]>Cc Grant Grundler <[email protected]>Cc Greg KH <[email protected]>Cc Ingo Molnar <[email protected]>Cc Jesse Barnes <[email protected]>Cc Matthew Wilcox <[email protected]>Cc Randy Dunlap <[email protected]>Cc Roland Dreier <[email protected]>Signed-off-by Yu Zhao <[email protected]>--- Documentation/ABI/testing/sysfs-bus-pci | 33 +++++++++++++++++++++++++++++++ 1 files changed, 33 insertions(+), 0 deletions(-)
Quoted 13 lines
Documentation/ABI/testing/sysfs-bus-pci Description: · line 13
Quoted 10 lines
Documentation/ABI/testing/sysfs-bus-pci +2
-0@@ -9,3 +9,36 @@ Description: that some devices may have malformatted data. If the underlying VPD has a writable section then the corresponding section of this file will be writable. + +What: /sys/bus/pci/devices/.../iov/enableAre you sure this is still the correct location with your change to struct device?
Nevermind, this is correct.
But the bigger problem is that userspace doesn't know when these attributes show up. So tools like udev and HAL and others can't look for them as they never get notified, and they don't even know if they should be looking for them or not.
Is there any way to tie these attributes to the "main" pci device so that they get created before the device is announced to the world? Doing that would solve this issue.
thanks,
greg k-h
PATCH v6 16/16 PCI: document the new PCI boot parameters
In reply to Yu ZhaoDocument the new PCI[x86] boot parameters.
--- Documentation/kernel-parameters.txt | 10 ++++++++++ 1 files changed, 10 insertions(+), 0 deletions(-)
Documentation/kernel-parameters.txt +10 -0
@@ -1677,6 +1677,16 @@ and is between 256 and 4096 characters. It is defined in the file cbmemsize=nn[KMG] The fixed amount of bus space which is reserved for the CardBus bridge's memory window. The default value is 64 megabytes. + assign-mmio=[dddd:]bb [X86] reassign memory resources of all + devices under bus [dddd:]bb (dddd is the domain + number and bb is the bus number). + assign-pio=[dddd:]bb [X86] reassign io port resources of all + devices under bus [dddd:]bb (dddd is the domain + number and bb is the bus number). + align-mmio=[dddd:]bb:dd.f [X86] relocate memory resources of a + device to minimum PAGE_SIZE alignment (dddd is + the domain number and bb, dd and f is the bus, + device and function number). pcie_aspm= [PCIE] Forcibly enable or disable PCIe Active State Power Management.
Signature · 1 line
1.5.6.4
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
Quoted 27 lines · Yu Zhao
Document the new PCI[x86] boot parameters.
Cc Alex Chiang <[email protected]>Cc Grant Grundler <[email protected]>Cc Greg KH <[email protected]>Cc Ingo Molnar <[email protected]>Cc Jesse Barnes <[email protected]>Cc Matthew Wilcox <[email protected]>Cc Randy Dunlap <[email protected]>Cc Roland Dreier <[email protected]>Signed-off-by Yu Zhao <[email protected]>--- Documentation/kernel-parameters.txt | 10 ++++++++++ 1 files changed, 10 insertions(+), 0 deletions(-)Documentation/kernel-parameters.txt +3
-0@@ -1677,6 +1677,16 @@ and is between 256 and 4096 characters. It is defined in the file cbmemsize=nn[KMG] The fixed amount of bus space which is reserved for the CardBus bridge's memory window. The default value is 64 megabytes. + assign-mmio=[dddd:]bb [X86] reassign memory resources of all + devices under bus [dddd:]bb (dddd is the domain + number and bb is the bus number).
Documentation/kernel-parameters.txt and is between 256 and 4096 characters. It is defined in the file · line 1692
Quoted 10 lines
Documentation/kernel-parameters.txt +7
-0+ assign-pio=[dddd:]bb [X86] reassign io port resources of all + devices under bus [dddd:]bb (dddd is the domain + number and bb is the bus number). + align-mmio=[dddd:]bb:dd.f [X86] relocate memory resources of a + device to minimum PAGE_SIZE alignment (dddd is + the domain number and bb, dd and f is the bus, + device and function number). pcie_aspm= [PCIE] Forcibly enable or disable PCIe Active State Power Management.
I think it's nicer to have the documentation change included in the patch that implements the change. For example, I think this and patch 9/16 "add boot option to align ..." should be folded into a single patch. And similarly for the other documentation patches.
Bjorn
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
In reply to Yu ZhaoQuoted 27 lines · Yu Zhao
Document the new PCI[x86] boot parameters.
Cc Alex Chiang <[email protected]>Cc Grant Grundler <[email protected]>Cc Greg KH <[email protected]>Cc Ingo Molnar <[email protected]>Cc Jesse Barnes <[email protected]>Cc Matthew Wilcox <[email protected]>Cc Randy Dunlap <[email protected]>Cc Roland Dreier <[email protected]>Signed-off-by Yu Zhao <[email protected]>--- Documentation/kernel-parameters.txt | 10 ++++++++++ 1 files changed, 10 insertions(+), 0 deletions(-)Documentation/kernel-parameters.txt +3
-0@@ -1677,6 +1677,16 @@ and is between 256 and 4096 characters. It is defined in the file cbmemsize=nn[KMG] The fixed amount of bus space which is reserved for the CardBus bridge's memory window. The default value is 64 megabytes. + assign-mmio=[dddd:]bb [X86] reassign memory resources of all + devices under bus [dddd:]bb (dddd is the domain + number and bb is the bus number).
Documentation/kernel-parameters.txt and is between 256 and 4096 characters. It is defined in the file · line 1692
Quoted 10 lines
Documentation/kernel-parameters.txt +7
-0+ assign-pio=[dddd:]bb [X86] reassign io port resources of all + devices under bus [dddd:]bb (dddd is the domain + number and bb is the bus number). + align-mmio=[dddd:]bb:dd.f [X86] relocate memory resources of a + device to minimum PAGE_SIZE alignment (dddd is + the domain number and bb, dd and f is the bus, + device and function number). pcie_aspm= [PCIE] Forcibly enable or disable PCIe Active State Power Management.
I think it's nicer to have the documentation change included in the patch that implements the change. For example, I think this and patch 9/16 "add boot option to align ..." should be folded into a single patch. And similarly for the other documentation patches.
Bjorn
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
Quoted 17 lines · Bjorn Helgaas
Quoted 16 lines · Yu Zhao
Document the new PCI[x86] boot parameters.
Cc Alex Chiang <[email protected]>Cc Grant Grundler <[email protected]>Cc Greg KH <[email protected]>Cc Ingo Molnar <[email protected]>Cc Jesse Barnes <[email protected]>Cc Matthew Wilcox <[email protected]>Cc Randy Dunlap <[email protected]>Cc Roland Dreier <[email protected]>Signed-off-by Yu Zhao <[email protected]>--- Documentation/kernel-parameters.txt | 10 ++++++++++ 1 files changed, 10 insertions(+), 0 deletions(-)
Quoted 12 lines
Documentation/kernel-parameters.txt and is between 256 and 4096 characters. It is defined in the file · line 1683
Quoted 12 lines
Documentation/kernel-parameters.txt +4
-0@@ -1677,6 +1677,16 @@ and is between 256 and 4096 characters. It is defined in the file cbmemsize=nn[KMG] The fixed amount of bus space which is reserved for the CardBus bridge's memory window. The default value is 64 megabytes. + assign-mmio=[dddd:]bb [X86] reassign memory resources of all + devices under bus [dddd:]bb (dddd is the domain + number and bb is the bus number). + assign-pio=[dddd:]bb [X86] reassign io port resources of all
"io" in text should be "IO" or "I/O". (Small "io" is OK as a parameter placeholder.)
Quoted 5 lines
Documentation/kernel-parameters.txt and is between 256 and 4096 characters. It is defined in the file · line 1688
Quoted 5 lines
Documentation/kernel-parameters.txt +5
-0+ devices under bus [dddd:]bb (dddd is the domain + number and bb is the bus number). + align-mmio=[dddd:]bb:dd.f [X86] relocate memory resources of a + device to minimum PAGE_SIZE alignment (dddd is + the domain number and bb, dd and f is the bus,
are the bus,
Quoted 11 lines
Documentation/kernel-parameters.txt and is between 256 and 4096 characters. It is defined in the file · line 1692
Quoted 4 lines
Documentation/kernel-parameters.txt +1
-0+ device and function number). pcie_aspm= [PCIE] Forcibly enable or disable PCIe Active State Power Management.I think it's nicer to have the documentation change included in the patch that implements the change. For example, I think this and patch 9/16 "add boot option to align ..." should be folded into a single patch. And similarly for the other documentation patches.
Bjorn
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
In reply to Bjorn HelgaasQuoted 17 lines · Bjorn Helgaas
Quoted 16 lines · Yu Zhao
Document the new PCI[x86] boot parameters.
Cc Alex Chiang <[email protected]>Cc Grant Grundler <[email protected]>Cc Greg KH <[email protected]>Cc Ingo Molnar <[email protected]>Cc Jesse Barnes <[email protected]>Cc Matthew Wilcox <[email protected]>Cc Randy Dunlap <[email protected]>Cc Roland Dreier <[email protected]>Signed-off-by Yu Zhao <[email protected]>--- Documentation/kernel-parameters.txt | 10 ++++++++++ 1 files changed, 10 insertions(+), 0 deletions(-)
Quoted 12 lines
Documentation/kernel-parameters.txt and is between 256 and 4096 characters. It is defined in the file · line 1683
Quoted 12 lines
Documentation/kernel-parameters.txt +4
-0@@ -1677,6 +1677,16 @@ and is between 256 and 4096 characters. It is defined in the file cbmemsize=nn[KMG] The fixed amount of bus space which is reserved for the CardBus bridge's memory window. The default value is 64 megabytes. + assign-mmio=[dddd:]bb [X86] reassign memory resources of all + devices under bus [dddd:]bb (dddd is the domain + number and bb is the bus number). + assign-pio=[dddd:]bb [X86] reassign io port resources of all
"io" in text should be "IO" or "I/O". (Small "io" is OK as a parameter placeholder.)
Quoted 5 lines
Documentation/kernel-parameters.txt and is between 256 and 4096 characters. It is defined in the file · line 1688
Quoted 5 lines
Documentation/kernel-parameters.txt +5
-0+ devices under bus [dddd:]bb (dddd is the domain + number and bb is the bus number). + align-mmio=[dddd:]bb:dd.f [X86] relocate memory resources of a + device to minimum PAGE_SIZE alignment (dddd is + the domain number and bb, dd and f is the bus,
are the bus,
Quoted 11 lines
Documentation/kernel-parameters.txt and is between 256 and 4096 characters. It is defined in the file · line 1692
Quoted 4 lines
Documentation/kernel-parameters.txt +1
-0+ device and function number). pcie_aspm= [PCIE] Forcibly enable or disable PCIe Active State Power Management.I think it's nicer to have the documentation change included in the patch that implements the change. For example, I think this and patch 9/16 "add boot option to align ..." should be folded into a single patch. And similarly for the other documentation patches.
Bjorn
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
In reply to Yu ZhaoQuoted 24 lines · Yu Zhao
Document the new PCI[x86] boot parameters.
Cc Alex Chiang <[email protected]>Cc Grant Grundler <[email protected]>Cc Greg KH <[email protected]>Cc Ingo Molnar <[email protected]>Cc Jesse Barnes <[email protected]>Cc Matthew Wilcox <[email protected]>Cc Randy Dunlap <[email protected]>Cc Roland Dreier <[email protected]>Signed-off-by Yu Zhao <[email protected]>--- Documentation/kernel-parameters.txt | 10 ++++++++++ 1 files changed, 10 insertions(+), 0 deletions(-)Documentation/kernel-parameters.txt +0
-0@@ -1677,6 +1677,16 @@ and is between 256 and 4096 characters. It is defined in the file cbmemsize=nn[KMG] The fixed amount of bus space which is reserved for the CardBus bridge's memory window. The default value is 64 megabytes.
Documentation/kernel-parameters.txt and is between 256 and 4096 characters. It is defined in the file · line 1689
Quoted 10 lines
Documentation/kernel-parameters.txt +10
-0+ assign-mmio=[dddd:]bb [X86] reassign memory resources of all + devices under bus [dddd:]bb (dddd is the domain + number and bb is the bus number). + assign-pio=[dddd:]bb [X86] reassign io port resources of all + devices under bus [dddd:]bb (dddd is the domain + number and bb is the bus number). + align-mmio=[dddd:]bb:dd.f [X86] relocate memory resources of a + device to minimum PAGE_SIZE alignment (dddd is + the domain number and bb, dd and f is the bus, + device and function number).
This seems like a big problem. How are we going to know to add these command line options for devices we haven't even seen/known about yet?
How do we know the bus ids aren't going to change between boots (hint, they are, pci bus ids change all the time...)
We need to be able to do this kind of thing dynamically, not fixed at boot time, which seems way to early to even know about this, right?
thanks,
greg k-h
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
Quoted 29 lines · Greg KH
Quoted 11 lines · Yu Zhao
Documentation/kernel-parameters.txt | 10 ++++++++++ 1 files changed, 10 insertions(+), 0 deletions(-)Documentation/kernel-parameters.txt +0
-0@@ -1677,6 +1677,16 @@ and is between 256 and 4096 characters. It is defined in the file cbmemsize=nn[KMG] The fixed amount of bus space which is reserved for the CardBus bridge's memory window. The default value is 64 megabytes.Documentation/kernel-parameters.txt and is between 256 and 4096 characters. It is defined in the file · line 1689
Quoted 10 lines
Documentation/kernel-parameters.txt +10
-0+ assign-mmio=[dddd:]bb [X86] reassign memory resources of all + devices under bus [dddd:]bb (dddd is the domain + number and bb is the bus number). + assign-pio=[dddd:]bb [X86] reassign io port resources of all + devices under bus [dddd:]bb (dddd is the domain + number and bb is the bus number). + align-mmio=[dddd:]bb:dd.f [X86] relocate memory resources of a + device to minimum PAGE_SIZE alignment (dddd is + the domain number and bb, dd and f is the bus, + device and function number).This seems like a big problem. How are we going to know to add these command line options for devices we haven't even seen/known about yet?
How do we know the bus ids aren't going to change between boots (hint, they are, pci bus ids change all the time...)
Quoted 6 lines
We need to be able to do this kind of thing dynamically, not fixed at boot time, which seems way to early to even know about this, right?
thanks,
greg k-h
Yes, I totally agree. Doing things dynamically is better.
The purpose of these parameters is to rebalance and align resources for device that has BARs encapsulated in various new capabilities (SR-IOV, etc.), because most of existing BIOSes don't take care of those BARs.
If we do resource rebalance after system is up, do you think there is any side effect or impact to other subsystem other than PCI (e.g. MTRR)?
I haven't had much thinking on the dynamical resource rebalance. If you have any idea about this, can you please suggest?
Regards,
Yu
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
Quoted 34 lines · Yu
Quoted 24 lines · Greg KH
Quoted 13 lines · Yu Zhao
Documentation/kernel-parameters.txt | 10 ++++++++++ 1 files changed, 10 insertions(+), 0 deletions(-)Documentation/kernel-parameters.txt +0
-0@@ -1677,6 +1677,16 @@ and is between 256 and 4096 characters. It isdefined in the file
Documentation/kernel-parameters.txt +0
-0cbmemsize=nn[KMG] The fixed amount of bus space which is reserved for the CardBus bridge's memory window. The default value is 64 megabytes.Documentation/kernel-parameters.txt and is between 256 and 4096 characters. It is defined in the file · line 1689
Quoted 10 lines
Documentation/kernel-parameters.txt +10
-0+ assign-mmio=[dddd:]bb [X86] reassign memory resources of all + devices under bus [dddd:]bb (dddd is the domain + number and bb is the bus number). + assign-pio=[dddd:]bb [X86] reassign io port resources of all + devices under bus [dddd:]bb (dddd is the domain + number and bb is the bus number). + align-mmio=[dddd:]bb:dd.f [X86] relocate memory resources of a + device to minimum PAGE_SIZE alignment (dddd is + the domain number and bb, dd and f is the bus, + device and function number).Quoted 8 lines
This seems like a big problem. How are we going to know to add these command line options for devices we haven't even seen/known about yet? How do we know the bus ids aren't going to change between boots (hint, they are, pci bus ids change all the time...)
We need to be able to do this kind of thing dynamically, not fixed at boot time, which seems way to early to even know about this, right? thanks,
greg k-h
Quoted 5 lines
Yes, I totally agree. Doing things dynamically is better.
The purpose of these parameters is to rebalance and align resources for device that has BARs encapsulated in various new capabilities (SR-IOV, etc.), because most of existing BIOSes don't take care of those BARs.
But how are you going to know what the proper device ids are going to be before the machine boots? I don't see how these options are ever going to work properly for a "real" user.
Quoted 2 lines
If we do resource rebalance after system is up, do you think there is any side effect or impact to other subsystem other than PCI (e.g. MTRR)?
I don't think so.
Quoted 2 lines
I haven't had much thinking on the dynamical resource rebalance. If you have any idea about this, can you please suggest?
Yeah, it's going to be hard :)
We've thought about this in the past, and even Microsoft said it was going to happen for Vista, but they realized in the end, like we did a few years previously, that it would require full support of all PCI drivers as well (if you rebalance stuff that is already bound to a driver.) So they dropped it.
When would you want to do this kind of rebalancing? Before any PCI driver is bound to any devices? Or afterwards?
thanks,
greg k-h
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
Quoted 49 lines · Greg KH
Quoted 32 lines · Yu
Quoted 23 lines · Greg KH
Quoted 12 lines · Yu Zhao
Documentation/kernel-parameters.txt | 10 ++++++++++ 1 files changed, 10 insertions(+), 0 deletions(-)Documentation/kernel-parameters.txt +0
-0@@ -1677,6 +1677,16 @@ and is between 256 and 4096 characters. It isdefined in the file
Documentation/kernel-parameters.txt +0
-0cbmemsize=nn[KMG] The fixed amount of bus space which is reserved for the CardBus bridge's memory window. The default value is 64 megabytes.Documentation/kernel-parameters.txt and is between 256 and 4096 characters. It is defined in the file · line 1689
Quoted 10 lines
Documentation/kernel-parameters.txt +10
-0+ assign-mmio=[dddd:]bb [X86] reassign memory resources of all + devices under bus [dddd:]bb (dddd is the domain + number and bb is the bus number). + assign-pio=[dddd:]bb [X86] reassign io port resources of all + devices under bus [dddd:]bb (dddd is the domain + number and bb is the bus number). + align-mmio=[dddd:]bb:dd.f [X86] relocate memory resources of a + device to minimum PAGE_SIZE alignment (dddd is + the domain number and bb, dd and f is the bus, + device and function number).Quoted 8 lines
This seems like a big problem. How are we going to know to add these command line options for devices we haven't even seen/known about yet? How do we know the bus ids aren't going to change between boots (hint, they are, pci bus ids change all the time...)
We need to be able to do this kind of thing dynamically, not fixed at boot time, which seems way to early to even know about this, right? thanks,
greg k-hQuoted 3 lines
Yes, I totally agree. Doing things dynamically is better.
The purpose of these parameters is to rebalance and align resources for device that has BARs encapsulated in various new capabilities (SR-IOV, etc.), because most of existing BIOSes don't take care of those BARs.
But how are you going to know what the proper device ids are going to be before the machine boots? I don't see how these options are ever going to work properly for a "real" user.
Quoted 1 line
If we do resource rebalance after system is up, do you think there is any side effect or impact to other subsystem other than PCI (e.g. MTRR)?
I don't think so.
Quoted 1 line
I haven't had much thinking on the dynamical resource rebalance. If you have any idea about this, can you please suggest?
Yeah, it's going to be hard :)
Quoted 8 lines
We've thought about this in the past, and even Microsoft said it was going to happen for Vista, but they realized in the end, like we did a few years previously, that it would require full support of all PCI drivers as well (if you rebalance stuff that is already bound to a driver.) So they dropped it.
When would you want to do this kind of rebalancing? Before any PCI driver is bound to any devices? Or afterwards?
I guess if we want the rebalance dynamic, then we should have it full -- the rebalance would be functional even after the driver is loaded.
But in most cases, there will be problem when we unload driver from a hard disk controller, etc. We can mount root on a ramdisk and do the rebalance there, but it's complicated for a real user.
So looks like doing rebalancing before any driver is bound to any device is also a nice idea, if user can get a shell to do rebalance before built-in PCI driver grabs device.
Regards,
Yu
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
Quoted 13 lines · Yu
Quoted 8 lines · Greg KH
We've thought about this in the past, and even Microsoft said it was going to happen for Vista, but they realized in the end, like we did a few years previously, that it would require full support of all PCI drivers as well (if you rebalance stuff that is already bound to a driver.) So they dropped it.
When would you want to do this kind of rebalancing? Before any PCI driver is bound to any devices? Or afterwards?
I guess if we want the rebalance dynamic, then we should have it full -- the rebalance would be functional even after the driver is loaded.
Quoted 7 lines
But in most cases, there will be problem when we unload driver from a hard disk controller, etc. We can mount root on a ramdisk and do the rebalance there, but it's complicated for a real user.
So looks like doing rebalancing before any driver is bound to any device is also a nice idea, if user can get a shell to do rebalance before built-in PCI driver grabs device.
Can we use the suspend/resume code to do this? Some drivers (sym2 for one) would definitely need to rerun some of their init code to cope with a BAR address changing.
Signature · 4 lines
Matthew Wilcox Intel Open Source Technology Centre "Bill, look, we understand that you're interested in selling us this operating system, but compare it to ours. We can't possibly take such a retrograde step."
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
In reply to Zhao, YuQuoted 13 lines · Yu
Quoted 8 lines · Greg KH
We've thought about this in the past, and even Microsoft said it was going to happen for Vista, but they realized in the end, like we did a few years previously, that it would require full support of all PCI drivers as well (if you rebalance stuff that is already bound to a driver.) So they dropped it.
When would you want to do this kind of rebalancing? Before any PCI driver is bound to any devices? Or afterwards?
I guess if we want the rebalance dynamic, then we should have it full -- the rebalance would be functional even after the driver is loaded.
Quoted 7 lines
But in most cases, there will be problem when we unload driver from a hard disk controller, etc. We can mount root on a ramdisk and do the rebalance there, but it's complicated for a real user.
So looks like doing rebalancing before any driver is bound to any device is also a nice idea, if user can get a shell to do rebalance before built-in PCI driver grabs device.
Can we use the suspend/resume code to do this? Some drivers (sym2 for one) would definitely need to rerun some of their init code to cope with a BAR address changing.
Signature · 4 lines
Matthew Wilcox Intel Open Source Technology Centre "Bill, look, we understand that you're interested in selling us this operating system, but compare it to ours. We can't possibly take such a retrograde step."
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
Quoted 18 lines · Matthew Wilcox
Quoted 17 lines · Yu
Quoted 8 lines · Greg KH
We've thought about this in the past, and even Microsoft said it was going to happen for Vista, but they realized in the end, like we did a few years previously, that it would require full support of all PCI drivers as well (if you rebalance stuff that is already bound to a driver.) So they dropped it.
When would you want to do this kind of rebalancing? Before any PCI driver is bound to any devices? Or afterwards?
I guess if we want the rebalance dynamic, then we should have it full -- the rebalance would be functional even after the driver is loaded.
But in most cases, there will be problem when we unload driver from a hard disk controller, etc. We can mount root on a ramdisk and do the rebalance there, but it's complicated for a real user.
Quoted 7 lines
Quoted 3 lines
So looks like doing rebalancing before any driver is bound to any device is also a nice idea, if user can get a shell to do rebalance before built-in PCI driver grabs device.
Can we use the suspend/resume code to do this? Some drivers (sym2 for one) would definitely need to rerun some of their init code to cope with a BAR address changing.
Yes, that is what I was thinking. But after some grep on the PCI device drivers, I feel frustrated because all those drivers only do 'ioremap' once at the 'probe' stage.
I believe this is the only problem that preclude us having the run-time resource rebalance. And I'm not sure how much effort we can fix it. Any comments?
Thanks,
Yu
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
In reply to Matthew WilcoxQuoted 18 lines · Matthew Wilcox
Quoted 17 lines · Yu
Quoted 8 lines · Greg KH
We've thought about this in the past, and even Microsoft said it was going to happen for Vista, but they realized in the end, like we did a few years previously, that it would require full support of all PCI drivers as well (if you rebalance stuff that is already bound to a driver.) So they dropped it.
When would you want to do this kind of rebalancing? Before any PCI driver is bound to any devices? Or afterwards?
I guess if we want the rebalance dynamic, then we should have it full -- the rebalance would be functional even after the driver is loaded.
But in most cases, there will be problem when we unload driver from a hard disk controller, etc. We can mount root on a ramdisk and do the rebalance there, but it's complicated for a real user.
Quoted 7 lines
Quoted 3 lines
So looks like doing rebalancing before any driver is bound to any device is also a nice idea, if user can get a shell to do rebalance before built-in PCI driver grabs device.
Can we use the suspend/resume code to do this? Some drivers (sym2 for one) would definitely need to rerun some of their init code to cope with a BAR address changing.
Yes, that is what I was thinking. But after some grep on the PCI device drivers, I feel frustrated because all those drivers only do 'ioremap' once at the 'probe' stage.
I believe this is the only problem that preclude us having the run-time resource rebalance. And I'm not sure how much effort we can fix it. Any comments?
Thanks,
Yu
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
On Thu, Dec 11, 2008 at 09:43:13AM +0800, Yu Zhao wrote:
...
Quoted 3 lines
I believe this is the only problem that preclude us having the run-time resource rebalance. And I'm not sure how much effort we can fix it. Any comments?
Figure out the right sequence for driver resume so the probe function can call resume as well?
Document the change and then start modifying drivers one-by-one. API changes are alot of work.
grant
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
Quoted 13 lines · Grant Grundler
Date: Wednesday, December 10, 2008, 10:33 PM
On Thu, Dec 11, 2008 at 09:43:13AM +0800, Yu Zhao wrote:
...Quoted 1 line
I believe this is the only problem that preclude us
having the run-time
Quoted 1 line
resource rebalance. And I'm not sure how much
effort we can fix it. Any
Quoted 1 line
comments?
Figure out the right sequence for driver resume so the probe function
can call resume as well?
Quoted 6 lines
Document the change and then start modifying drivers one-by-one.
API changes are alot of work.grant
--
I've been lurking awaiting to see such a discussion. Alerting the PCI drivers that their resources have been changed from underneath them by extending the suspend/resume model, or perhaps (heresy?) adding a new callback entry point specifically for instructing PCI drivers to re-read their BARs, etc. would be a step in the right direction to enable this whole re-balancing work; well, granted root/paging devs bound to PCI devs could be tricky. It seems entirely natural (to me) that Microsoft would shudder at the amount of work and verification of all drivers to do this, but the Linux community's attitude seems to embrace sweeping changes ;-).
--
LH
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
In reply to Grant GrundlerQuoted 13 lines · Grant Grundler
Date: Wednesday, December 10, 2008, 10:33 PM
On Thu, Dec 11, 2008 at 09:43:13AM +0800, Yu Zhao wrote:
...Quoted 1 line
I believe this is the only problem that preclude us
having the run-time
Quoted 1 line
resource rebalance. And I'm not sure how much
effort we can fix it. Any
Quoted 1 line
comments?
Figure out the right sequence for driver resume so the probe function
can call resume as well?
Quoted 6 lines
Document the change and then start modifying drivers one-by-one.
API changes are alot of work.grant
--
I've been lurking awaiting to see such a discussion. Alerting the PCI drivers that their resources have been changed from underneath them by extending the suspend/resume model, or perhaps (heresy?) adding a new callback entry point specifically for instructing PCI drivers to re-read their BARs, etc. would be a step in the right direction to enable this whole re-balancing work; well, granted root/paging devs bound to PCI devs could be tricky. It seems entirely natural (to me) that Microsoft would shudder at the amount of work and verification of all drivers to do this, but the Linux community's attitude seems to embrace sweeping changes ;-).
--
LH
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
In reply to Yu ZhaoOn Thu, Dec 11, 2008 at 09:43:13AM +0800, Yu Zhao wrote:
...
Quoted 3 lines
I believe this is the only problem that preclude us having the run-time resource rebalance. And I'm not sure how much effort we can fix it. Any comments?
Figure out the right sequence for driver resume so the probe function can call resume as well?
Document the change and then start modifying drivers one-by-one. API changes are alot of work.
grant
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
In reply to Zhao, YuQuoted 60 lines · Yu
Quoted 47 lines · Greg KH
Quoted 33 lines · Yu
Quoted 24 lines · Greg KH
Quoted 13 lines · Yu Zhao
Documentation/kernel-parameters.txt | 10 ++++++++++ 1 files changed, 10 insertions(+), 0 deletions(-)Documentation/kernel-parameters.txt +0
-0@@ -1677,6 +1677,16 @@ and is between 256 and 4096 characters. It isdefined in the file
Documentation/kernel-parameters.txt +0
-0cbmemsize=nn[KMG] The fixed amount of bus space which is reserved for the CardBus bridge's memory window. The default value is 64 megabytes.Documentation/kernel-parameters.txt and is between 256 and 4096 characters. It is defined in the file · line 1689
Quoted 10 lines
Documentation/kernel-parameters.txt +10
-0+ assign-mmio=[dddd:]bb [X86] reassign memory resources of all + devices under bus [dddd:]bb (dddd is the domain + number and bb is the bus number). + assign-pio=[dddd:]bb [X86] reassign io port resources of all + devices under bus [dddd:]bb (dddd is the domain + number and bb is the bus number). + align-mmio=[dddd:]bb:dd.f [X86] relocate memory resources of a + device to minimum PAGE_SIZE alignment (dddd is + the domain number and bb, dd and f is the bus, + device and function number).Quoted 8 lines
This seems like a big problem. How are we going to know to add these command line options for devices we haven't even seen/known about yet? How do we know the bus ids aren't going to change between boots (hint, they are, pci bus ids change all the time...)
We need to be able to do this kind of thing dynamically, not fixed at boot time, which seems way to early to even know about this, right? thanks,
greg k-hQuoted 5 lines
Yes, I totally agree. Doing things dynamically is better.
The purpose of these parameters is to rebalance and align resources for device that has BARs encapsulated in various new capabilities (SR-IOV, etc.), because most of existing BIOSes don't take care of those BARs.
But how are you going to know what the proper device ids are going to be before the machine boots? I don't see how these options are ever going to work properly for a "real" user.
Quoted 2 lines
If we do resource rebalance after system is up, do you think there is any side effect or impact to other subsystem other than PCI (e.g. MTRR)?
I don't think so.
Quoted 2 lines
I haven't had much thinking on the dynamical resource rebalance. If you have any idea about this, can you please suggest?
Quoted 8 lines
Yeah, it's going to be hard :)
We've thought about this in the past, and even Microsoft said it was going to happen for Vista, but they realized in the end, like we did a few years previously, that it would require full support of all PCI drivers as well (if you rebalance stuff that is already bound to a driver.) So they dropped it.
When would you want to do this kind of rebalancing? Before any PCI driver is bound to any devices? Or afterwards?I guess if we want the rebalance dynamic, then we should have it full -- the rebalance would be functional even after the driver is loaded.
Quoted 7 lines
But in most cases, there will be problem when we unload driver from a hard disk controller, etc. We can mount root on a ramdisk and do the rebalance there, but it's complicated for a real user.
So looks like doing rebalancing before any driver is bound to any device is also a nice idea, if user can get a shell to do rebalance before built-in PCI driver grabs device.
That's not going to work, it needs to happen before any PCI device is bound, which is before init runs.
thanks,
greg k-h
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
In reply to Zhao, YuQuoted 60 lines · Yu
Quoted 47 lines · Greg KH
Quoted 33 lines · Yu
Quoted 24 lines · Greg KH
Quoted 13 lines · Yu Zhao
Documentation/kernel-parameters.txt | 10 ++++++++++ 1 files changed, 10 insertions(+), 0 deletions(-)Documentation/kernel-parameters.txt +0
-0@@ -1677,6 +1677,16 @@ and is between 256 and 4096 characters. It isdefined in the file
Documentation/kernel-parameters.txt +0
-0cbmemsize=nn[KMG] The fixed amount of bus space which is reserved for the CardBus bridge's memory window. The default value is 64 megabytes.Documentation/kernel-parameters.txt and is between 256 and 4096 characters. It is defined in the file · line 1689
Quoted 10 lines
Documentation/kernel-parameters.txt +10
-0+ assign-mmio=[dddd:]bb [X86] reassign memory resources of all + devices under bus [dddd:]bb (dddd is the domain + number and bb is the bus number). + assign-pio=[dddd:]bb [X86] reassign io port resources of all + devices under bus [dddd:]bb (dddd is the domain + number and bb is the bus number). + align-mmio=[dddd:]bb:dd.f [X86] relocate memory resources of a + device to minimum PAGE_SIZE alignment (dddd is + the domain number and bb, dd and f is the bus, + device and function number).Quoted 8 lines
This seems like a big problem. How are we going to know to add these command line options for devices we haven't even seen/known about yet? How do we know the bus ids aren't going to change between boots (hint, they are, pci bus ids change all the time...)
We need to be able to do this kind of thing dynamically, not fixed at boot time, which seems way to early to even know about this, right? thanks,
greg k-hQuoted 5 lines
Yes, I totally agree. Doing things dynamically is better.
The purpose of these parameters is to rebalance and align resources for device that has BARs encapsulated in various new capabilities (SR-IOV, etc.), because most of existing BIOSes don't take care of those BARs.
But how are you going to know what the proper device ids are going to be before the machine boots? I don't see how these options are ever going to work properly for a "real" user.
Quoted 2 lines
If we do resource rebalance after system is up, do you think there is any side effect or impact to other subsystem other than PCI (e.g. MTRR)?
I don't think so.
Quoted 2 lines
I haven't had much thinking on the dynamical resource rebalance. If you have any idea about this, can you please suggest?
Quoted 8 lines
Yeah, it's going to be hard :)
We've thought about this in the past, and even Microsoft said it was going to happen for Vista, but they realized in the end, like we did a few years previously, that it would require full support of all PCI drivers as well (if you rebalance stuff that is already bound to a driver.) So they dropped it.
When would you want to do this kind of rebalancing? Before any PCI driver is bound to any devices? Or afterwards?I guess if we want the rebalance dynamic, then we should have it full -- the rebalance would be functional even after the driver is loaded.
Quoted 7 lines
But in most cases, there will be problem when we unload driver from a hard disk controller, etc. We can mount root on a ramdisk and do the rebalance there, but it's complicated for a real user.
So looks like doing rebalancing before any driver is bound to any device is also a nice idea, if user can get a shell to do rebalance before built-in PCI driver grabs device.
That's not going to work, it needs to happen before any PCI device is bound, which is before init runs.
thanks,
greg k-h
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
Quoted 52 lines · Greg KH
Quoted 51 lines · Yu
Quoted 50 lines · Greg KH
Quoted 32 lines · Yu
Quoted 23 lines · Greg KH
Quoted 12 lines · Yu Zhao
Documentation/kernel-parameters.txt | 10 ++++++++++ 1 files changed, 10 insertions(+), 0 deletions(-)Documentation/kernel-parameters.txt +0
-0@@ -1677,6 +1677,16 @@ and is between 256 and 4096 characters. It isdefined in the file
Documentation/kernel-parameters.txt +0
-0cbmemsize=nn[KMG] The fixed amount of bus space which is reserved for the CardBus bridge's memory window. The default value is 64 megabytes.Documentation/kernel-parameters.txt and is between 256 and 4096 characters. It is defined in the file · line 1689
Quoted 10 lines
Documentation/kernel-parameters.txt +10
-0+ assign-mmio=[dddd:]bb [X86] reassign memory resources of all + devices under bus [dddd:]bb (dddd is the domain + number and bb is the bus number). + assign-pio=[dddd:]bb [X86] reassign io port resources of all + devices under bus [dddd:]bb (dddd is the domain + number and bb is the bus number). + align-mmio=[dddd:]bb:dd.f [X86] relocate memory resources of a + device to minimum PAGE_SIZE alignment (dddd is + the domain number and bb, dd and f is the bus, + device and function number).Quoted 8 lines
This seems like a big problem. How are we going to know to add these command line options for devices we haven't even seen/known about yet? How do we know the bus ids aren't going to change between boots (hint, they are, pci bus ids change all the time...)
We need to be able to do this kind of thing dynamically, not fixed at boot time, which seems way to early to even know about this, right? thanks,
greg k-hQuoted 3 lines
Yes, I totally agree. Doing things dynamically is better.
The purpose of these parameters is to rebalance and align resources for device that has BARs encapsulated in various new capabilities (SR-IOV, etc.), because most of existing BIOSes don't take care of those BARs.
But how are you going to know what the proper device ids are going to be before the machine boots? I don't see how these options are ever going to work properly for a "real" user.
Quoted 1 line
If we do resource rebalance after system is up, do you think there is any side effect or impact to other subsystem other than PCI (e.g. MTRR)?
I don't think so.
Quoted 1 line
I haven't had much thinking on the dynamical resource rebalance. If you have any idea about this, can you please suggest?
Yeah, it's going to be hard :)
We've thought about this in the past, and even Microsoft said it was going to happen for Vista, but they realized in the end, like we did a few years previously, that it would require full support of all PCI drivers as well (if you rebalance stuff that is already bound to a driver.) So they dropped it.
When would you want to do this kind of rebalancing? Before any PCI driver is bound to any devices? Or afterwards?
Quoted 8 lines
Quoted 5 lines
I guess if we want the rebalance dynamic, then we should have it full -- the rebalance would be functional even after the driver is loaded.
But in most cases, there will be problem when we unload driver from a hard disk controller, etc. We can mount root on a ramdisk and do the rebalance there, but it's complicated for a real user.
So looks like doing rebalancing before any driver is bound to any device is also a nice idea, if user can get a shell to do rebalance before built-in PCI driver grabs device.
That's not going to work, it needs to happen before any PCI device is bound, which is before init runs.
I don't think it can work either. Then we have to do rebalance after the driver bounding. But what should we do if we can't unload the driver (hard disk controller, etc.)?
Thanks,
Yu
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
Quoted 69 lines · Yu
Quoted 68 lines · Greg KH
Quoted 67 lines · Yu
Quoted 48 lines · Greg KH
Quoted 33 lines · Yu
Quoted 24 lines · Greg KH
Quoted 13 lines · Yu Zhao
Documentation/kernel-parameters.txt | 10 ++++++++++ 1 files changed, 10 insertions(+), 0 deletions(-)Documentation/kernel-parameters.txt +0
-0@@ -1677,6 +1677,16 @@ and is between 256 and 4096 characters. It isdefined in the file
Documentation/kernel-parameters.txt +0
-0cbmemsize=nn[KMG] The fixed amount of bus space which is reserved for the CardBus bridge's memory window. The default value is 64 megabytes.Documentation/kernel-parameters.txt
Quoted 10 lines
Documentation/kernel-parameters.txt +10
-0+ assign-mmio=[dddd:]bb [X86] reassign memory resources of all + devices under bus [dddd:]bb (dddd is the domain + number and bb is the bus number). + assign-pio=[dddd:]bb [X86] reassign io port resources of all + devices under bus [dddd:]bb (dddd is the domain + number and bb is the bus number). + align-mmio=[dddd:]bb:dd.f [X86] relocate memory resources of a + device to minimum PAGE_SIZE alignment (dddd is + the domain number and bb, dd and f is the bus, + device and function number).Quoted 8 lines
This seems like a big problem. How are we going to know to add these command line options for devices we haven't even seen/known about yet? How do we know the bus ids aren't going to change between boots (hint, they are, pci bus ids change all the time...)
We need to be able to do this kind of thing dynamically, not fixed at boot time, which seems way to early to even know about this, right? thanks,
greg k-hQuoted 5 lines
Yes, I totally agree. Doing things dynamically is better.
The purpose of these parameters is to rebalance and align resources for device that has BARs encapsulated in various new capabilities (SR-IOV, etc.), because most of existing BIOSes don't take care of those BARs.
But how are you going to know what the proper device ids are going to be before the machine boots? I don't see how these options are ever going to work properly for a "real" user.
Quoted 3 lines
If we do resource rebalance after system is up, do you think there is any side effect or impact to other subsystem other than PCI (e.g. MTRR)?
I don't think so.
Quoted 2 lines
I haven't had much thinking on the dynamical resource rebalance. If you have any idea about this, can you please suggest?
Quoted 8 lines
Yeah, it's going to be hard :)
We've thought about this in the past, and even Microsoft said it was going to happen for Vista, but they realized in the end, like we did a few years previously, that it would require full support of all PCI drivers as well (if you rebalance stuff that is already bound to a driver.) So they dropped it.
When would you want to do this kind of rebalancing? Before any PCI driver is bound to any devices? Or afterwards?I guess if we want the rebalance dynamic, then we should have it full -- the rebalance would be functional even after the driver is loaded.
But in most cases, there will be problem when we unload driver from a hard disk controller, etc. We can mount root on a ramdisk and do the rebalance there, but it's complicated for a real user.
So looks like doing rebalancing before any driver is bound to any device is also a nice idea, if user can get a shell to do rebalance before built-in PCI driver grabs device.
Quoted 6 lines
Quoted 2 lines
That's not going to work, it needs to happen before any PCI device is bound, which is before init runs.
I don't think it can work either. Then we have to do rebalance after the driver bounding. But what should we do if we can't unload the driver (hard disk controller, etc.)?
Well, to do it "correctly" you are going to have to tell the driver to shut itself down, and reinitialize itself.
Turns out, that doesn't really work for disk and network devices without dropping the connection (well, network devices should be fine probably).
So you just can't do this, sorry. That's why the BIOS handles all of these issues in a PCI hotplug system.
How does the hardware people think we are going to handle this in the OS? It's not something that any operating system can do, is it part of the IOV PCI spec somewhere?
thanks,
greg k-h
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
In reply to Zhao, YuQuoted 69 lines · Yu
Quoted 68 lines · Greg KH
Quoted 67 lines · Yu
Quoted 48 lines · Greg KH
Quoted 33 lines · Yu
Quoted 24 lines · Greg KH
Quoted 13 lines · Yu Zhao
Documentation/kernel-parameters.txt | 10 ++++++++++ 1 files changed, 10 insertions(+), 0 deletions(-)Documentation/kernel-parameters.txt +0
-0@@ -1677,6 +1677,16 @@ and is between 256 and 4096 characters. It isdefined in the file
Documentation/kernel-parameters.txt +0
-0cbmemsize=nn[KMG] The fixed amount of bus space which is reserved for the CardBus bridge's memory window. The default value is 64 megabytes.Documentation/kernel-parameters.txt
Quoted 10 lines
Documentation/kernel-parameters.txt +10
-0+ assign-mmio=[dddd:]bb [X86] reassign memory resources of all + devices under bus [dddd:]bb (dddd is the domain + number and bb is the bus number). + assign-pio=[dddd:]bb [X86] reassign io port resources of all + devices under bus [dddd:]bb (dddd is the domain + number and bb is the bus number). + align-mmio=[dddd:]bb:dd.f [X86] relocate memory resources of a + device to minimum PAGE_SIZE alignment (dddd is + the domain number and bb, dd and f is the bus, + device and function number).Quoted 8 lines
This seems like a big problem. How are we going to know to add these command line options for devices we haven't even seen/known about yet? How do we know the bus ids aren't going to change between boots (hint, they are, pci bus ids change all the time...)
We need to be able to do this kind of thing dynamically, not fixed at boot time, which seems way to early to even know about this, right? thanks,
greg k-hQuoted 5 lines
Yes, I totally agree. Doing things dynamically is better.
The purpose of these parameters is to rebalance and align resources for device that has BARs encapsulated in various new capabilities (SR-IOV, etc.), because most of existing BIOSes don't take care of those BARs.
But how are you going to know what the proper device ids are going to be before the machine boots? I don't see how these options are ever going to work properly for a "real" user.
Quoted 3 lines
If we do resource rebalance after system is up, do you think there is any side effect or impact to other subsystem other than PCI (e.g. MTRR)?
I don't think so.
Quoted 2 lines
I haven't had much thinking on the dynamical resource rebalance. If you have any idea about this, can you please suggest?
Quoted 8 lines
Yeah, it's going to be hard :)
We've thought about this in the past, and even Microsoft said it was going to happen for Vista, but they realized in the end, like we did a few years previously, that it would require full support of all PCI drivers as well (if you rebalance stuff that is already bound to a driver.) So they dropped it.
When would you want to do this kind of rebalancing? Before any PCI driver is bound to any devices? Or afterwards?I guess if we want the rebalance dynamic, then we should have it full -- the rebalance would be functional even after the driver is loaded.
But in most cases, there will be problem when we unload driver from a hard disk controller, etc. We can mount root on a ramdisk and do the rebalance there, but it's complicated for a real user.
So looks like doing rebalancing before any driver is bound to any device is also a nice idea, if user can get a shell to do rebalance before built-in PCI driver grabs device.
Quoted 6 lines
Quoted 2 lines
That's not going to work, it needs to happen before any PCI device is bound, which is before init runs.
I don't think it can work either. Then we have to do rebalance after the driver bounding. But what should we do if we can't unload the driver (hard disk controller, etc.)?
Well, to do it "correctly" you are going to have to tell the driver to shut itself down, and reinitialize itself.
Turns out, that doesn't really work for disk and network devices without dropping the connection (well, network devices should be fine probably).
So you just can't do this, sorry. That's why the BIOS handles all of these issues in a PCI hotplug system.
How does the hardware people think we are going to handle this in the OS? It's not something that any operating system can do, is it part of the IOV PCI spec somewhere?
thanks,
greg k-h
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
Quoted 46 lines · Greg KH
Quoted 30 lines · Yu
Quoted 29 lines · Greg KH
Quoted 28 lines · Yu
Quoted 27 lines · Greg KH
Quoted 9 lines · Yu
Quoted 8 lines · Greg KH
This seems like a big problem. How are we going to know to add these command line options for devices we haven't even seen/known about yet? How do we know the bus ids aren't going to change between boots (hint, they are, pci bus ids change all the time...)
We need to be able to do this kind of thing dynamically, not fixed at boot time, which seems way to early to even know about this, right? thanks,
greg k-hQuoted 3 lines
Yes, I totally agree. Doing things dynamically is better.
The purpose of these parameters is to rebalance and align resources for device that has BARs encapsulated in various new capabilities (SR-IOV, etc.), because most of existing BIOSes don't take care of those BARs.
But how are you going to know what the proper device ids are going to be before the machine boots? I don't see how these options are ever going to work properly for a "real" user.
Quoted 1 line
If we do resource rebalance after system is up, do you think there is any side effect or impact to other subsystem other than PCI (e.g. MTRR)?
I don't think so.
Quoted 1 line
I haven't had much thinking on the dynamical resource rebalance. If you have any idea about this, can you please suggest?
Yeah, it's going to be hard :)
We've thought about this in the past, and even Microsoft said it was going to happen for Vista, but they realized in the end, like we did a few years previously, that it would require full support of all PCI drivers as well (if you rebalance stuff that is already bound to a driver.) So they dropped it.
When would you want to do this kind of rebalancing? Before any PCI driver is bound to any devices? Or afterwards?Quoted 8 lines
Quoted 7 lines
Quoted 5 lines
I guess if we want the rebalance dynamic, then we should have it full -- the rebalance would be functional even after the driver is loaded.
But in most cases, there will be problem when we unload driver from a hard disk controller, etc. We can mount root on a ramdisk and do the rebalance there, but it's complicated for a real user.
So looks like doing rebalancing before any driver is bound to any device is also a nice idea, if user can get a shell to do rebalance before built-in PCI driver grabs device.
That's not going to work, it needs to happen before any PCI device is bound, which is before init runs.
I don't think it can work either. Then we have to do rebalance after the driver bounding. But what should we do if we can't unload the driver (hard disk controller, etc.)?
Well, to do it "correctly" you are going to have to tell the driver to shut itself down, and reinitialize itself.
Turns out, that doesn't really work for disk and network devices without dropping the connection (well, network devices should be fine probably).
Quoted 6 lines
So you just can't do this, sorry. That's why the BIOS handles all of these issues in a PCI hotplug system.
How does the hardware people think we are going to handle this in the OS? It's not something that any operating system can do, is it part of the IOV PCI spec somewhere?
No, it's not part of the PCI IOV spec.
I just want the IOV (and whole PCI subsystem) have more flexibility on various BIOSes. So can we reconsider about resource rebalance as boot option, or should we forget about this idea?
Regards,
Yu
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
Quoted 10 lines · Yu
Quoted 9 lines
Well, to do it "correctly" you are going to have to tell the driver to shut itself down, and reinitialize itself.
Turns out, that doesn't really work for disk and network devices without dropping the connection (well, network devices should be fine probably). So you just can't do this, sorry. That's why the BIOS handles all of these issues in a PCI hotplug system.
How does the hardware people think we are going to handle this in the OS? It's not something that any operating system can do, is it part of the IOV PCI spec somewhere?
Quoted 5 lines
No, it's not part of the PCI IOV spec.
I just want the IOV (and whole PCI subsystem) have more flexibility on various BIOSes. So can we reconsider about resource rebalance as boot option, or should we forget about this idea?
As you have proposed it, the boot option will not work at all, so I think we need to forget about it. Especially if it is not really needed.
thanks,
greg k-h
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
In reply to Zhao, YuQuoted 10 lines · Yu
Quoted 9 lines
Well, to do it "correctly" you are going to have to tell the driver to shut itself down, and reinitialize itself.
Turns out, that doesn't really work for disk and network devices without dropping the connection (well, network devices should be fine probably). So you just can't do this, sorry. That's why the BIOS handles all of these issues in a PCI hotplug system.
How does the hardware people think we are going to handle this in the OS? It's not something that any operating system can do, is it part of the IOV PCI spec somewhere?
Quoted 5 lines
No, it's not part of the PCI IOV spec.
I just want the IOV (and whole PCI subsystem) have more flexibility on various BIOSes. So can we reconsider about resource rebalance as boot option, or should we forget about this idea?
As you have proposed it, the boot option will not work at all, so I think we need to forget about it. Especially if it is not really needed.
thanks,
greg k-h
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
Quoted 10 lines · Greg KH
Quoted 9 lines · Yu
Quoted 9 lines
Well, to do it "correctly" you are going to have to tell the driver to shut itself down, and reinitialize itself.
Turns out, that doesn't really work for disk and network devices without dropping the connection (well, network devices should be fine probably). So you just can't do this, sorry. That's why the BIOS handles all of these issues in a PCI hotplug system.
How does the hardware people think we are going to handle this in the OS? It's not something that any operating system can do, is it part of the IOV PCI spec somewhere?
Quoted 7 lines
Quoted 3 lines
No, it's not part of the PCI IOV spec.
I just want the IOV (and whole PCI subsystem) have more flexibility on various BIOSes. So can we reconsider about resource rebalance as boot option, or should we forget about this idea?
As you have proposed it, the boot option will not work at all, so I think we need to forget about it. Especially if it is not really needed.
I guess at least one thing would work if people don't want to boot twice: give the bus number 0 as rebalance starting point, then all system resources would be reshuffled :-)
Thanks,
Yu
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
Quoted 16 lines · Yu
Quoted 15 lines · Greg KH
Quoted 14 lines · Yu
Quoted 9 lines
Well, to do it "correctly" you are going to have to tell the driver to shut itself down, and reinitialize itself.
Turns out, that doesn't really work for disk and network devices without dropping the connection (well, network devices should be fine probably). So you just can't do this, sorry. That's why the BIOS handles all of these issues in a PCI hotplug system.
How does the hardware people think we are going to handle this in the OS? It's not something that any operating system can do, is it part of the IOV PCI spec somewhere?No, it's not part of the PCI IOV spec.
I just want the IOV (and whole PCI subsystem) have more flexibility on various BIOSes. So can we reconsider about resource rebalance as boot option, or should we forget about this idea?
Quoted 7 lines
Quoted 3 lines
As you have proposed it, the boot option will not work at all, so I think we need to forget about it. Especially if it is not really needed.
I guess at least one thing would work if people don't want to boot twice:
give the bus number 0 as rebalance starting point, then all system resources would be reshuffled :-)
Hm, but don't we do that today with our basic resource reservation logic at boot time? What would be different about this kind of proposal?
thanks,
greg k-h
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
Quoted 20 lines · Greg KH
Quoted 19 lines · Yu
Quoted 18 lines · Greg KH
Quoted 9 lines · Yu
Quoted 9 lines
Well, to do it "correctly" you are going to have to tell the driver to shut itself down, and reinitialize itself.
Turns out, that doesn't really work for disk and network devices without dropping the connection (well, network devices should be fine probably). So you just can't do this, sorry. That's why the BIOS handles all of these issues in a PCI hotplug system.
How does the hardware people think we are going to handle this in the OS? It's not something that any operating system can do, is it part of the IOV PCI spec somewhere?Quoted 5 lines
No, it's not part of the PCI IOV spec.
I just want the IOV (and whole PCI subsystem) have more flexibility on various BIOSes. So can we reconsider about resource rebalance as boot option, or should we forget about this idea?
As you have proposed it, the boot option will not work at all, so I think we need to forget about it. Especially if it is not really needed.
Quoted 6 lines
Quoted 3 lines
I guess at least one thing would work if people don't want to boot twice:
give the bus number 0 as rebalance starting point, then all system resources would be reshuffled :-)Hm, but don't we do that today with our basic resource reservation logic at boot time? What would be different about this kind of proposal?
The generic PCI core can do this but this feature is kind of disabled by low level PCI code in x86. The low level code tries to reserve resource according to configuration from BIOS. If the BIOS is wrong, the allocation would fail and the generic PCI core couldn't repair it because the bridge resources may have been allocated by the PCI low level and the PCI core can't expand them to find enough resource for the subordinates.
The proposal is to disable x86 PCI low level to allocation resources according to BIOS so PCI core can fully control the resource allocation. The PCI core takes all resources from BARs it knows into account and configure the resource windows on the bridges according to its own calculation.
Regards,
Yu
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
In reply to Greg KHQuoted 14 lines · Greg KH
Quoted 13 lines · Yu
Quoted 12 lines · Greg KH
Quoted 11 lines · Yu
Quoted 9 lines
Well, to do it "correctly" you are going to have to tell the driver to shut itself down, and reinitialize itself.
Turns out, that doesn't really work for disk and network devices without dropping the connection (well, network devices should be fine probably). So you just can't do this, sorry. That's why the BIOS handles all of these issues in a PCI hotplug system.
How does the hardware people think we are going to handle this in the OS? It's not something that any operating system can do, is it part of the IOV PCI spec somewhere?No, it's not part of the PCI IOV spec.
Quoted 8 lines
Quoted 5 lines
Quoted 4 lines
Quoted 1 line
I just want the IOV (and whole PCI subsystem) have more flexibility on various BIOSes. So can we reconsider about resource rebalance as boot option, or should we forget about this idea?
As you have proposed it, the boot option will not work at all, so I think we need to forget about it. Especially if it is not really needed.
I guess at least one thing would work if people don't want to boot twice: give the bus number 0 as rebalance starting point, then all system resources would be reshuffled :-)
Hm, but don't we do that today with our basic resource reservation logic at boot time? What would be different about this kind of proposal?
The generic PCI core can do this but this feature is kind of disabled by low level PCI code in x86. The low level code tries to reserve resource according to configuration from BIOS. If the BIOS is wrong, the allocation would fail and the generic PCI core couldn't repair it because the bridge resources may have been allocated by the PCI low level and the PCI core can't expand them to find enough resource for the subordinates.
The proposal is to disable x86 PCI low level to allocation resources according to BIOS so PCI core can fully control the resource allocation. The PCI core takes all resources from BARs it knows into account and configure the resource windows on the bridges according to its own calculation.
Regards,
Yu
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
Quoted 30 lines · Yu Zhao
Quoted 20 lines · Greg KH
Quoted 19 lines · Yu
Quoted 18 lines · Greg KH
Quoted 17 lines · Yu
Quoted 4 lines
Well, to do it "correctly" you are going to have to tell the driver to shut itself down, and reinitialize itself.
Turns out, that doesn't really work for disk and network devices withoutQuoted 8 lines
dropping the connection (well, network devices should be fine probably).
So you just can't do this, sorry. That's why the BIOS handles all of these issues in a PCI hotplug system.
How does the hardware people think we are going to handle this in the OS? It's not something that any operating system can do, is it part of
the IOV PCI spec somewhere?No, it's not part of the PCI IOV spec.
I just want the IOV (and whole PCI subsystem) have more flexibility on various BIOSes. So can we reconsider about resource rebalance as boot option, or should we forget about this idea?
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
As you have proposed it, the boot option will not work at all, so I think we need to forget about it. Especially if it is not really needed.
I guess at least one thing would work if people don't want to boot twice:
give the bus number 0 as rebalance starting point, then all system resources would be reshuffled :-)Hm, but don't we do that today with our basic resource reservation logic at boot time? What would be different about this kind of proposal?
Quoted 6 lines
The generic PCI core can do this but this feature is kind of disabled by low level PCI code in x86. The low level code tries to reserve resource according to configuration from BIOS. If the BIOS is wrong, the allocation would fail and the generic PCI core couldn't repair it because the bridge resources may have been allocated by the PCI low level and the PCI core can't expand them to find enough resource for the subordinates.
Yes, we do this on purpose.
Quoted 5 lines
The proposal is to disable x86 PCI low level to allocation resources according to BIOS so PCI core can fully control the resource allocation. The PCI core takes all resources from BARs it knows into account and configure the resource windows on the bridges according to its own calculation.
Ah, so you mean we should revert back to the way we use to do x86 PCI resource allocation from about a year and a half ago to about 8 years ago?
Hint, there was a reason why we switched over to using the BIOS instead of doing it ourselves. Turns out we have to trust the BIOS here, as that is exactly what other operating systems do. Trying to do it on our own was too fragile and resulted in too many problems over time.
Go look at the archives for when this all was switched, you'll see the reasons why.
So no, we will not be going back to the way we used to do things, we changed for a reason :)
thanks,
greg k-h
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
In reply to Yu ZhaoQuoted 30 lines · Yu Zhao
Quoted 20 lines · Greg KH
Quoted 19 lines · Yu
Quoted 18 lines · Greg KH
Quoted 17 lines · Yu
Quoted 4 lines
Well, to do it "correctly" you are going to have to tell the driver to shut itself down, and reinitialize itself.
Turns out, that doesn't really work for disk and network devices withoutQuoted 8 lines
dropping the connection (well, network devices should be fine probably).
So you just can't do this, sorry. That's why the BIOS handles all of these issues in a PCI hotplug system.
How does the hardware people think we are going to handle this in the OS? It's not something that any operating system can do, is it part of
the IOV PCI spec somewhere?No, it's not part of the PCI IOV spec.
I just want the IOV (and whole PCI subsystem) have more flexibility on various BIOSes. So can we reconsider about resource rebalance as boot option, or should we forget about this idea?
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
As you have proposed it, the boot option will not work at all, so I think we need to forget about it. Especially if it is not really needed.
I guess at least one thing would work if people don't want to boot twice:
give the bus number 0 as rebalance starting point, then all system resources would be reshuffled :-)Hm, but don't we do that today with our basic resource reservation logic at boot time? What would be different about this kind of proposal?
Quoted 6 lines
The generic PCI core can do this but this feature is kind of disabled by low level PCI code in x86. The low level code tries to reserve resource according to configuration from BIOS. If the BIOS is wrong, the allocation would fail and the generic PCI core couldn't repair it because the bridge resources may have been allocated by the PCI low level and the PCI core can't expand them to find enough resource for the subordinates.
Yes, we do this on purpose.
Quoted 5 lines
The proposal is to disable x86 PCI low level to allocation resources according to BIOS so PCI core can fully control the resource allocation. The PCI core takes all resources from BARs it knows into account and configure the resource windows on the bridges according to its own calculation.
Ah, so you mean we should revert back to the way we use to do x86 PCI resource allocation from about a year and a half ago to about 8 years ago?
Hint, there was a reason why we switched over to using the BIOS instead of doing it ourselves. Turns out we have to trust the BIOS here, as that is exactly what other operating systems do. Trying to do it on our own was too fragile and resulted in too many problems over time.
Go look at the archives for when this all was switched, you'll see the reasons why.
So no, we will not be going back to the way we used to do things, we changed for a reason :)
thanks,
greg k-h
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
Quoted 54 lines · Greg KH
Quoted 27 lines · Yu Zhao
Quoted 26 lines · Greg KH
Quoted 25 lines · Yu
Quoted 13 lines · Greg KH
Quoted 12 lines · Yu
Quoted 12 lines
Well, to do it "correctly" you are going to have to tell the driver to shut itself down, and reinitialize itself.
Turns out, that doesn't really work for disk and network devices without
dropping the connection (well, network devices should be fine probably).
So you just can't do this, sorry. That's why the BIOS handles all of these issues in a PCI hotplug system.
How does the hardware people think we are going to handle this in the OS? It's not something that any operating system can do, is it part of
the IOV PCI spec somewhere?Quoted 8 lines
Quoted 5 lines
No, it's not part of the PCI IOV spec.
I just want the IOV (and whole PCI subsystem) have more flexibility on various BIOSes. So can we reconsider about resource rebalance as boot option, or should we forget about this idea?
As you have proposed it, the boot option will not work at all, so I think we need to forget about it. Especially if it is not really needed.
I guess at least one thing would work if people don't want to boot twice:
give the bus number 0 as rebalance starting point, then all system resources would be reshuffled :-)Quoted 8 lines
Quoted 2 lines
Hm, but don't we do that today with our basic resource reservation logic at boot time? What would be different about this kind of proposal?
The generic PCI core can do this but this feature is kind of disabled by low level PCI code in x86. The low level code tries to reserve resource according to configuration from BIOS. If the BIOS is wrong, the allocation would fail and the generic PCI core couldn't repair it because the bridge resources may have been allocated by the PCI low level and the PCI core can't expand them to find enough resource for the subordinates.
Yes, we do this on purpose.
Quoted 5 lines
The proposal is to disable x86 PCI low level to allocation resources according to BIOS so PCI core can fully control the resource allocation. The PCI core takes all resources from BARs it knows into account and configure the resource windows on the bridges according to its own calculation.
Ah, so you mean we should revert back to the way we use to do x86 PCI resource allocation from about a year and a half ago to about 8 years ago?
Hint, there was a reason why we switched over to using the BIOS instead of doing it ourselves. Turns out we have to trust the BIOS here, as that is exactly what other operating systems do. Trying to do it on our own was too fragile and resulted in too many problems over time.
Quoted 5 lines
Go look at the archives for when this all was switched, you'll see the reasons why.
So no, we will not be going back to the way we used to do things, we changed for a reason :)
So it's really a long story, and I'm glad to see the reason.
Actually there was no such thing in early SR-IOV patches, but months ago I heard some complaints that pushed me to do this kind of reverse. Looks like I have to let these complaints turn to BIOS people from now on :-)
Regards,
Yu
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
In reply to Greg KHQuoted 38 lines · Greg KH
Quoted 15 lines · Yu Zhao
Quoted 14 lines · Greg KH
Quoted 13 lines · Yu
Quoted 12 lines · Greg KH
Quoted 11 lines · Yu
Quoted 9 lines
Well, to do it "correctly" you are going to have to tell the driver to shut itself down, and reinitialize itself.
Turns out, that doesn't really work for disk and network devices without dropping the connection (well, network devices should be fine probably). So you just can't do this, sorry. That's why the BIOS handles all of these issues in a PCI hotplug system.
How does the hardware people think we are going to handle this in the OS? It's not something that any operating system can do, is it part of the IOV PCI spec somewhere?No, it's not part of the PCI IOV spec.
Quoted 8 lines
Quoted 7 lines
Quoted 5 lines
Quoted 4 lines
Quoted 1 line
I just want the IOV (and whole PCI subsystem) have more flexibility on various BIOSes. So can we reconsider about resource rebalance as boot option, or should we forget about this idea?
As you have proposed it, the boot option will not work at all, so I think we need to forget about it. Especially if it is not really needed.
I guess at least one thing would work if people don't want to boot twice: give the bus number 0 as rebalance starting point, then all system resources would be reshuffled :-)
Hm, but don't we do that today with our basic resource reservation logic at boot time? What would be different about this kind of proposal?
The generic PCI core can do this but this feature is kind of disabled by low level PCI code in x86. The low level code tries to reserve resource according to configuration from BIOS. If the BIOS is wrong, the allocation would fail and the generic PCI core couldn't repair it because the bridge resources may have been allocated by the PCI low level and the PCI core can't expand them to find enough resource for the subordinates.
Yes, we do this on purpose.
Quoted 1 line
The proposal is to disable x86 PCI low level to allocation resources according to BIOS so PCI core can fully control the resource allocation. The PCI core takes all resources from BARs it knows into account and configure the resource windows on the bridges according to its own calculation.
Ah, so you mean we should revert back to the way we use to do x86 PCI resource allocation from about a year and a half ago to about 8 years ago?
Hint, there was a reason why we switched over to using the BIOS instead of doing it ourselves. Turns out we have to trust the BIOS here, as that is exactly what other operating systems do. Trying to do it on our own was too fragile and resulted in too many problems over time.
Quoted 5 lines
Go look at the archives for when this all was switched, you'll see the reasons why.
So no, we will not be going back to the way we used to do things, we changed for a reason :)
So it's really a long story, and I'm glad to see the reason.
Actually there was no such thing in early SR-IOV patches, but months ago I heard some complaints that pushed me to do this kind of reverse. Looks like I have to let these complaints turn to BIOS people from now on :-)
Regards,
Yu
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
In reply to Yu ZhaoDear all,
I'm glad to hear this. In fact, I'm developing for BIOS area. This feature is very useful when your system have one or more PCI/PCIe hotplug slot. Generally, BIOS reserve amount of resource for empty hotplug slot by default, but it is not always enough for all device. We have many kind of Express Modules which consume different amount of resouce, generally we reserve a small number of resouce for this, so, sometime, some Express Modules hotplug without card installed at boot time, it will not useable.
Then, Microsoft say they implement PCI Multi-level Resource Rebanlence in Vista and Server 2008, you can refer
http://www.microsoft.com/whdc/archive/multilevel-rebal.mspx
http://www.microsoft.com/whdc/connect/pci/PCI-rsc.mspx
They use a method of ACPI to tell OS that you can ignore the resource allocation of PCI devices below the bridge. I think this is more useful than specify the BUS number to ignore resouce allocation, because the BUS number often change due to some need by BIOS or new PCI/PCIe device added in system. Users generally do not know the system architecture and can not specify the BUS number of the root bridge, while if you specify the _DSM method like MS to the root bridge of hotplug slot, it is a much easier way to archive for BIOS writers.
PS:
Since my mail address was blocked by the maillist, this mail may not reach people who only in linux kernel maillist.
Thanks.
Jensen
Quoted 52 lines · Yu Zhao
Quoted 29 lines · Greg KH
Quoted 27 lines · Yu
Quoted 25 lines · Greg KH
Quoted 13 lines · Yu
Well, to do it "correctly" you are going to have to tell the driver to
Quoted 12 lines
shut itself down, and reinitialize itself.
Turns out, that doesn't really work for disk and network devices without
dropping the connection (well, network devices should be fine probably).
So you just can't do this, sorry. That's why the BIOS handles all of these issues in a PCI hotplug system.
How does the hardware people think we are going to handle this in the OS? It's not something that any operating system can do, is it part of
the IOV PCI spec somewhere?Quoted 6 lines
No, it's not part of the PCI IOV spec.
I just want the IOV (and whole PCI subsystem) have more flexibility on various BIOSes. So can we reconsider about resource rebalance as boot option, or should we forget about this idea?
As you have proposed it, the boot option will not work at all, so I think we need to forget about it. Especially if it is not really needed.
Quoted 8 lines
Quoted 4 lines
I guess at least one thing would work if people don't want to boot twice:
give the bus number 0 as rebalance starting point, then all system resources would be reshuffled :-)Hm, but don't we do that today with our basic resource reservation logic at boot time? What would be different about this kind of proposal?
The generic PCI core can do this but this feature is kind of disabled by low level PCI code in x86. The low level code tries to reserve resource according to configuration from BIOS. If the BIOS is wrong, the allocation would fail and the generic PCI core couldn't repair it because the bridge resources may have been allocated by the PCI low level and the PCI core can't expand them to find enough resource for the subordinates.
The proposal is to disable x86 PCI low level to allocation resources according to BIOS so PCI core can fully control the resource allocation. The PCI core takes all resources from BARs it knows into account and configure the resource windows on the bridges according to its own calculation.
Quoted 9 lines
Regards,
Yu--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linux-foundation.org/mailman/listinfo/virtualization
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
Quoted 9 lines · freevanx
Dear all,
I'm glad to hear this. In fact, I'm developing for BIOS area. This feature is very useful when your system have one or more PCI/PCIe hotplug slot. Generally, BIOS reserve amount of resource for empty hotplug slot by default, but it is not always enough for all device. We have many kind of Express Modules which consume different amount of resouce, generally we reserve a small number of resouce for this, so, sometime, some Express Modules hotplug without card installed at boot time, it will not useable.
Then fix the BIOS :)
Seriously, that is what the PCI hotplug spec says to do, right?
Quoted 4 lines
Then, Microsoft say they implement PCI Multi-level Resource Rebanlence in Vista and Server 2008, you can refer
http://www.microsoft.com/whdc/archive/multilevel-rebal.mspx
http://www.microsoft.com/whdc/connect/pci/PCI-rsc.mspx
But they did not implement this for Vista, and pulled it before it shipped, right? That is what the driver development documentation for Vista said that I read.
Do you know if they are going to add it back for Windows 7? If so, then we should probably look into this, otherwise, no need to, as the BIOSes will be fixed properly.
Quoted 8 lines
They use a method of ACPI to tell OS that you can ignore the resource allocation of PCI devices below the bridge. I think this is more useful than specify the BUS number to ignore resouce allocation, because the BUS number often change due to some need by BIOS or new PCI/PCIe device added in system. Users generally do not know the system architecture and can not specify the BUS number of the root bridge, while if you specify the
_DSMmethod like MS to the root bridge of hotplug slot, it is a much easier way to archive for BIOS writers.
Yes, push the burden of getting this right onto the OS developers, instead of doing it properly in the BIOS, how fun :(
Seriously, it isn't that hard to reserve enough space on most machines in the BIOS to get this correct. It only gets messy when you have hundreds of hotplug PCI slots and bridges. Even then, the BIOS writers have been able to resolve this for a while due to this kind of hardware shipping successfully with Linux for many years now.
Quoted 3 lines
PS:
Since my mail address was blocked by the maillist, this mail may not reach people who only in linux kernel maillist.
It is being blocked because you are sending out html email.
Please reconfigure your gmail client to not do that, and your mail will go through just fine.
thanks,
greg k-h
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
In reply to freevanxQuoted 9 lines · freevanx
Dear all,
I'm glad to hear this. In fact, I'm developing for BIOS area. This feature is very useful when your system have one or more PCI/PCIe hotplug slot. Generally, BIOS reserve amount of resource for empty hotplug slot by default, but it is not always enough for all device. We have many kind of Express Modules which consume different amount of resouce, generally we reserve a small number of resouce for this, so, sometime, some Express Modules hotplug without card installed at boot time, it will not useable.
Then fix the BIOS :)
Seriously, that is what the PCI hotplug spec says to do, right?
Quoted 4 lines
Then, Microsoft say they implement PCI Multi-level Resource Rebanlence in Vista and Server 2008, you can refer
http://www.microsoft.com/whdc/archive/multilevel-rebal.mspx
http://www.microsoft.com/whdc/connect/pci/PCI-rsc.mspx
But they did not implement this for Vista, and pulled it before it shipped, right? That is what the driver development documentation for Vista said that I read.
Do you know if they are going to add it back for Windows 7? If so, then we should probably look into this, otherwise, no need to, as the BIOSes will be fixed properly.
Quoted 8 lines
They use a method of ACPI to tell OS that you can ignore the resource allocation of PCI devices below the bridge. I think this is more useful than specify the BUS number to ignore resouce allocation, because the BUS number often change due to some need by BIOS or new PCI/PCIe device added in system. Users generally do not know the system architecture and can not specify the BUS number of the root bridge, while if you specify the
_DSMmethod like MS to the root bridge of hotplug slot, it is a much easier way to archive for BIOS writers.
Yes, push the burden of getting this right onto the OS developers, instead of doing it properly in the BIOS, how fun :(
Seriously, it isn't that hard to reserve enough space on most machines in the BIOS to get this correct. It only gets messy when you have hundreds of hotplug PCI slots and bridges. Even then, the BIOS writers have been able to resolve this for a while due to this kind of hardware shipping successfully with Linux for many years now.
Quoted 3 lines
PS:
Since my mail address was blocked by the maillist, this mail may not reach people who only in linux kernel maillist.
It is being blocked because you are sending out html email.
Please reconfigure your gmail client to not do that, and your mail will go through just fine.
thanks,
greg k-h
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
In reply to Zhao, YuQuoted 16 lines · Yu
Quoted 15 lines · Greg KH
Quoted 14 lines · Yu
Quoted 9 lines
Well, to do it "correctly" you are going to have to tell the driver to shut itself down, and reinitialize itself.
Turns out, that doesn't really work for disk and network devices without dropping the connection (well, network devices should be fine probably). So you just can't do this, sorry. That's why the BIOS handles all of these issues in a PCI hotplug system.
How does the hardware people think we are going to handle this in the OS? It's not something that any operating system can do, is it part of the IOV PCI spec somewhere?No, it's not part of the PCI IOV spec.
I just want the IOV (and whole PCI subsystem) have more flexibility on various BIOSes. So can we reconsider about resource rebalance as boot option, or should we forget about this idea?
Quoted 7 lines
Quoted 3 lines
As you have proposed it, the boot option will not work at all, so I think we need to forget about it. Especially if it is not really needed.
I guess at least one thing would work if people don't want to boot twice:
give the bus number 0 as rebalance starting point, then all system resources would be reshuffled :-)
Hm, but don't we do that today with our basic resource reservation logic at boot time? What would be different about this kind of proposal?
thanks,
greg k-h
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
In reply to Greg KHQuoted 12 lines · Greg KH
Quoted 11 lines · Yu
Quoted 9 lines
Well, to do it "correctly" you are going to have to tell the driver to shut itself down, and reinitialize itself.
Turns out, that doesn't really work for disk and network devices without dropping the connection (well, network devices should be fine probably). So you just can't do this, sorry. That's why the BIOS handles all of these issues in a PCI hotplug system.
How does the hardware people think we are going to handle this in the OS? It's not something that any operating system can do, is it part of the IOV PCI spec somewhere?No, it's not part of the PCI IOV spec.
Quoted 7 lines
Quoted 3 lines
I just want the IOV (and whole PCI subsystem) have more flexibility on various BIOSes. So can we reconsider about resource rebalance as boot option, or should we forget about this idea?
As you have proposed it, the boot option will not work at all, so I think we need to forget about it. Especially if it is not really needed.
I guess at least one thing would work if people don't want to boot
twice: give the bus number 0 as rebalance starting point, then all system resources would be reshuffled :-)
Thanks,
Yu
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
In reply to Greg KHQuoted 58 lines · Greg KH
Quoted 42 lines · Yu
Quoted 41 lines · Greg KH
Quoted 40 lines · Yu
Quoted 24 lines · Greg KH
Quoted 9 lines · Yu
Quoted 8 lines · Greg KH
This seems like a big problem. How are we going to know to add these command line options for devices we haven't even seen/known about yet? How do we know the bus ids aren't going to change between boots (hint, they are, pci bus ids change all the time...)
We need to be able to do this kind of thing dynamically, not fixed at boot time, which seems way to early to even know about this, right? thanks,
greg k-hQuoted 5 lines
Yes, I totally agree. Doing things dynamically is better.
The purpose of these parameters is to rebalance and align resources for device that has BARs encapsulated in various new capabilities (SR-IOV, etc.), because most of existing BIOSes don't take care of those BARs.
But how are you going to know what the proper device ids are going to be before the machine boots? I don't see how these options are ever going to work properly for a "real" user.
Quoted 3 lines
If we do resource rebalance after system is up, do you think there is any side effect or impact to other subsystem other than PCI (e.g. MTRR)?
I don't think so.
Quoted 2 lines
I haven't had much thinking on the dynamical resource rebalance. If you have any idea about this, can you please suggest?
Quoted 8 lines
Yeah, it's going to be hard :)
We've thought about this in the past, and even Microsoft said it was going to happen for Vista, but they realized in the end, like we did a few years previously, that it would require full support of all PCI drivers as well (if you rebalance stuff that is already bound to a driver.) So they dropped it.
When would you want to do this kind of rebalancing? Before any PCI driver is bound to any devices? Or afterwards?I guess if we want the rebalance dynamic, then we should have it full -- the rebalance would be functional even after the driver is loaded.
But in most cases, there will be problem when we unload driver from a hard disk controller, etc. We can mount root on a ramdisk and do the rebalance there, but it's complicated for a real user.
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
So looks like doing rebalancing before any driver is bound to any device is also a nice idea, if user can get a shell to do rebalance before built-in PCI driver grabs device.
That's not going to work, it needs to happen before any PCI device is bound, which is before init runs.
I don't think it can work either. Then we have to do rebalance after the driver bounding. But what should we do if we can't unload the driver (hard disk controller, etc.)?
Well, to do it "correctly" you are going to have to tell the driver to shut itself down, and reinitialize itself.
Turns out, that doesn't really work for disk and network devices without dropping the connection (well, network devices should be fine probably).
Quoted 6 lines
So you just can't do this, sorry. That's why the BIOS handles all of these issues in a PCI hotplug system.
How does the hardware people think we are going to handle this in the OS? It's not something that any operating system can do, is it part of the IOV PCI spec somewhere?
No, it's not part of the PCI IOV spec.
I just want the IOV (and whole PCI subsystem) have more flexibility on various BIOSes. So can we reconsider about resource rebalance as boot option, or should we forget about this idea?
Regards,
Yu
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
In reply to Greg KHQuoted 64 lines · Greg KH
Quoted 63 lines · Yu
Quoted 47 lines · Greg KH
Quoted 33 lines · Yu
Quoted 24 lines · Greg KH
Quoted 13 lines · Yu Zhao
Documentation/kernel-parameters.txt | 10 ++++++++++ 1 files changed, 10 insertions(+), 0 deletions(-)Documentation/kernel-parameters.txt +0
-0@@ -1677,6 +1677,16 @@ and is between 256 and 4096 characters. It isdefined in the file
Documentation/kernel-parameters.txt +0
-0cbmemsize=nn[KMG] The fixed amount of bus space which is reserved for the CardBus bridge's memory window. The default value is 64 megabytes.Documentation/kernel-parameters.txt and is between 256 and 4096 characters. It is defined in the file · line 1689
Quoted 10 lines
Documentation/kernel-parameters.txt +10
-0+ assign-mmio=[dddd:]bb [X86] reassign memory resources of all + devices under bus [dddd:]bb (dddd is the domain + number and bb is the bus number). + assign-pio=[dddd:]bb [X86] reassign io port resources of all + devices under bus [dddd:]bb (dddd is the domain + number and bb is the bus number). + align-mmio=[dddd:]bb:dd.f [X86] relocate memory resources of a + device to minimum PAGE_SIZE alignment (dddd is + the domain number and bb, dd and f is the bus, + device and function number).Quoted 8 lines
This seems like a big problem. How are we going to know to add these command line options for devices we haven't even seen/known about yet? How do we know the bus ids aren't going to change between boots (hint, they are, pci bus ids change all the time...)
We need to be able to do this kind of thing dynamically, not fixed at boot time, which seems way to early to even know about this, right? thanks,
greg k-hQuoted 5 lines
Yes, I totally agree. Doing things dynamically is better.
The purpose of these parameters is to rebalance and align resources for device that has BARs encapsulated in various new capabilities (SR-IOV, etc.), because most of existing BIOSes don't take care of those BARs.
But how are you going to know what the proper device ids are going to be before the machine boots? I don't see how these options are ever going to work properly for a "real" user.
Quoted 2 lines
If we do resource rebalance after system is up, do you think there is any side effect or impact to other subsystem other than PCI (e.g. MTRR)?
I don't think so.
Quoted 2 lines
I haven't had much thinking on the dynamical resource rebalance. If you have any idea about this, can you please suggest?
Quoted 8 lines
Yeah, it's going to be hard :)
We've thought about this in the past, and even Microsoft said it was going to happen for Vista, but they realized in the end, like we did a few years previously, that it would require full support of all PCI drivers as well (if you rebalance stuff that is already bound to a driver.) So they dropped it.
When would you want to do this kind of rebalancing? Before any PCI driver is bound to any devices? Or afterwards?I guess if we want the rebalance dynamic, then we should have it full -- the rebalance would be functional even after the driver is loaded.
But in most cases, there will be problem when we unload driver from a hard disk controller, etc. We can mount root on a ramdisk and do the rebalance there, but it's complicated for a real user.
Quoted 6 lines
Quoted 3 lines
So looks like doing rebalancing before any driver is bound to any device is also a nice idea, if user can get a shell to do rebalance before built-in PCI driver grabs device.
That's not going to work, it needs to happen before any PCI device is bound, which is before init runs.
I don't think it can work either. Then we have to do rebalance after the driver bounding. But what should we do if we can't unload the driver (hard disk controller, etc.)?
Thanks,
Yu
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
In reply to Greg KHHi!
Quoted 21 lines
Quoted 21 lines
Quoted 5 lines
Quoted 2 lines
If we do resource rebalance after system is up, do you think there is any side effect or impact to other subsystem other than PCI (e.g. MTRR)?
I don't think so.
Quoted 2 lines
I haven't had much thinking on the dynamical resource rebalance. If you have any idea about this, can you please suggest?
Quoted 8 lines
Yeah, it's going to be hard :)
We've thought about this in the past, and even Microsoft said it was going to happen for Vista, but they realized in the end, like we did a few years previously, that it would require full support of all PCI drivers as well (if you rebalance stuff that is already bound to a driver.) So they dropped it.
When would you want to do this kind of rebalancing? Before any PCI driver is bound to any devices? Or afterwards?I guess if we want the rebalance dynamic, then we should have it full -- the rebalance would be functional even after the driver is loaded.
But in most cases, there will be problem when we unload driver from a hard disk controller, etc. We can mount root on a ramdisk and do the rebalance there, but it's complicated for a real user.
Quoted 6 lines
Quoted 3 lines
So looks like doing rebalancing before any driver is bound to any device is also a nice idea, if user can get a shell to do rebalance before built-in PCI driver grabs device.
That's not going to work, it needs to happen before any PCI device is bound, which is before init runs.
We could run shell from early initrd... And PCI is not required for initrd, right?
(Ok, I guess this is in we could do it but is it worth the cost?-category...)
Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
Quoted 25 lines
Quoted 25 lines
Quoted 24 lines
Quoted 5 lines
Quoted 2 lines
If we do resource rebalance after system is up, do you think there is any side effect or impact to other subsystem other than PCI (e.g. MTRR)?
I don't think so.
Quoted 2 lines
I haven't had much thinking on the dynamical resource rebalance. If you have any idea about this, can you please suggest?
Quoted 8 lines
Yeah, it's going to be hard :)
We've thought about this in the past, and even Microsoft said it was going to happen for Vista, but they realized in the end, like we did a few years previously, that it would require full support of all PCI drivers as well (if you rebalance stuff that is already bound to a driver.) So they dropped it.
When would you want to do this kind of rebalancing? Before any PCI driver is bound to any devices? Or afterwards?I guess if we want the rebalance dynamic, then we should have it full -- the rebalance would be functional even after the driver is loaded.
But in most cases, there will be problem when we unload driver from a hard disk controller, etc. We can mount root on a ramdisk and do the rebalance there, but it's complicated for a real user.
So looks like doing rebalancing before any driver is bound to any device is also a nice idea, if user can get a shell to do rebalance before built-in PCI driver grabs device.
Quoted 5 lines
Quoted 2 lines
That's not going to work, it needs to happen before any PCI device is bound, which is before init runs.
We could run shell from early initrd... And PCI is not required for initrd, right?
You can't be sure of that - compile your ATA driver =y and you'll definitely end up using PCI.
Alex
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
In reply to Pavel MachekQuoted 35 lines
Quoted 35 lines
Quoted 34 lines
Quoted 8 lines
Quoted 4 lines
If we do resource rebalance after system is up, do you think there is any
side effect or impact to other subsystem other than PCI (e.g. MTRR)?I don't think so.
Quoted 3 lines
I haven't had much thinking on the dynamical resource rebalance. If you
have any idea about this, can you please suggest?Quoted 10 lines
Yeah, it's going to be hard :)
We've thought about this in the past, and even Microsoft said it was
going to happen for Vista, but they realized in the end, like we did a
few years previously, that it would require full support of all PCI drivers as well (if you rebalance stuff that is already bound to a driver.) So they dropped it.
When would you want to do this kind of rebalancing? Before any PCI driver is bound to any devices? Or afterwards?I guess if we want the rebalance dynamic, then we should have it full --
the rebalance would be functional even after the driver is loaded.But in most cases, there will be problem when we unload driver from a hard
disk controller, etc. We can mount root on a ramdisk and do the rebalance
there, but it's complicated for a real user.So looks like doing rebalancing before any driver is bound to any device is
also a nice idea, if user can get a shell to do rebalance before built-in
PCI driver grabs device.
Quoted 5 lines
Quoted 2 lines
That's not going to work, it needs to happen before any PCI device is bound, which is before init runs.
We could run shell from early initrd... And PCI is not required for initrd, right?
You can't be sure of that - compile your ATA driver =y and you'll definitely end up using PCI.
Alex
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
In reply to Greg KHHi!
Quoted 21 lines
Quoted 21 lines
Quoted 5 lines
Quoted 2 lines
If we do resource rebalance after system is up, do you think there is any side effect or impact to other subsystem other than PCI (e.g. MTRR)?
I don't think so.
Quoted 2 lines
I haven't had much thinking on the dynamical resource rebalance. If you have any idea about this, can you please suggest?
Quoted 8 lines
Yeah, it's going to be hard :)
We've thought about this in the past, and even Microsoft said it was going to happen for Vista, but they realized in the end, like we did a few years previously, that it would require full support of all PCI drivers as well (if you rebalance stuff that is already bound to a driver.) So they dropped it.
When would you want to do this kind of rebalancing? Before any PCI driver is bound to any devices? Or afterwards?I guess if we want the rebalance dynamic, then we should have it full -- the rebalance would be functional even after the driver is loaded.
But in most cases, there will be problem when we unload driver from a hard disk controller, etc. We can mount root on a ramdisk and do the rebalance there, but it's complicated for a real user.
Quoted 6 lines
Quoted 3 lines
So looks like doing rebalancing before any driver is bound to any device is also a nice idea, if user can get a shell to do rebalance before built-in PCI driver grabs device.
That's not going to work, it needs to happen before any PCI device is bound, which is before init runs.
We could run shell from early initrd... And PCI is not required for initrd, right?
(Ok, I guess this is in we could do it but is it worth the cost?-category...)
Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
In reply to Greg KHQuoted 54 lines · Greg KH
Quoted 33 lines · Yu
Quoted 24 lines · Greg KH
Quoted 13 lines · Yu Zhao
Documentation/kernel-parameters.txt | 10 ++++++++++ 1 files changed, 10 insertions(+), 0 deletions(-)Documentation/kernel-parameters.txt +0
-0@@ -1677,6 +1677,16 @@ and is between 256 and 4096 characters. It isdefined in the file
Documentation/kernel-parameters.txt +0
-0cbmemsize=nn[KMG] The fixed amount of bus space which is reserved for the CardBus bridge's memory window. The default value is 64 megabytes.Documentation/kernel-parameters.txt and is between 256 and 4096 characters. It is defined in the file · line 1689
Quoted 10 lines
Documentation/kernel-parameters.txt +10
-0+ assign-mmio=[dddd:]bb [X86] reassign memory resources of all + devices under bus [dddd:]bb (dddd is the domain + number and bb is the bus number). + assign-pio=[dddd:]bb [X86] reassign io port resources of all + devices under bus [dddd:]bb (dddd is the domain + number and bb is the bus number). + align-mmio=[dddd:]bb:dd.f [X86] relocate memory resources of a + device to minimum PAGE_SIZE alignment (dddd is + the domain number and bb, dd and f is the bus, + device and function number).Quoted 8 lines
This seems like a big problem. How are we going to know to add these command line options for devices we haven't even seen/known about yet? How do we know the bus ids aren't going to change between boots (hint, they are, pci bus ids change all the time...)
We need to be able to do this kind of thing dynamically, not fixed at boot time, which seems way to early to even know about this, right? thanks,
greg k-hQuoted 5 lines
Yes, I totally agree. Doing things dynamically is better.
The purpose of these parameters is to rebalance and align resources for device that has BARs encapsulated in various new capabilities (SR-IOV, etc.), because most of existing BIOSes don't take care of those BARs.
But how are you going to know what the proper device ids are going to be before the machine boots? I don't see how these options are ever going to work properly for a "real" user.
Quoted 2 lines
If we do resource rebalance after system is up, do you think there is any side effect or impact to other subsystem other than PCI (e.g. MTRR)?
I don't think so.
Quoted 2 lines
I haven't had much thinking on the dynamical resource rebalance. If you have any idea about this, can you please suggest?
Yeah, it's going to be hard :)
Quoted 8 lines
We've thought about this in the past, and even Microsoft said it was going to happen for Vista, but they realized in the end, like we did a few years previously, that it would require full support of all PCI drivers as well (if you rebalance stuff that is already bound to a driver.) So they dropped it.
When would you want to do this kind of rebalancing? Before any PCI driver is bound to any devices? Or afterwards?
I guess if we want the rebalance dynamic, then we should have it full -- the rebalance would be functional even after the driver is loaded.
But in most cases, there will be problem when we unload driver from a hard disk controller, etc. We can mount root on a ramdisk and do the rebalance there, but it's complicated for a real user.
So looks like doing rebalancing before any driver is bound to any device is also a nice idea, if user can get a shell to do rebalance before built-in PCI driver grabs device.
Regards,
Yu
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
In reply to Zhao, YuQuoted 34 lines · Yu
Quoted 24 lines · Greg KH
Quoted 13 lines · Yu Zhao
Documentation/kernel-parameters.txt | 10 ++++++++++ 1 files changed, 10 insertions(+), 0 deletions(-)Documentation/kernel-parameters.txt +0
-0@@ -1677,6 +1677,16 @@ and is between 256 and 4096 characters. It isdefined in the file
Documentation/kernel-parameters.txt +0
-0cbmemsize=nn[KMG] The fixed amount of bus space which is reserved for the CardBus bridge's memory window. The default value is 64 megabytes.Documentation/kernel-parameters.txt and is between 256 and 4096 characters. It is defined in the file · line 1689
Quoted 10 lines
Documentation/kernel-parameters.txt +10
-0+ assign-mmio=[dddd:]bb [X86] reassign memory resources of all + devices under bus [dddd:]bb (dddd is the domain + number and bb is the bus number). + assign-pio=[dddd:]bb [X86] reassign io port resources of all + devices under bus [dddd:]bb (dddd is the domain + number and bb is the bus number). + align-mmio=[dddd:]bb:dd.f [X86] relocate memory resources of a + device to minimum PAGE_SIZE alignment (dddd is + the domain number and bb, dd and f is the bus, + device and function number).Quoted 8 lines
This seems like a big problem. How are we going to know to add these command line options for devices we haven't even seen/known about yet? How do we know the bus ids aren't going to change between boots (hint, they are, pci bus ids change all the time...)
We need to be able to do this kind of thing dynamically, not fixed at boot time, which seems way to early to even know about this, right? thanks,
greg k-h
Quoted 5 lines
Yes, I totally agree. Doing things dynamically is better.
The purpose of these parameters is to rebalance and align resources for device that has BARs encapsulated in various new capabilities (SR-IOV, etc.), because most of existing BIOSes don't take care of those BARs.
But how are you going to know what the proper device ids are going to be before the machine boots? I don't see how these options are ever going to work properly for a "real" user.
Quoted 2 lines
If we do resource rebalance after system is up, do you think there is any side effect or impact to other subsystem other than PCI (e.g. MTRR)?
I don't think so.
Quoted 2 lines
I haven't had much thinking on the dynamical resource rebalance. If you have any idea about this, can you please suggest?
Yeah, it's going to be hard :)
We've thought about this in the past, and even Microsoft said it was going to happen for Vista, but they realized in the end, like we did a few years previously, that it would require full support of all PCI drivers as well (if you rebalance stuff that is already bound to a driver.) So they dropped it.
When would you want to do this kind of rebalancing? Before any PCI driver is bound to any devices? Or afterwards?
thanks,
greg k-h
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
In reply to Greg KHQuoted 29 lines · Greg KH
Quoted 11 lines · Yu Zhao
Documentation/kernel-parameters.txt | 10 ++++++++++ 1 files changed, 10 insertions(+), 0 deletions(-)Documentation/kernel-parameters.txt +0
-0@@ -1677,6 +1677,16 @@ and is between 256 and 4096 characters. It is defined in the file cbmemsize=nn[KMG] The fixed amount of bus space which is reserved for the CardBus bridge's memory window. The default value is 64 megabytes.Documentation/kernel-parameters.txt and is between 256 and 4096 characters. It is defined in the file · line 1689
Quoted 10 lines
Documentation/kernel-parameters.txt +10
-0+ assign-mmio=[dddd:]bb [X86] reassign memory resources of all + devices under bus [dddd:]bb (dddd is the domain + number and bb is the bus number). + assign-pio=[dddd:]bb [X86] reassign io port resources of all + devices under bus [dddd:]bb (dddd is the domain + number and bb is the bus number). + align-mmio=[dddd:]bb:dd.f [X86] relocate memory resources of a + device to minimum PAGE_SIZE alignment (dddd is + the domain number and bb, dd and f is the bus, + device and function number).This seems like a big problem. How are we going to know to add these command line options for devices we haven't even seen/known about yet?
How do we know the bus ids aren't going to change between boots (hint, they are, pci bus ids change all the time...)
Quoted 6 lines
We need to be able to do this kind of thing dynamically, not fixed at boot time, which seems way to early to even know about this, right?
thanks,
greg k-h
Yes, I totally agree. Doing things dynamically is better.
The purpose of these parameters is to rebalance and align resources for device that has BARs encapsulated in various new capabilities (SR-IOV, etc.), because most of existing BIOSes don't take care of those BARs.
If we do resource rebalance after system is up, do you think there is any side effect or impact to other subsystem other than PCI (e.g. MTRR)?
I haven't had much thinking on the dynamical resource rebalance. If you have any idea about this, can you please suggest?
Regards,
Yu
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
In reply to Yu ZhaoQuoted 24 lines · Yu Zhao
Document the new PCI[x86] boot parameters.
Cc Alex Chiang <[email protected]>Cc Grant Grundler <[email protected]>Cc Greg KH <[email protected]>Cc Ingo Molnar <[email protected]>Cc Jesse Barnes <[email protected]>Cc Matthew Wilcox <[email protected]>Cc Randy Dunlap <[email protected]>Cc Roland Dreier <[email protected]>Signed-off-by Yu Zhao <[email protected]>--- Documentation/kernel-parameters.txt | 10 ++++++++++ 1 files changed, 10 insertions(+), 0 deletions(-)Documentation/kernel-parameters.txt +0
-0@@ -1677,6 +1677,16 @@ and is between 256 and 4096 characters. It is defined in the file cbmemsize=nn[KMG] The fixed amount of bus space which is reserved for the CardBus bridge's memory window. The default value is 64 megabytes.
Documentation/kernel-parameters.txt and is between 256 and 4096 characters. It is defined in the file · line 1689
Quoted 10 lines
Documentation/kernel-parameters.txt +10
-0+ assign-mmio=[dddd:]bb [X86] reassign memory resources of all + devices under bus [dddd:]bb (dddd is the domain + number and bb is the bus number). + assign-pio=[dddd:]bb [X86] reassign io port resources of all + devices under bus [dddd:]bb (dddd is the domain + number and bb is the bus number). + align-mmio=[dddd:]bb:dd.f [X86] relocate memory resources of a + device to minimum PAGE_SIZE alignment (dddd is + the domain number and bb, dd and f is the bus, + device and function number).
This seems like a big problem. How are we going to know to add these command line options for devices we haven't even seen/known about yet?
How do we know the bus ids aren't going to change between boots (hint, they are, pci bus ids change all the time...)
We need to be able to do this kind of thing dynamically, not fixed at boot time, which seems way to early to even know about this, right?
thanks,
greg k-h
PATCH v6 16/16 PCI: document the new PCI boot parameters
In reply to Yu ZhaoDocument the new PCI[x86] boot parameters.
--- Documentation/kernel-parameters.txt | 10 ++++++++++ 1 files changed, 10 insertions(+), 0 deletions(-)
Documentation/kernel-parameters.txt +10 -0
@@ -1677,6 +1677,16 @@ and is between 256 and 4096 characters. It is defined in the file cbmemsize=nn[KMG] The fixed amount of bus space which is reserved for the CardBus bridge's memory window. The default value is 64 megabytes. + assign-mmio=[dddd:]bb [X86] reassign memory resources of all + devices under bus [dddd:]bb (dddd is the domain + number and bb is the bus number). + assign-pio=[dddd:]bb [X86] reassign io port resources of all + devices under bus [dddd:]bb (dddd is the domain + number and bb is the bus number). + align-mmio=[dddd:]bb:dd.f [X86] relocate memory resources of a + device to minimum PAGE_SIZE alignment (dddd is + the domain number and bb, dd and f is the bus, + device and function number). pcie_aspm= [PCIE] Forcibly enable or disable PCIe Active State Power Management.
Signature · 1 line
1.5.6.4
Quoted 7 lines · Yu Zhao
Greetings,
Following patches are intended to support SR-IOV capability in the Linux kernel. With these patches, people can turn a PCI device with the capability into multiple ones from software perspective, which will benefit KVM and achieve other purposes such as QoS, security, and etc.
Is there any actual users of this API around yet? How was it tested as there is no hardware to test on? Which drivers are going to have to be rewritten to take advantage of this new interface?
thanks,
greg k-h
Greetings (from a new lurker to the list),
To your question Greg, "yes" and "sort of" ;-). I have started taking a look at these patches with a strong interest in understanding how they work. I've built a kernel with them and tried out a few things with real SR-IOV hardware.
--
Lance Hartmann
Quoted 14 lines · Greg KH
Is there any actual users of this API around yet? How was it tested as
there is no hardware to test on? Which drivers are going to have to be
rewritten to take advantage of this new interface?thanks,
greg k-h
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linux-foundation.org/mailman/listinfo/virtualization
Quoted 2 lines · H L
Greetings (from a new lurker to the list),
Welcome!
Quoted 4 lines
To your question Greg, "yes" and "sort of" ;-). I have started taking a look at these patches with a strong interest in understanding how they work. I've built a kernel with them and tried out a few things with real SR-IOV hardware.
Did you have to modify individual drivers to take advantage of this code? It looks like the core code will run on this type of hardware, but there seems to be no real advantage until a driver is modified to use it, right?
Or am I missing some great advantage to having this code without modified drivers?
thanks,
greg k-h
Quoted 2 lines · H L
Greetings (from a new lurker to the list),
Welcome!
Quoted 4 lines
To your question Greg, "yes" and "sort of" ;-). I have started taking a look at these patches with a strong interest in understanding how they work. I've built a kernel with them and tried out a few things with real SR-IOV hardware.
Did you have to modify individual drivers to take advantage of this code? It looks like the core code will run on this type of hardware, but there seems to be no real advantage until a driver is modified to use it, right?
Or am I missing some great advantage to having this code without modified drivers?
thanks,
greg k-h
I have not modified any existing drivers, but instead I threw together a bare-bones module enabling me to make a call to pci_iov_register() and then poke at an SR-IOV adapter's /sys entries for which no driver was loaded.
It appears from my perusal thus far that drivers using these new SR-IOV patches will require modification; i.e. the driver associated with the Physical Function (PF) will be required to make the pci_iov_register() call along with the requisite notify() function. Essentially this suggests to me a model for the PF driver to perform any "global actions" or setup on behalf of VFs before enabling them after which VF drivers could be associated.
I have so far only seen Yu Zhao's "7-patch" set. I've not yet looked at his subsequently tendered "15-patch" set so I don't know what has changed. The hardware/firmware implementation for any given SR-IOV compatible device, will determine the extent of differences required between a PF driver and a VF driver.
--
Lance Hartmann
Quoted 30 lines · Greg KH
Date: Thursday, November 6, 2008, 9:43 AM
Quoted 2 lines · H L
Greetings (from a new lurker to the list),
Welcome!
Quoted 1 line
To your question Greg, "yes" and "sort
of" ;-). I have started taking
Quoted 1 line
a look at these patches with a strong interest in
understanding how
Quoted 1 line
they work. I've built a kernel with them and
tried out a few things
Quoted 1 line
with real SR-IOV hardware.
Did you have to modify individual drivers to take advantage of this
code? It looks like the core code will run on this type of hardware,
but there seems to be no real advantage until a driver is modified to
use it, right?Or am I missing some great advantage to having this code without
modified drivers?thanks,
greg k-h
A: No.
Q: Should I include quotations after my reply?
Quoted 5 lines · H L
I have not modified any existing drivers, but instead I threw together a bare-bones module enabling me to make a call to
pci_iov_register()and then poke at an SR-IOV adapter's /sys entries for which no driver was loaded.
Quoted 7 lines
It appears from my perusal thus far that drivers using these new SR-IOV patches will require modification; i.e. the driver associated with the Physical Function (PF) will be required to make the
pci_iov_register()call along with the requisitenotify()function. Essentially this suggests to me a model for the PF driver to perform any "global actions" or setup on behalf of VFs before enabling them after which VF drivers could be associated.
Where would the VF drivers have to be associated? On the "pci_dev" level or on a higher one?
Will all drivers that want to bind to a "VF" device need to be rewritten?
Quoted 5 lines
I have so far only seen Yu Zhao's "7-patch" set. I've not yet looked at his subsequently tendered "15-patch" set so I don't know what has
changed. The hardware/firmware implementation for any given SR-IOV
compatible device, will determine the extent of differences required between a PF driver and a VF driver.
Yeah, that's what I'm worried/curious about. Without seeing the code for such a driver, how can we properly evaluate if this infrastructure is the correct one and the proper way to do all of this?
thanks,
greg k-h
A: No.
Q: Should I include quotations after my reply?
Quoted 5 lines · H L
I have not modified any existing drivers, but instead I threw together a bare-bones module enabling me to make a call to
pci_iov_register()and then poke at an SR-IOV adapter's /sys entries for which no driver was loaded.
Quoted 7 lines
It appears from my perusal thus far that drivers using these new SR-IOV patches will require modification; i.e. the driver associated with the Physical Function (PF) will be required to make the
pci_iov_register()call along with the requisitenotify()function. Essentially this suggests to me a model for the PF driver to perform any "global actions" or setup on behalf of VFs before enabling them after which VF drivers could be associated.
Where would the VF drivers have to be associated? On the "pci_dev" level or on a higher one?
Will all drivers that want to bind to a "VF" device need to be rewritten?
Quoted 5 lines
I have so far only seen Yu Zhao's "7-patch" set. I've not yet looked at his subsequently tendered "15-patch" set so I don't know what has
changed. The hardware/firmware implementation for any given SR-IOV
compatible device, will determine the extent of differences required between a PF driver and a VF driver.
Yeah, that's what I'm worried/curious about. Without seeing the code for such a driver, how can we properly evaluate if this infrastructure is the correct one and the proper way to do all of this?
thanks,
greg k-h
Quoted 15 lines · H L
Quoted 1 line
I have not modified any existing drivers, but instead I threw
together
Quoted 11 lines
a bare-bones module enabling me to make a call to
pci_iov_register()and then poke at an SR-IOV adapter's /sys entries for which no driver was loaded.It appears from my perusal thus far that drivers using these new SR-IOV patches will require modification; i.e. the driver associated with the Physical Function (PF) will be required to make the
pci_iov_register()call along with the requisitenotify()function. Essentially this suggests to me a model for the PF driver to perform any "global actions" or setup on behalf of VFs before enabling them after which VF drivers could be associated.
Quoted 2 lines
Where would the VF drivers have to be associated? On the "
pci_dev" level or on a higher one?
A VF appears to the Linux OS as a standard (full, additional) PCI device. The driver is associated in the same way as for a normal PCI device. Ideally, you would use SR-IOV devices on a virtualized system, for example, using Xen. A VF can then be assigned to a guest domain as a full PCI device.
Quoted 2 lines
Will all drivers that want to bind to a "VF" device need to be rewritten?
Currently, any vendor providing a SR-IOV device needs to provide a PF driver and a VF driver that runs on their hardware. A VF driver does not necessarily need to know much about SR-IOV but just run on the presented PCI device. You might want to have a communication channel between PF and VF driver though, for various reasons, if such a channel is not provided in hardware.
Quoted 9 lines
Quoted 5 lines
I have so far only seen Yu Zhao's "7-patch" set. I've not yet looked at his subsequently tendered "15-patch" set so I don't know what has
changed. The hardware/firmware implementation for any given SR-IOV
compatible device, will determine the extent of differences required between a PF driver and a VF driver.Yeah, that's what I'm worried/curious about. Without seeing the code for such a driver, how can we properly evaluate if this infrastructure is the correct one and the proper way to do all of this?
Yu's API allows a PF driver to register with the Linux PCI code and use it to activate VFs and allocate their resources. The PF driver needs to be modified to work with that API. While you can argue about how that API is supposed to look like, it is clear that such an API is required in some form. The PF driver needs to know when VFs are active as it might want to allocate further (device-specific) resources to VFs or initiate further (device-specific) configurations. While probably a lot of SR-IOV specific code has to be in the PF driver, there is also support required from the Linux PCI subsystem, which is to some extend provided by Yu's patches.
Anna
Quoted 15 lines · H L
Quoted 1 line
I have not modified any existing drivers, but instead I threw
together
Quoted 11 lines
a bare-bones module enabling me to make a call to
pci_iov_register()and then poke at an SR-IOV adapter's /sys entries for which no driver was loaded.It appears from my perusal thus far that drivers using these new SR-IOV patches will require modification; i.e. the driver associated with the Physical Function (PF) will be required to make the
pci_iov_register()call along with the requisitenotify()function. Essentially this suggests to me a model for the PF driver to perform any "global actions" or setup on behalf of VFs before enabling them after which VF drivers could be associated.
Quoted 2 lines
Where would the VF drivers have to be associated? On the "
pci_dev" level or on a higher one?
A VF appears to the Linux OS as a standard (full, additional) PCI device. The driver is associated in the same way as for a normal PCI device. Ideally, you would use SR-IOV devices on a virtualized system, for example, using Xen. A VF can then be assigned to a guest domain as a full PCI device.
Quoted 2 lines
Will all drivers that want to bind to a "VF" device need to be rewritten?
Currently, any vendor providing a SR-IOV device needs to provide a PF driver and a VF driver that runs on their hardware. A VF driver does not necessarily need to know much about SR-IOV but just run on the presented PCI device. You might want to have a communication channel between PF and VF driver though, for various reasons, if such a channel is not provided in hardware.
Quoted 9 lines
Quoted 5 lines
I have so far only seen Yu Zhao's "7-patch" set. I've not yet looked at his subsequently tendered "15-patch" set so I don't know what has
changed. The hardware/firmware implementation for any given SR-IOV
compatible device, will determine the extent of differences required between a PF driver and a VF driver.Yeah, that's what I'm worried/curious about. Without seeing the code for such a driver, how can we properly evaluate if this infrastructure is the correct one and the proper way to do all of this?
Yu's API allows a PF driver to register with the Linux PCI code and use it to activate VFs and allocate their resources. The PF driver needs to be modified to work with that API. While you can argue about how that API is supposed to look like, it is clear that such an API is required in some form. The PF driver needs to know when VFs are active as it might want to allocate further (device-specific) resources to VFs or initiate further (device-specific) configurations. While probably a lot of SR-IOV specific code has to be in the PF driver, there is also support required from the Linux PCI subsystem, which is to some extend provided by Yu's patches.
Anna
Quoted 15 lines · Anna
Quoted 15 lines · H L
Quoted 1 line
I have not modified any existing drivers, but instead I threw
together
Quoted 11 lines
a bare-bones module enabling me to make a call to
pci_iov_register()and then poke at an SR-IOV adapter's /sys entries for which no driver was loaded.It appears from my perusal thus far that drivers using these new SR-IOV patches will require modification; i.e. the driver associated with the Physical Function (PF) will be required to make the
pci_iov_register()call along with the requisitenotify()function. Essentially this suggests to me a model for the PF driver to perform any "global actions" or setup on behalf of VFs before enabling them after which VF drivers could be associated.
Quoted 8 lines
Quoted 2 lines
Where would the VF drivers have to be associated? On the "
pci_dev" level or on a higher one?A VF appears to the Linux OS as a standard (full, additional) PCI device. The driver is associated in the same way as for a normal PCI device. Ideally, you would use SR-IOV devices on a virtualized system, for example, using Xen. A VF can then be assigned to a guest domain as a full PCI device.
It's that "second" part that I'm worried about. How is that going to happen? Do you have any patches that show this kind of "assignment"?
Quoted 5 lines
Quoted 2 lines
Will all drivers that want to bind to a "VF" device need to be rewritten?
Currently, any vendor providing a SR-IOV device needs to provide a PF driver and a VF driver that runs on their hardware.
Are there any such drivers available yet?
Quoted 4 lines
A VF driver does not necessarily need to know much about SR-IOV but just run on the presented PCI device. You might want to have a communication channel between PF and VF driver though, for various reasons, if such a channel is not provided in hardware.
Agreed, but what does that channel look like in Linux?
I have some ideas of what I think it should look like, but if people already have code, I'd love to see that as well.
Quoted 10 lines
Quoted 9 lines
Quoted 5 lines
I have so far only seen Yu Zhao's "7-patch" set. I've not yet looked at his subsequently tendered "15-patch" set so I don't know what has
changed. The hardware/firmware implementation for any given SR-IOV
compatible device, will determine the extent of differences required between a PF driver and a VF driver.Yeah, that's what I'm worried/curious about. Without seeing the code for such a driver, how can we properly evaluate if this infrastructure is the correct one and the proper way to do all of this?
Quoted 5 lines
Yu's API allows a PF driver to register with the Linux PCI code and use it to activate VFs and allocate their resources. The PF driver needs to be modified to work with that API. While you can argue about how that API is supposed to look like, it is clear that such an API is required in some form.
I totally agree, I'm arguing about what that API looks like :)
I want to see some code...
thanks,
greg k-h
Quoted 15 lines · Anna
Quoted 15 lines · H L
Quoted 1 line
I have not modified any existing drivers, but instead I threw
together
Quoted 11 lines
a bare-bones module enabling me to make a call to
pci_iov_register()and then poke at an SR-IOV adapter's /sys entries for which no driver was loaded.It appears from my perusal thus far that drivers using these new SR-IOV patches will require modification; i.e. the driver associated with the Physical Function (PF) will be required to make the
pci_iov_register()call along with the requisitenotify()function. Essentially this suggests to me a model for the PF driver to perform any "global actions" or setup on behalf of VFs before enabling them after which VF drivers could be associated.
Quoted 8 lines
Quoted 2 lines
Where would the VF drivers have to be associated? On the "
pci_dev" level or on a higher one?A VF appears to the Linux OS as a standard (full, additional) PCI device. The driver is associated in the same way as for a normal PCI device. Ideally, you would use SR-IOV devices on a virtualized system, for example, using Xen. A VF can then be assigned to a guest domain as a full PCI device.
It's that "second" part that I'm worried about. How is that going to happen? Do you have any patches that show this kind of "assignment"?
Quoted 5 lines
Quoted 2 lines
Will all drivers that want to bind to a "VF" device need to be rewritten?
Currently, any vendor providing a SR-IOV device needs to provide a PF driver and a VF driver that runs on their hardware.
Are there any such drivers available yet?
Quoted 4 lines
A VF driver does not necessarily need to know much about SR-IOV but just run on the presented PCI device. You might want to have a communication channel between PF and VF driver though, for various reasons, if such a channel is not provided in hardware.
Agreed, but what does that channel look like in Linux?
I have some ideas of what I think it should look like, but if people already have code, I'd love to see that as well.
Quoted 10 lines
Quoted 9 lines
Quoted 5 lines
I have so far only seen Yu Zhao's "7-patch" set. I've not yet looked at his subsequently tendered "15-patch" set so I don't know what has
changed. The hardware/firmware implementation for any given SR-IOV
compatible device, will determine the extent of differences required between a PF driver and a VF driver.Yeah, that's what I'm worried/curious about. Without seeing the code for such a driver, how can we properly evaluate if this infrastructure is the correct one and the proper way to do all of this?
Quoted 5 lines
Yu's API allows a PF driver to register with the Linux PCI code and use it to activate VFs and allocate their resources. The PF driver needs to be modified to work with that API. While you can argue about how that API is supposed to look like, it is clear that such an API is required in some form.
I totally agree, I'm arguing about what that API looks like :)
I want to see some code...
thanks,
greg k-h
Quoted 30 lines
Subject: Re: [PATCH 0/16 v6] PCI: Linux kernel SR-IOV support
Quoted 4 lines · Anna
Quoted 4 lines · H L
Quoted 1 line
I have not modified any existing drivers, but instead I threw
together
Quoted 1 line
a bare-bones module enabling me to make a call to
pci_iov_register()Quoted 1 line
Quoted 1 line
Quoted 1 line
and then poke at an SR-IOV adapter's /sys entries for which no
driver
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
was loaded.
It appears from my perusal thus far that drivers using these new SR-IOV patches will require modification; i.e. the driver
associated
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
with the Physical Function (PF) will be required to make the
pci_iov_register()call along with the requisitenotify()function.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Essentially this suggests to me a model for the PF driver to
perform
Quoted 1 line
Quoted 1 line
Quoted 1 line
any "global actions" or setup on behalf of VFs before enabling
them
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
after which VF drivers could be associated.
Where would the VF drivers have to be associated? On the "
pci_dev" level or on a higher one?A VF appears to the Linux OS as a standard (full, additional) PCI device. The driver is associated in the same way as for a normal PCI device. Ideally, you would use SR-IOV devices on a virtualized
Quoted 7 lines
system,
Quoted 1 line
for example, using Xen. A VF can then be assigned to a guest domain
as
Quoted 1 line
a full PCI device.
It's that "second" part that I'm worried about. How is that going to happen? Do you have any patches that show this kind of "assignment"?
That depends on your setup. Using Xen, you could assign the VF to a guest domain like any other PCI device, e.g. using PCI pass-through. For VMware, KVM, there are standard ways to do that, too. I currently don't see why SR-IOV devices would need any specific, non-standard mechanism for device assignment.
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
Will all drivers that want to bind to a "VF" device need to be rewritten?
Currently, any vendor providing a SR-IOV device needs to provide a PF driver and a VF driver that runs on their hardware.
Are there any such drivers available yet?
I don't know.
Quoted 9 lines
Quoted 4 lines
A VF driver does not necessarily need to know much about SR-IOV but just run on the presented PCI device. You might want to have a communication channel between PF and VF driver though, for various reasons, if such a channel is not provided in hardware.
Agreed, but what does that channel look like in Linux?
I have some ideas of what I think it should look like, but if people already have code, I'd love to see that as well.
At this point I would guess that this code is vendor specific, as are the drivers. The issue I see is that most likely drivers will run in different environments, for example, in Xen the PF driver runs in a driver domain while a VF driver runs in a guest VM. So a communication channel would need to be either Xen specific, or vendor specific. Also, a guest using the VF might run Windows while the PF might be controlled under Linux.
Anna
Quoted 30 lines
Subject: Re: [PATCH 0/16 v6] PCI: Linux kernel SR-IOV support
Quoted 4 lines · Anna
Quoted 4 lines · H L
Quoted 1 line
I have not modified any existing drivers, but instead I threw
together
Quoted 1 line
a bare-bones module enabling me to make a call to
pci_iov_register()Quoted 1 line
Quoted 1 line
Quoted 1 line
and then poke at an SR-IOV adapter's /sys entries for which no
driver
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
was loaded.
It appears from my perusal thus far that drivers using these new SR-IOV patches will require modification; i.e. the driver
associated
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
with the Physical Function (PF) will be required to make the
pci_iov_register()call along with the requisitenotify()function.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Essentially this suggests to me a model for the PF driver to
perform
Quoted 1 line
Quoted 1 line
Quoted 1 line
any "global actions" or setup on behalf of VFs before enabling
them
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
after which VF drivers could be associated.
Where would the VF drivers have to be associated? On the "
pci_dev" level or on a higher one?A VF appears to the Linux OS as a standard (full, additional) PCI device. The driver is associated in the same way as for a normal PCI device. Ideally, you would use SR-IOV devices on a virtualized
Quoted 7 lines
system,
Quoted 1 line
for example, using Xen. A VF can then be assigned to a guest domain
as
Quoted 1 line
a full PCI device.
It's that "second" part that I'm worried about. How is that going to happen? Do you have any patches that show this kind of "assignment"?
That depends on your setup. Using Xen, you could assign the VF to a guest domain like any other PCI device, e.g. using PCI pass-through. For VMware, KVM, there are standard ways to do that, too. I currently don't see why SR-IOV devices would need any specific, non-standard mechanism for device assignment.
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
Will all drivers that want to bind to a "VF" device need to be rewritten?
Currently, any vendor providing a SR-IOV device needs to provide a PF driver and a VF driver that runs on their hardware.
Are there any such drivers available yet?
I don't know.
Quoted 9 lines
Quoted 4 lines
A VF driver does not necessarily need to know much about SR-IOV but just run on the presented PCI device. You might want to have a communication channel between PF and VF driver though, for various reasons, if such a channel is not provided in hardware.
Agreed, but what does that channel look like in Linux?
I have some ideas of what I think it should look like, but if people already have code, I'd love to see that as well.
At this point I would guess that this code is vendor specific, as are the drivers. The issue I see is that most likely drivers will run in different environments, for example, in Xen the PF driver runs in a driver domain while a VF driver runs in a guest VM. So a communication channel would need to be either Xen specific, or vendor specific. Also, a guest using the VF might run Windows while the PF might be controlled under Linux.
Anna
Quoted 4 lines · Greg KH
It's that "second" part that I'm worried about. How is that going to happen? Do you have any patches that show this kind of "assignment"?
For kvm, this is in 2.6.28-rc.
Note there are two ways to assign a device to a guest:
- run the VF driver in the guest: this has the advantage of best performance, but requires pinning all guest memory, makes live migration a tricky proposition, and ties the guest to the underlying hardware.
- run the VF driver in the host, and use virtio to connect the guest to the host: allows paging the guest and allows straightforward live migration, but reduces performance, and hides any features not exposed by virtio from the guest.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 7 lines · Avi Kivity
Quoted 4 lines · Greg KH
It's that "second" part that I'm worried about. How is that going to happen? Do you have any patches that show this kind of "assignment"?
For kvm, this is in 2.6.28-rc.
Where? I just looked and couldn't find anything, but odds are I was looking in the wrong place :(
Quoted 5 lines
Note there are two ways to assign a device to a guest:
- run the VF driver in the guest: this has the advantage of best performance, but requires pinning all guest memory, makes live migration a tricky proposition, and ties the guest to the underlying hardware.
Is this what you would prefer for kvm?
Quoted 4 lines
- run the VF driver in the host, and use virtio to connect the guest to the
host: allows paging the guest and allows straightforward live migration, but reduces performance, and hides any features not exposed by virtio from the guest.
thanks,
greg k-h
Quoted 7 lines · Avi Kivity
Quoted 4 lines · Greg KH
It's that "second" part that I'm worried about. How is that going to happen? Do you have any patches that show this kind of "assignment"?
For kvm, this is in 2.6.28-rc.
Where? I just looked and couldn't find anything, but odds are I was looking in the wrong place :(
Quoted 5 lines
Note there are two ways to assign a device to a guest:
- run the VF driver in the guest: this has the advantage of best performance, but requires pinning all guest memory, makes live migration a tricky proposition, and ties the guest to the underlying hardware.
Is this what you would prefer for kvm?
Quoted 4 lines
- run the VF driver in the host, and use virtio to connect the guest to the
host: allows paging the guest and allows straightforward live migration, but reduces performance, and hides any features not exposed by virtio from the guest.
thanks,
greg k-h
Quoted 9 lines · Greg KH
Quoted 7 lines · Avi Kivity
Quoted 5 lines · Greg KH
It's that "second" part that I'm worried about. How is that going to happen? Do you have any patches that show this kind of "assignment"?
Quoted 7 lines
Quoted 2 lines
For kvm, this is in 2.6.28-rc.
Where? I just looked and couldn't find anything, but odds are I was looking in the wrong place :(
arch/x86/kvm/vtd.c: iommu integration (allows assigning the device's memory resources)virt/kvm/irq*: interrupt redirection (allows assigning the device's interrupt resources)
the rest (pci config space, pio redirection) are in userspace.
Quoted 10 lines
Quoted 6 lines
Note there are two ways to assign a device to a guest:
- run the VF driver in the guest: this has the advantage of best performance, but requires pinning all guest memory, makes live migration a tricky proposition, and ties the guest to the underlying hardware.
Is this what you would prefer for kvm?
It's not my personal preference, but it is a supported configuration. For some use cases it is the only one that makes sense.
Again, VF-in-guest and VF-in-host both have their places. And since Linux can be both guest and host, it's best if the VF driver knows nothing about SR-IOV; it's just a pci driver. The PF driver should emulate anything that SR-IOV does not provide (like missing pci config space).
Signature · 2 lines
I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain.
Quoted 12 lines · Avi Kivity
Quoted 11 lines · Greg KH
Quoted 8 lines · Avi Kivity
Quoted 4 lines · Greg KH
It's that "second" part that I'm worried about. How is that going to happen? Do you have any patches that show this kind of "assignment"?
For kvm, this is in 2.6.28-rc.
Quoted 7 lines
Quoted 4 lines
Where? I just looked and couldn't find anything, but odds are I was looking in the wrong place :(
arch/x86/kvm/vtd.c: iommu integration (allows assigning the device's memory resources)
That file is not in 2.6.28-rc4 :(
Quoted 2 lines
virt/kvm/irq*: interrupt redirection (allows assigning the device's interrupt resources)
I only see virt/kvm/irq_comm.c in 2.6.28-rc4.
Quoted 1 line
the rest (pci config space, pio redirection) are in userspace.
So you don't need these pci core changes at all?
Quoted 9 lines
Quoted 8 lines
Quoted 5 lines
Note there are two ways to assign a device to a guest:
- run the VF driver in the guest: this has the advantage of best performance, but requires pinning all guest memory, makes live migration a tricky proposition, and ties the guest to the underlying hardware.
Is this what you would prefer for kvm?
Quoted 7 lines
It's not my personal preference, but it is a supported configuration. For some use cases it is the only one that makes sense.
Again, VF-in-guest and VF-in-host both have their places. And since Linux can be both guest and host, it's best if the VF driver knows nothing about SR-IOV; it's just a pci driver. The PF driver should emulate anything that SR-IOV does not provide (like missing pci config space).
Yes, we need both.
thanks,
greg k-h
Quoted 7 lines · Greg KH
Quoted 3 lines
arch/x86/kvm/vtd.c: iommu integration (allows assigning the device's memory resources)That file is not in 2.6.28-rc4 :(
Sorry, was moved to virt/kvm/ for ia64's benefit.
Quoted 8 lines
Quoted 4 lines
virt/kvm/irq*: interrupt redirection (allows assigning the device's interrupt resources)I only see
virt/kvm/irq_comm.cin 2.6.28-rc4.
kvm_main.c in that directory also has some related bits.
Quoted 6 lines
Quoted 2 lines
the rest (pci config space, pio redirection) are in userspace.
So you don't need these pci core changes at all?
Not beyond those required for SR-IOV and iommu support.
Signature · 2 lines
I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain.
Quoted 12 lines · Avi Kivity
Quoted 11 lines · Greg KH
Quoted 8 lines · Avi Kivity
Quoted 4 lines · Greg KH
It's that "second" part that I'm worried about. How is that going to happen? Do you have any patches that show this kind of "assignment"?
For kvm, this is in 2.6.28-rc.
Quoted 7 lines
Quoted 4 lines
Where? I just looked and couldn't find anything, but odds are I was looking in the wrong place :(
arch/x86/kvm/vtd.c: iommu integration (allows assigning the device's memory resources)
That file is not in 2.6.28-rc4 :(
Quoted 2 lines
virt/kvm/irq*: interrupt redirection (allows assigning the device's interrupt resources)
I only see virt/kvm/irq_comm.c in 2.6.28-rc4.
Quoted 1 line
the rest (pci config space, pio redirection) are in userspace.
So you don't need these pci core changes at all?
Quoted 9 lines
Quoted 8 lines
Quoted 5 lines
Note there are two ways to assign a device to a guest:
- run the VF driver in the guest: this has the advantage of best performance, but requires pinning all guest memory, makes live migration a tricky proposition, and ties the guest to the underlying hardware.
Is this what you would prefer for kvm?
Quoted 7 lines
It's not my personal preference, but it is a supported configuration. For some use cases it is the only one that makes sense.
Again, VF-in-guest and VF-in-host both have their places. And since Linux can be both guest and host, it's best if the VF driver knows nothing about SR-IOV; it's just a pci driver. The PF driver should emulate anything that SR-IOV does not provide (like missing pci config space).
Yes, we need both.
thanks,
greg k-h
Quoted 4 lines · Greg KH
It's that "second" part that I'm worried about. How is that going to happen? Do you have any patches that show this kind of "assignment"?
For kvm, this is in 2.6.28-rc.
Note there are two ways to assign a device to a guest:
- run the VF driver in the guest: this has the advantage of best performance, but requires pinning all guest memory, makes live migration a tricky proposition, and ties the guest to the underlying hardware.
- run the VF driver in the host, and use virtio to connect the guest to the host: allows paging the guest and allows straightforward live migration, but reduces performance, and hides any features not exposed by virtio from the guest.
Signature · 1 line
error compiling committee.c: too many arguments to function
[Anna, can you fix your word-wrapping please? Your lines appear to be infinitely long which is most unpleasant to reply to]
Quoted 8 lines · Anna
Quoted 2 lines
Where would the VF drivers have to be associated? On the "
pci_dev" level or on a higher one?A VF appears to the Linux OS as a standard (full, additional) PCI device. The driver is associated in the same way as for a normal PCI device. Ideally, you would use SR-IOV devices on a virtualized system, for example, using Xen. A VF can then be assigned to a guest domain as a full PCI device.
It's not clear thats the right solution. If the VF devices are _only_ going to be used by the guest, then arguably, we don't want to create pci_devs for them in the host. (I think it _is_ the right answer, but I want to make it clear there's multiple opinions on this).
Quoted 9 lines
Quoted 2 lines
Will all drivers that want to bind to a "VF" device need to be rewritten?
Currently, any vendor providing a SR-IOV device needs to provide a PF driver and a VF driver that runs on their hardware. A VF driver does not necessarily need to know much about SR-IOV but just run on the presented PCI device. You might want to have a communication channel between PF and VF driver though, for various reasons, if such a channel is not provided in hardware.
That is one model. Another model is to provide one driver that can handle both PF and VF devices. A third model is to provide, say, a Windows VF driver and a Xen PF driver and only support Windows-on-Xen. (This last would probably be an exercise in foot-shooting, but nevertheless, I've heard it mooted).
Quoted 4 lines
Quoted 3 lines
Yeah, that's what I'm worried/curious about. Without seeing the code for such a driver, how can we properly evaluate if this infrastructure is the correct one and the proper way to do all of this?
Quoted 9 lines
Yu's API allows a PF driver to register with the Linux PCI code and use it to activate VFs and allocate their resources. The PF driver needs to be modified to work with that API. While you can argue about how that API is supposed to look like, it is clear that such an API is required in some form. The PF driver needs to know when VFs are active as it might want to allocate further (device-specific) resources to VFs or initiate further (device-specific) configurations. While probably a lot of SR-IOV specific code has to be in the PF driver, there is also support required from the Linux PCI subsystem, which is to some extend provided by Yu's patches.
Everyone agrees that some support is necessary. The question is exactly what it looks like. I must confess to not having reviewed this latest patch series yet -- I'm a little burned out on patch review.
Signature · 4 lines
Matthew Wilcox Intel Open Source Technology Centre "Bill, look, we understand that you're interested in selling us this operating system, but compare it to ours. We can't possibly take such a retrograde step."
Quoted 15 lines · Matthew Wilcox
[Anna, can you fix your word-wrapping please? Your lines appear to be infinitely long which is most unpleasant to reply to]
Quoted 9 lines · Anna
Quoted 3 lines
Where would the VF drivers have to be associated? On the "
pci_dev" level or on a higher one?A VF appears to the Linux OS as a standard (full, additional) PCI device. The driver is associated in the same way as for a normal PCI device. Ideally, you would use SR-IOV devices on a virtualized system, for example, using Xen. A VF can then be assigned to a guest domain as a full PCI device.
Quoted 5 lines
It's not clear thats the right solution. If the VF devices are _only_ going to be used by the guest, then arguably, we don't want to create
pci_devsfor them in the host. (I think it _is_ the right answer, but I want to make it clear there's multiple opinions on this).
The VFs shouldn't be limited to being used by the guest.
SR-IOV is actually an incredibly painful thing. You need to have a VF driver in the guest, do hardware pass through, have a PV driver stub in the guest that's hypervisor specific (a VF is not usable on it's own), have a device specific backend in the VMM, and if you want to do live migration, have another PV driver in the guest that you can do teaming with. Just a mess.
What we would rather do in KVM, is have the VFs appear in the host as standard network devices. We would then like to back our existing PV driver to this VF directly bypassing the host networking stack. A key feature here is being able to fill the VF's receive queue with guest memory instead of host kernel memory so that you can get zero-copy receive traffic. This will perform just as well as doing passthrough (at least) and avoid all that ugliness of dealing with SR-IOV in the guest.
This eliminates all of the mess of various drivers in the guest and all the associated baggage of doing hardware passthrough.
So IMHO, having VFs be usable in the host is absolutely critical because I think it's the only reasonable usage model.
Regards,
Anthony Liguori
Quoted 5 lines · Anthony Liguori
Quoted 4 lines
It's not clear thats the right solution. If the VF devices are _only_ going to be used by the guest, then arguably, we don't want to create
pci_devsfor them in the host. (I think it _is_ the right answer, but I want to make it clear there's multiple opinions on this).
Quoted 8 lines
The VFs shouldn't be limited to being used by the guest.
SR-IOV is actually an incredibly painful thing. You need to have a VF driver in the guest, do hardware pass through, have a PV driver stub in the guest that's hypervisor specific (a VF is not usable on it's own), have a device specific backend in the VMM, and if you want to do live migration, have another PV driver in the guest that you can do teaming with. Just a mess.
Not to mention that you basically have to statically allocate them up front.
Quoted 7 lines
What we would rather do in KVM, is have the VFs appear in the host as standard network devices. We would then like to back our existing PV driver to this VF directly bypassing the host networking stack. A key feature here is being able to fill the VF's receive queue with guest memory instead of host kernel memory so that you can get zero-copy receive traffic. This will perform just as well as doing passthrough (at least) and avoid all that ugliness of dealing with SR-IOV in the guest.
This argues for ignoring the SR-IOV mess completely. Just have the host driver expose multiple 'ethN' devices.
Quoted 13 lines
This eliminates all of the mess of various drivers in the guest and all the associated baggage of doing hardware passthrough.
So IMHO, having VFs be usable in the host is absolutely critical because I think it's the only reasonable usage model.
Regards,
Anthony Liguori
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Signature · 4 lines
Matthew Wilcox Intel Open Source Technology Centre "Bill, look, we understand that you're interested in selling us this operating system, but compare it to ours. We can't possibly take such a retrograde step."
Quoted 10 lines · Matthew Wilcox
Quoted 7 lines
What we would rather do in KVM, is have the VFs appear in the host as standard network devices. We would then like to back our existing PV driver to this VF directly bypassing the host networking stack. A key feature here is being able to fill the VF's receive queue with guest memory instead of host kernel memory so that you can get zero-copy receive traffic. This will perform just as well as doing passthrough (at least) and avoid all that ugliness of dealing with SR-IOV in the guest.
This argues for ignoring the SR-IOV mess completely. Just have the host driver expose multiple 'ethN' devices.
That would work, but do we want to do that for every different type of driver?
thanks,
greg k-h
While we are arguing what the software model the SR-IOV should be, let me ask two simple questions first:
1, What does the SR-IOV looks like?
2, Why do we need to support it?
I'm sure people have different understandings from their own view points. No one is wrong, but, please don't make thing complicated and don't ignore user requirements.
PCI SIG and hardware vendors create such thing intending to make hardware resource in one PCI device be shared from different software instances -- I guess all of us agree with this. No doubt PF is real function in the PCI device, but VF is different? No, it also has its own Bus, Device and Function numbers, and PCI configuration space and Memory Space (MMIO). To be more detailed, it can response to and initiate PCI Transaction Layer Protocol packets, which means it can do everything a PF can in PCI level. From these obvious behaviors, we can conclude PCI SIG model VF as a normal PCI device function, even it's not standalone.
As you know the Linux kernel is the base of various virtual machine monitors such as KVM, Xen, OpenVZ and VServer. We need SR-IOV support in the kernel because mostly it helps high-end users (IT departments, HPC, etc.) to share limited hardware resources among hundreds or even thousands virtual machines and hence reduce the cost. How can we make these virtual machine monitors utilize the advantage of SR-IOV without spending too much effort meanwhile remaining architectural correctness? I believe making VF represent as much closer as a normal PCI device (struct pci_dev) is the best way in current situation, because this is not only what the hardware designers expect us to do but also the usage model that KVM, Xen and other VMMs have already supported.
I agree that API in the SR-IOV pacth is arguable and the concerns such as lack of PF driver, etc. are also valid. But I personally think these stuff are not essential problems to me and other SR-IOV driver developers. People can refine things but don't want to recreate things in another totally different way especially that way doesn't bring them obvious benefits.
As I can see that we are now reaching a point that a decision must be made, I know this is such difficult thing in an open and free community but fortunately we have a lot of talented and experienced people here. So let's make it happen, and keep our loyal users happy!
Thanks,
Yu
While we are arguing what the software model the SR-IOV should be, let me ask two simple questions first:
1, What does the SR-IOV looks like?
2, Why do we need to support it?
I'm sure people have different understandings from their own view points. No one is wrong, but, please don't make thing complicated and don't ignore user requirements.
PCI SIG and hardware vendors create such thing intending to make hardware resource in one PCI device be shared from different software instances -- I guess all of us agree with this. No doubt PF is real function in the PCI device, but VF is different? No, it also has its own Bus, Device and Function numbers, and PCI configuration space and Memory Space (MMIO). To be more detailed, it can response to and initiate PCI Transaction Layer Protocol packets, which means it can do everything a PF can in PCI level. From these obvious behaviors, we can conclude PCI SIG model VF as a normal PCI device function, even it's not standalone.
As you know the Linux kernel is the base of various virtual machine monitors such as KVM, Xen, OpenVZ and VServer. We need SR-IOV support in the kernel because mostly it helps high-end users (IT departments, HPC, etc.) to share limited hardware resources among hundreds or even thousands virtual machines and hence reduce the cost. How can we make these virtual machine monitors utilize the advantage of SR-IOV without spending too much effort meanwhile remaining architectural correctness? I believe making VF represent as much closer as a normal PCI device (struct pci_dev) is the best way in current situation, because this is not only what the hardware designers expect us to do but also the usage model that KVM, Xen and other VMMs have already supported.
I agree that API in the SR-IOV pacth is arguable and the concerns such as lack of PF driver, etc. are also valid. But I personally think these stuff are not essential problems to me and other SR-IOV driver developers. People can refine things but don't want to recreate things in another totally different way especially that way doesn't bring them obvious benefits.
As I can see that we are now reaching a point that a decision must be made, I know this is such difficult thing in an open and free community but fortunately we have a lot of talented and experienced people here. So let's make it happen, and keep our loyal users happy!
Thanks,
Yu
Quoted 5 lines · Yu Zhao
While we are arguing what the software model the SR-IOV should be, let me ask two simple questions first:
1, What does the SR-IOV looks like?
2, Why do we need to support it?
I don't think we need to worry about those questions, as we can see what the SR-IOV interface looks like by looking at the PCI spec, and we know Linux needs to support it, as Linux needs to support everything :)
(note, community members that can not see the PCI specs at this point in time, please know that we are working on resolving these issues, hopefully we will have some good news within a month or so.)
Quoted 11 lines
As you know the Linux kernel is the base of various virtual machine monitors such as KVM, Xen, OpenVZ and VServer. We need SR-IOV support in the kernel because mostly it helps high-end users (IT departments, HPC, etc.) to share limited hardware resources among hundreds or even thousands virtual machines and hence reduce the cost. How can we make these virtual machine monitors utilize the advantage of SR-IOV without spending too much effort meanwhile remaining architectural correctness? I believe making VF represent as much closer as a normal PCI device (struct
pci_dev) is the best way in current situation, because this is not only what the hardware designers expect us to do but also the usage model that KVM, Xen and other VMMs have already supported.
But would such an api really take advantage of the new IOV interfaces that are exposed by the new device type?
Quoted 3 lines
I agree that API in the SR-IOV pacth is arguable and the concerns such as lack of PF driver, etc. are also valid. But I personally think these stuff are not essential problems to me and other SR-IOV driver developers.
How can the lack of a PF driver not be a valid concern at this point in time? Without such a driver written, how can we know that the SR-IOV interface as created is sufficient, or that it even works properly?
Here's what I see we need to have before we can evaluate if the IOV core PCI patches are acceptable:
- a driver that uses this interface
- a PF driver that uses this interface.
Without those, we can't determine if the infrastructure provided by the IOV core even is sufficient, right?
Rumor has it that there is both of the above things floating around, can someone please post them to the linux-pci list so that we can see how this all works together?
thanks,
greg k-h
Quoted 36 lines
Subject: Re: [PATCH 0/16 v6] PCI: Linux kernel SR-IOV support
Importance: HighQuoted 1 line · Yu Zhao
While we are arguing what the software model the SR-IOV should be,
let me
Quoted 4 lines
ask two simple questions first:
1, What does the SR-IOV looks like?
2, Why do we need to support it?I don't think we need to worry about those questions, as we can see what
the SR-IOV interface looks like by looking at the PCI spec, and we know Linux needs to support it, as Linux needs to support everything :)(note, community members that can not see the PCI specs at this point in
time, please know that we are working on resolving these issues, hopefully we will have some good news within a month or so.)Quoted 2 lines
As you know the Linux kernel is the base of various virtual machine monitors such as KVM, Xen, OpenVZ and VServer. We need SR-IOV support
in
Quoted 1 line
the kernel because mostly it helps high-end users (IT departments,
HPC,
Quoted 1 line
etc.) to share limited hardware resources among hundreds or even
thousands
Quoted 1 line
virtual machines and hence reduce the cost. How can we make these
virtual
Quoted 1 line
machine monitors utilize the advantage of SR-IOV without spending too
much
Quoted 1 line
effort meanwhile remaining architectural correctness? I believe
making VF
Quoted 1 line
represent as much closer as a normal PCI device (struct
pci_dev) isthe
Quoted 8 lines
Quoted 1 line
best way in current situation, because this is not only what the
hardware
Quoted 1 line
designers expect us to do but also the usage model that KVM, Xen and
other
Quoted 1 line
VMMs have already supported.
But would such an api really take advantage of the new IOV interfaces that are exposed by the new device type?
I agree with what Yu says. The idea is to have hardware capabilities to virtualize a PCI device in a way that those virtual devices can represent full PCI devices. The advantage of that is that those virtual device can then be used like any other standard PCI device, meaning we can use existing OS tools, configuration mechanism etc. to start working with them. Also, when using a virtualization-based system, e.g. Xen or KVM, we do not need to introduce new mechanisms to make use of SR-IOV, because we can handle VFs as full PCI devices.
A virtual PCI device in hardware (a VF) can be as powerful or complex as you like, or it can be very simple. But the big advantage of SR-IOV is that hardware presents a complete PCI device to the OS - as opposed to some resources, or queues, that need specific new configuration and assignment mechanisms in order to use them with a guest OS (like, for example, VMDq or similar technologies).
Anna
Quoted 8 lines
-----Original Message-----
From: Fischer, Anna [mailto:[email protected]]
Sent: Saturday, November 08, 2008 3:10 AM
To: Greg KH; Yu Zhao
Cc: Matthew Wilcox; Anthony Liguori; H L; [email protected]; [email protected]; Chiang, Alexander;
Quoted 1 line
Quoted 8 lines
[email protected]; [email protected]; [email protected]; [email protected]; Leonid Grossman; [email protected]; [email protected]; [email protected]
Subject: RE: [PATCH 0/16 v6] PCI: Linux kernel SR-IOV supportQuoted 1 line
But would such an api really take advantage of the new IOV
interfaces
Quoted 3 lines
Quoted 1 line
that are exposed by the new device type?
I agree with what Yu says. The idea is to have hardware capabilities
to
Quoted 1 line
virtualize a PCI device in a way that those virtual devices can
represent
Quoted 1 line
full PCI devices. The advantage of that is that those virtual device
can
Quoted 3 lines
then be used like any other standard PCI device, meaning we can use existing
OS tools, configuration mechanism etc. to start working with them.
Also,
Quoted 3 lines
when
using a virtualization-based system, e.g. Xen or KVM, we do not need to introduce new mechanisms to make use of SR-IOV, because we can
handle
Quoted 3 lines
VFs as full PCI devices.
A virtual PCI device in hardware (a VF) can be as powerful or complex
as
Quoted 7 lines
you like, or it can be very simple. But the big advantage of SR-IOV is that hardware presents a complete PCI device to the OS - as opposed to some resources, or queues, that need specific new configuration and assignment mechanisms in order to use them with a guest OS (like, for example, VMDq or similar technologies).
Anna
Ditto.
Taking netdev interface as an example - a queue pair is a great way to scale across cpu cores in a single OS image, but it is just not a good way to share device across multiple OS images.
The best unit of virtualization is a VF that is implemented as a complete netdev pci device (not a subset of a pci device).
This way, native netdev device drivers can work for direct hw access to
a VF "as is", and most/all Linux networking features (including VMQ) will work in a guest.
Also, guest migration for netdev interfaces (both direct and virtual) can be supported via native Linux mechanism (bonding driver), while Dom0 can retain "veto power" over any guest direct interface operation it deems privileged (vlan, mac address, promisc mode, bandwidth allocation between VFs, etc.).
Leonid
Quoted 8 lines
-----Original Message-----
From: Fischer, Anna [mailto:[email protected]]
Sent: Saturday, November 08, 2008 3:10 AM
To: Greg KH; Yu Zhao
Cc: Matthew Wilcox; Anthony Liguori; H L; [email protected]; [email protected]; Chiang, Alexander;
Quoted 1 line
Quoted 8 lines
[email protected]; [email protected]; [email protected]; [email protected]; Leonid Grossman; [email protected]; [email protected]; [email protected]
Subject: RE: [PATCH 0/16 v6] PCI: Linux kernel SR-IOV supportQuoted 1 line
But would such an api really take advantage of the new IOV
interfaces
Quoted 3 lines
Quoted 1 line
that are exposed by the new device type?
I agree with what Yu says. The idea is to have hardware capabilities
to
Quoted 1 line
virtualize a PCI device in a way that those virtual devices can
represent
Quoted 1 line
full PCI devices. The advantage of that is that those virtual device
can
Quoted 3 lines
then be used like any other standard PCI device, meaning we can use existing
OS tools, configuration mechanism etc. to start working with them.
Also,
Quoted 3 lines
when
using a virtualization-based system, e.g. Xen or KVM, we do not need to introduce new mechanisms to make use of SR-IOV, because we can
handle
Quoted 3 lines
VFs as full PCI devices.
A virtual PCI device in hardware (a VF) can be as powerful or complex
as
Quoted 7 lines
you like, or it can be very simple. But the big advantage of SR-IOV is that hardware presents a complete PCI device to the OS - as opposed to some resources, or queues, that need specific new configuration and assignment mechanisms in order to use them with a guest OS (like, for example, VMDq or similar technologies).
Anna
Ditto.
Taking netdev interface as an example - a queue pair is a great way to scale across cpu cores in a single OS image, but it is just not a good way to share device across multiple OS images.
The best unit of virtualization is a VF that is implemented as a complete netdev pci device (not a subset of a pci device).
This way, native netdev device drivers can work for direct hw access to
a VF "as is", and most/all Linux networking features (including VMQ) will work in a guest.
Also, guest migration for netdev interfaces (both direct and virtual) can be supported via native Linux mechanism (bonding driver), while Dom0 can retain "veto power" over any guest direct interface operation it deems privileged (vlan, mac address, promisc mode, bandwidth allocation between VFs, etc.).
Leonid
Quoted 36 lines
Subject: Re: [PATCH 0/16 v6] PCI: Linux kernel SR-IOV support
Importance: HighQuoted 1 line · Yu Zhao
While we are arguing what the software model the SR-IOV should be,
let me
Quoted 4 lines
ask two simple questions first:
1, What does the SR-IOV looks like?
2, Why do we need to support it?I don't think we need to worry about those questions, as we can see what
the SR-IOV interface looks like by looking at the PCI spec, and we know Linux needs to support it, as Linux needs to support everything :)(note, community members that can not see the PCI specs at this point in
time, please know that we are working on resolving these issues, hopefully we will have some good news within a month or so.)Quoted 2 lines
As you know the Linux kernel is the base of various virtual machine monitors such as KVM, Xen, OpenVZ and VServer. We need SR-IOV support
in
Quoted 1 line
the kernel because mostly it helps high-end users (IT departments,
HPC,
Quoted 1 line
etc.) to share limited hardware resources among hundreds or even
thousands
Quoted 1 line
virtual machines and hence reduce the cost. How can we make these
virtual
Quoted 1 line
machine monitors utilize the advantage of SR-IOV without spending too
much
Quoted 1 line
effort meanwhile remaining architectural correctness? I believe
making VF
Quoted 1 line
represent as much closer as a normal PCI device (struct
pci_dev) isthe
Quoted 8 lines
Quoted 1 line
best way in current situation, because this is not only what the
hardware
Quoted 1 line
designers expect us to do but also the usage model that KVM, Xen and
other
Quoted 1 line
VMMs have already supported.
But would such an api really take advantage of the new IOV interfaces that are exposed by the new device type?
I agree with what Yu says. The idea is to have hardware capabilities to virtualize a PCI device in a way that those virtual devices can represent full PCI devices. The advantage of that is that those virtual device can then be used like any other standard PCI device, meaning we can use existing OS tools, configuration mechanism etc. to start working with them. Also, when using a virtualization-based system, e.g. Xen or KVM, we do not need to introduce new mechanisms to make use of SR-IOV, because we can handle VFs as full PCI devices.
A virtual PCI device in hardware (a VF) can be as powerful or complex as you like, or it can be very simple. But the big advantage of SR-IOV is that hardware presents a complete PCI device to the OS - as opposed to some resources, or queues, that need specific new configuration and assignment mechanisms in order to use them with a guest OS (like, for example, VMDq or similar technologies).
Anna
Quoted 7 lines · Greg KH
Quoted 5 lines · Yu Zhao
While we are arguing what the software model the SR-IOV should be, let me ask two simple questions first:
1, What does the SR-IOV looks like?
2, Why do we need to support it?
Quoted 7 lines
I don't think we need to worry about those questions, as we can see what the SR-IOV interface looks like by looking at the PCI spec, and we know Linux needs to support it, as Linux needs to support everything :)
(note, community members that can not see the PCI specs at this point in time, please know that we are working on resolving these issues, hopefully we will have some good news within a month or so.)
Thanks for doing this!
Quoted 13 lines
Quoted 12 lines
As you know the Linux kernel is the base of various virtual machine monitors such as KVM, Xen, OpenVZ and VServer. We need SR-IOV support in the kernel because mostly it helps high-end users (IT departments, HPC, etc.) to share limited hardware resources among hundreds or even thousands virtual machines and hence reduce the cost. How can we make these virtual machine monitors utilize the advantage of SR-IOV without spending too much effort meanwhile remaining architectural correctness? I believe making VF represent as much closer as a normal PCI device (struct
pci_dev) is the best way in current situation, because this is not only what the hardware designers expect us to do but also the usage model that KVM, Xen and other VMMs have already supported.
Quoted 2 lines
But would such an api really take advantage of the new IOV interfaces that are exposed by the new device type?
The SR-IOV is a very straightforward capability -- it can only reside in the Physical Function's (the real device) config space and controls the allocation of the Virtual Function by several registers. What we can do in the PCI layer is to make the SR-IOV device spawn VF upon user request, and register VF to the PCI core. The functionality of SR-IOV device (both the PF and VF) can vary at a large range and their drivers (same as normal PCI device driver) are responsible for handling device specific stuff.
So it looks like we can get all work done in the PCI layer with only two
interfaces: one for the PF driver to register itself as a SR-IOV capable driver, expose the sysfs (or ioctl) interface to receive user request, and allocate 'pci_dev' for VF; another one to cleanup all stuff when the PF driver unregisters itself (e.g., the driver is removed or the device is going to power-saving mode.).
Quoted 9 lines
Quoted 4 lines
I agree that API in the SR-IOV pacth is arguable and the concerns such as lack of PF driver, etc. are also valid. But I personally think these stuff are not essential problems to me and other SR-IOV driver developers.
How can the lack of a PF driver not be a valid concern at this point in time? Without such a driver written, how can we know that the SR-IOV interface as created is sufficient, or that it even works properly?
Quoted 7 lines
Here's what I see we need to have before we can evaluate if the IOV core PCI patches are acceptable:
- a driver that uses this interface
- a PF driver that uses this interface.Without those, we can't determine if the infrastructure provided by the IOV core even is sufficient, right?
Yes, using a PF driver to evaluate the SR-IOV core is necessary. And only the PF driver can use the interface since the VF shouldn't have the SR-IOV capability in its config space according to the spec.
Regards,
Yu
Quoted 11 lines
Rumor has it that there is both of the above things floating around, can someone please post them to the linux-pci list so that we can see how this all works together?
thanks,
greg k-h
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 7 lines · Greg KH
Quoted 5 lines · Yu Zhao
While we are arguing what the software model the SR-IOV should be, let me ask two simple questions first:
1, What does the SR-IOV looks like?
2, Why do we need to support it?
Quoted 7 lines
I don't think we need to worry about those questions, as we can see what the SR-IOV interface looks like by looking at the PCI spec, and we know Linux needs to support it, as Linux needs to support everything :)
(note, community members that can not see the PCI specs at this point in time, please know that we are working on resolving these issues, hopefully we will have some good news within a month or so.)
Thanks for doing this!
Quoted 13 lines
Quoted 12 lines
As you know the Linux kernel is the base of various virtual machine monitors such as KVM, Xen, OpenVZ and VServer. We need SR-IOV support in the kernel because mostly it helps high-end users (IT departments, HPC, etc.) to share limited hardware resources among hundreds or even thousands virtual machines and hence reduce the cost. How can we make these virtual machine monitors utilize the advantage of SR-IOV without spending too much effort meanwhile remaining architectural correctness? I believe making VF represent as much closer as a normal PCI device (struct
pci_dev) is the best way in current situation, because this is not only what the hardware designers expect us to do but also the usage model that KVM, Xen and other VMMs have already supported.
Quoted 2 lines
But would such an api really take advantage of the new IOV interfaces that are exposed by the new device type?
The SR-IOV is a very straightforward capability -- it can only reside in the Physical Function's (the real device) config space and controls the allocation of the Virtual Function by several registers. What we can do in the PCI layer is to make the SR-IOV device spawn VF upon user request, and register VF to the PCI core. The functionality of SR-IOV device (both the PF and VF) can vary at a large range and their drivers (same as normal PCI device driver) are responsible for handling device specific stuff.
So it looks like we can get all work done in the PCI layer with only two
interfaces: one for the PF driver to register itself as a SR-IOV capable driver, expose the sysfs (or ioctl) interface to receive user request, and allocate 'pci_dev' for VF; another one to cleanup all stuff when the PF driver unregisters itself (e.g., the driver is removed or the device is going to power-saving mode.).
Quoted 9 lines
Quoted 4 lines
I agree that API in the SR-IOV pacth is arguable and the concerns such as lack of PF driver, etc. are also valid. But I personally think these stuff are not essential problems to me and other SR-IOV driver developers.
How can the lack of a PF driver not be a valid concern at this point in time? Without such a driver written, how can we know that the SR-IOV interface as created is sufficient, or that it even works properly?
Quoted 7 lines
Here's what I see we need to have before we can evaluate if the IOV core PCI patches are acceptable:
- a driver that uses this interface
- a PF driver that uses this interface.Without those, we can't determine if the infrastructure provided by the IOV core even is sufficient, right?
Yes, using a PF driver to evaluate the SR-IOV core is necessary. And only the PF driver can use the interface since the VF shouldn't have the SR-IOV capability in its config space according to the spec.
Regards,
Yu
Quoted 11 lines
Rumor has it that there is both of the above things floating around, can someone please post them to the linux-pci list so that we can see how this all works together?
thanks,
greg k-h
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 5 lines · Yu Zhao
While we are arguing what the software model the SR-IOV should be, let me ask two simple questions first:
1, What does the SR-IOV looks like?
2, Why do we need to support it?
I don't think we need to worry about those questions, as we can see what the SR-IOV interface looks like by looking at the PCI spec, and we know Linux needs to support it, as Linux needs to support everything :)
(note, community members that can not see the PCI specs at this point in time, please know that we are working on resolving these issues, hopefully we will have some good news within a month or so.)
Quoted 11 lines
As you know the Linux kernel is the base of various virtual machine monitors such as KVM, Xen, OpenVZ and VServer. We need SR-IOV support in the kernel because mostly it helps high-end users (IT departments, HPC, etc.) to share limited hardware resources among hundreds or even thousands virtual machines and hence reduce the cost. How can we make these virtual machine monitors utilize the advantage of SR-IOV without spending too much effort meanwhile remaining architectural correctness? I believe making VF represent as much closer as a normal PCI device (struct
pci_dev) is the best way in current situation, because this is not only what the hardware designers expect us to do but also the usage model that KVM, Xen and other VMMs have already supported.
But would such an api really take advantage of the new IOV interfaces that are exposed by the new device type?
Quoted 3 lines
I agree that API in the SR-IOV pacth is arguable and the concerns such as lack of PF driver, etc. are also valid. But I personally think these stuff are not essential problems to me and other SR-IOV driver developers.
How can the lack of a PF driver not be a valid concern at this point in time? Without such a driver written, how can we know that the SR-IOV interface as created is sufficient, or that it even works properly?
Here's what I see we need to have before we can evaluate if the IOV core PCI patches are acceptable:
- a driver that uses this interface
- a PF driver that uses this interface.
Without those, we can't determine if the infrastructure provided by the IOV core even is sufficient, right?
Rumor has it that there is both of the above things floating around, can someone please post them to the linux-pci list so that we can see how this all works together?
thanks,
greg k-h
Quoted 10 lines · Matthew Wilcox
Quoted 7 lines
What we would rather do in KVM, is have the VFs appear in the host as standard network devices. We would then like to back our existing PV driver to this VF directly bypassing the host networking stack. A key feature here is being able to fill the VF's receive queue with guest memory instead of host kernel memory so that you can get zero-copy receive traffic. This will perform just as well as doing passthrough (at least) and avoid all that ugliness of dealing with SR-IOV in the guest.
This argues for ignoring the SR-IOV mess completely. Just have the host driver expose multiple 'ethN' devices.
That would work, but do we want to do that for every different type of driver?
thanks,
greg k-h
Quoted 10 lines · Matthew Wilcox
Quoted 8 lines
What we would rather do in KVM, is have the VFs appear in the host as standard network devices. We would then like to back our existing PV driver to this VF directly bypassing the host networking stack. A key feature here is being able to fill the VF's receive queue with guest memory instead of host kernel memory so that you can get zero-copy receive traffic. This will perform just as well as doing passthrough (at least) and avoid all that ugliness of dealing with SR-IOV in the guest.
This argues for ignoring the SR-IOV mess completely.
It does, but VF-in-host is not the only model that we want to support. It's just the most appealing.
There will definitely be people who want to run VF-in-guest.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 5 lines · Anthony Liguori
Quoted 4 lines
It's not clear thats the right solution. If the VF devices are _only_ going to be used by the guest, then arguably, we don't want to create
pci_devsfor them in the host. (I think it _is_ the right answer, but I want to make it clear there's multiple opinions on this).
Quoted 8 lines
The VFs shouldn't be limited to being used by the guest.
SR-IOV is actually an incredibly painful thing. You need to have a VF driver in the guest, do hardware pass through, have a PV driver stub in the guest that's hypervisor specific (a VF is not usable on it's own), have a device specific backend in the VMM, and if you want to do live migration, have another PV driver in the guest that you can do teaming with. Just a mess.
Not to mention that you basically have to statically allocate them up front.
Quoted 7 lines
What we would rather do in KVM, is have the VFs appear in the host as standard network devices. We would then like to back our existing PV driver to this VF directly bypassing the host networking stack. A key feature here is being able to fill the VF's receive queue with guest memory instead of host kernel memory so that you can get zero-copy receive traffic. This will perform just as well as doing passthrough (at least) and avoid all that ugliness of dealing with SR-IOV in the guest.
This argues for ignoring the SR-IOV mess completely. Just have the host driver expose multiple 'ethN' devices.
Quoted 13 lines
This eliminates all of the mess of various drivers in the guest and all the associated baggage of doing hardware passthrough.
So IMHO, having VFs be usable in the host is absolutely critical because I think it's the only reasonable usage model.
Regards,
Anthony Liguori
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Signature · 4 lines
Matthew Wilcox Intel Open Source Technology Centre "Bill, look, we understand that you're interested in selling us this operating system, but compare it to ours. We can't possibly take such a retrograde step."
Quoted 12 lines
What we would rather do in KVM, is have the VFs appear in the host as standard network devices. We would then like to back our existing PV driver to this VF directly bypassing the host networking stack. A key feature here is being able to fill the VF's receive queue with guest memory instead of host kernel memory so that you can get zero-copy
receive traffic. This will perform just as well as doing passthrough (at least) and avoid all that ugliness of dealing with SR-IOV in the guest.
Anthony:
This is already addressed by VMDq solution(or so called netchannel2), right? Qing He is debugging the KVM side patch and pretty much close to end.
For this single purpose, we don't need SR-IOV. BTW at least Intel SR-IOV NIC also supports VMDq, so you can achieve this by simply use "native" VMDq enabled driver here, plus the work we are debugging now.
Thx, eddie
Quoted 12 lines
What we would rather do in KVM, is have the VFs appear in the host as standard network devices. We would then like to back our existing PV driver to this VF directly bypassing the host networking stack. A key feature here is being able to fill the VF's receive queue with guest memory instead of host kernel memory so that you can get zero-copy
receive traffic. This will perform just as well as doing passthrough (at least) and avoid all that ugliness of dealing with SR-IOV in the guest.
Anthony:
This is already addressed by VMDq solution(or so called netchannel2), right? Qing He is debugging the KVM side patch and pretty much close to end.
For this single purpose, we don't need SR-IOV. BTW at least Intel SR-IOV NIC also supports VMDq, so you can achieve this by simply use "native" VMDq enabled driver here, plus the work we are debugging now.
Thx, eddie
Quoted 22 lines · Anthony Liguori
Quoted 15 lines · Matthew Wilcox
[Anna, can you fix your word-wrapping please? Your lines appear to be infinitely long which is most unpleasant to reply to]
Quoted 9 lines · Anna
Quoted 3 lines
Where would the VF drivers have to be associated? On the "
pci_dev" level or on a higher one?A VF appears to the Linux OS as a standard (full, additional) PCI device. The driver is associated in the same way as for a normal PCI device. Ideally, you would use SR-IOV devices on a virtualized system, for example, using Xen. A VF can then be assigned to a guest domain as a full PCI device.
Quoted 5 lines
It's not clear thats the right solution. If the VF devices are _only_ going to be used by the guest, then arguably, we don't want to create
pci_devsfor them in the host. (I think it _is_ the right answer, but I want to make it clear there's multiple opinions on this).
Quoted 8 lines
The VFs shouldn't be limited to being used by the guest.
SR-IOV is actually an incredibly painful thing. You need to have a VF driver in the guest, do hardware pass through, have a PV driver stub in the guest that's hypervisor specific (a VF is not usable on it's own), have a device specific backend in the VMM, and if you want to do live migration, have another PV driver in the guest that you can do teaming with. Just a mess.
Actually "a PV driver stub in the guest" _was_ correct; I admit that I stated so at a virt mini summit more than a half year ago ;-). But the things have changed, and such a stub is no longer required (at least in our implementation). The major benefit of VF drivers now is that they are VMM-agnostic.
Quoted 10 lines
What we would rather do in KVM, is have the VFs appear in the host as standard network devices. We would then like to back our existing PV driver to this VF directly bypassing the host networking stack. A key feature here is being able to fill the VF's receive queue with guest memory instead of host kernel memory so that you can get zero-copy receive traffic. This will perform just as well as doing passthrough (at
least) and avoid all that ugliness of dealing with SR-IOV in the guest.
Quoted 5 lines
This eliminates all of the mess of various drivers in the guest and all the associated baggage of doing hardware passthrough.
So IMHO, having VFs be usable in the host is absolutely critical because I think it's the only reasonable usage model.
As Eddie said, VMDq is better for this model, and the feature is already available today. It is much simpler because it was designed for such purposes. It does not require hardware pass-through (e.g. VT-d) or VFs as a PCI device, either.
Quoted 8 lines
Regards,
Anthony Liguori
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected] More majordomo info at
http://vger.kernel.org/majordomo-info.html
.
Jun Nakajima | Intel Open Source Technology Center
Quoted 22 lines · Anthony Liguori
Quoted 15 lines · Matthew Wilcox
[Anna, can you fix your word-wrapping please? Your lines appear to be infinitely long which is most unpleasant to reply to]
Quoted 9 lines · Anna
Quoted 3 lines
Where would the VF drivers have to be associated? On the "
pci_dev" level or on a higher one?A VF appears to the Linux OS as a standard (full, additional) PCI device. The driver is associated in the same way as for a normal PCI device. Ideally, you would use SR-IOV devices on a virtualized system, for example, using Xen. A VF can then be assigned to a guest domain as a full PCI device.
Quoted 5 lines
It's not clear thats the right solution. If the VF devices are _only_ going to be used by the guest, then arguably, we don't want to create
pci_devsfor them in the host. (I think it _is_ the right answer, but I want to make it clear there's multiple opinions on this).
Quoted 8 lines
The VFs shouldn't be limited to being used by the guest.
SR-IOV is actually an incredibly painful thing. You need to have a VF driver in the guest, do hardware pass through, have a PV driver stub in the guest that's hypervisor specific (a VF is not usable on it's own), have a device specific backend in the VMM, and if you want to do live migration, have another PV driver in the guest that you can do teaming with. Just a mess.
Actually "a PV driver stub in the guest" _was_ correct; I admit that I stated so at a virt mini summit more than a half year ago ;-). But the things have changed, and such a stub is no longer required (at least in our implementation). The major benefit of VF drivers now is that they are VMM-agnostic.
Quoted 10 lines
What we would rather do in KVM, is have the VFs appear in the host as standard network devices. We would then like to back our existing PV driver to this VF directly bypassing the host networking stack. A key feature here is being able to fill the VF's receive queue with guest memory instead of host kernel memory so that you can get zero-copy receive traffic. This will perform just as well as doing passthrough (at
least) and avoid all that ugliness of dealing with SR-IOV in the guest.
Quoted 5 lines
This eliminates all of the mess of various drivers in the guest and all the associated baggage of doing hardware passthrough.
So IMHO, having VFs be usable in the host is absolutely critical because I think it's the only reasonable usage model.
As Eddie said, VMDq is better for this model, and the feature is already available today. It is much simpler because it was designed for such purposes. It does not require hardware pass-through (e.g. VT-d) or VFs as a PCI device, either.
Quoted 8 lines
Regards,
Anthony Liguori
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected] More majordomo info at
http://vger.kernel.org/majordomo-info.html
.
Jun Nakajima | Intel Open Source Technology Center
Quoted 9 lines · Anthony Liguori
What we would rather do in KVM, is have the VFs appear in the host as standard network devices. We would then like to back our existing PV driver to this VF directly bypassing the host networking stack. A key feature here is being able to fill the VF's receive queue with guest memory instead of host kernel memory so that you can get zero-copy receive traffic. This will perform just as well as doing passthrough (at least) and avoid all that ugliness of dealing with SR-IOV in the guest.
But you shift a lot of ugliness into the host network stack again. Not sure that is a good trade off.
Also it would always require context switches and I believe one of the reasons for the PV/VF model is very low latency IO and having heavyweight switches to the host and back would be against that.
-Andi
Signature · 1 line
[email protected]
Quoted 12 lines · Andi Kleen
Quoted 9 lines · Anthony Liguori
What we would rather do in KVM, is have the VFs appear in the host as standard network devices. We would then like to back our existing PV driver to this VF directly bypassing the host networking stack. A key feature here is being able to fill the VF's receive queue with guest memory instead of host kernel memory so that you can get zero-copy receive traffic. This will perform just as well as doing passthrough (at least) and avoid all that ugliness of dealing with SR-IOV in the guest.
Quoted 3 lines
But you shift a lot of ugliness into the host network stack again. Not sure that is a good trade off.
The net effect will be positive. We will finally have aio networking from userspace (can send process memory without resorting to sendfile()), and we'll be able to assign a queue to a process (which will enable all sorts of interesting high performance things; basically VJ channels without kernel involvement).
Quoted 4 lines
Also it would always require context switches and I believe one of the reasons for the PV/VF model is very low latency IO and having heavyweight switches to the host and back would be against that.
It's true that latency would suffer (or alternatively cpu consumption would increase).
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 12 lines · Andi Kleen
Quoted 9 lines · Anthony Liguori
What we would rather do in KVM, is have the VFs appear in the host as standard network devices. We would then like to back our existing PV driver to this VF directly bypassing the host networking stack. A key feature here is being able to fill the VF's receive queue with guest memory instead of host kernel memory so that you can get zero-copy receive traffic. This will perform just as well as doing passthrough (at least) and avoid all that ugliness of dealing with SR-IOV in the guest.
Quoted 3 lines
But you shift a lot of ugliness into the host network stack again. Not sure that is a good trade off.
The net effect will be positive. We will finally have aio networking from userspace (can send process memory without resorting to sendfile()), and we'll be able to assign a queue to a process (which will enable all sorts of interesting high performance things; basically VJ channels without kernel involvement).
Quoted 4 lines
Also it would always require context switches and I believe one of the reasons for the PV/VF model is very low latency IO and having heavyweight switches to the host and back would be against that.
It's true that latency would suffer (or alternatively cpu consumption would increase).
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 9 lines · Anthony Liguori
What we would rather do in KVM, is have the VFs appear in the host as standard network devices. We would then like to back our existing PV driver to this VF directly bypassing the host networking stack. A key feature here is being able to fill the VF's receive queue with guest memory instead of host kernel memory so that you can get zero-copy receive traffic. This will perform just as well as doing passthrough (at least) and avoid all that ugliness of dealing with SR-IOV in the guest.
But you shift a lot of ugliness into the host network stack again. Not sure that is a good trade off.
Also it would always require context switches and I believe one of the reasons for the PV/VF model is very low latency IO and having heavyweight switches to the host and back would be against that.
-Andi
Signature · 1 line
[email protected]
Quoted 10 lines · Andi Kleen
Quoted 8 lines · Anthony Liguori
What we would rather do in KVM, is have the VFs appear in the host as standard network devices. We would then like to back our existing PV driver to this VF directly bypassing the host networking stack. A key feature here is being able to fill the VF's receive queue with guest memory instead of host kernel memory so that you can get zero-copy receive traffic. This will perform just as well as doing passthrough (at least) and avoid all that ugliness of dealing with SR-IOV in the guest.
Quoted 6 lines
But you shift a lot of ugliness into the host network stack again. Not sure that is a good trade off.
Also it would always require context switches and I believe one of the reasons for the PV/VF model is very low latency IO and having heavyweight switches to the host and back would be against that.
I don't think it's established that PV/VF will have less latency than using virtio-net. virtio-net requires a world switch to send a group of packets. The cost of this (if it stays in kernel) is only a few thousand cycles on the most modern processors.
Using VT-d means that for every DMA fetch that misses in the IOTLB, you potentially have to do four memory fetches to main memory. There will be additional packet latency using VT-d compared to native, it's just not known how much at this time.
Regards,
Anthony Liguori
Quoted 2 lines
-Andi
Quoted 3 lines · Anthony Liguori
I don't think it's established that PV/VF will have less latency than using virtio-net. virtio-net requires a world switch to send a group of packets. The cost of this (if it stays in kernel) is only a few thousand cycles on the most modern processors.
Using VT-d means that for every DMA fetch that misses in the IOTLB, you potentially have to do four memory fetches to main memory. There will be additional packet latency using VT-d compared to native, it's just not known how much at this time.
If the IOTLB has intermediate TLB entries like the processor, we're talking just one or two fetches. That's a lot less than the cacheline bouncing that virtio and kvm interrupt injection incur right now.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 7 lines · Avi Kivity
Quoted 3 lines · Anthony Liguori
I don't think it's established that PV/VF will have less latency than using virtio-net. virtio-net requires a world switch to send a group of packets. The cost of this (if it stays in kernel) is only a few thousand cycles on the most modern processors.
Using VT-d means that for every DMA fetch that misses in the IOTLB, you potentially have to do four memory fetches to main memory. There will be additional packet latency using VT-d compared to native, it's just not known how much at this time.
If the IOTLB has intermediate TLB entries like the processor, we're talking just one or two fetches. That's a lot less than the cacheline bouncing that virtio and kvm interrupt injection incur right now.
The PCI SIG Address Translation Service (ATS) specifies a way that uses an Address Translation Cache (ATC) in the Endpoint to reduce the latency.
The Linux kernel support for ATS capability will come soon.
Thanks,
Yu
Quoted 6 lines · Avi Kivity
Quoted 5 lines · Anthony Liguori
I don't think it's established that PV/VF will have less latency than using virtio-net. virtio-net requires a world switch to send a group of packets. The cost of this (if it stays in kernel) is only a few thousand cycles on the most modern processors.
Quoted 9 lines
Quoted 4 lines
Using VT-d means that for every DMA fetch that misses in the IOTLB, you potentially have to do four memory fetches to main memory. There will be additional packet latency using VT-d compared to native, it's just not known how much at this time.
If the IOTLB has intermediate TLB entries like the processor, we're talking just one or two fetches. That's a lot less than the cacheline bouncing that virtio and kvm interrupt injection incur right now.
The PCI SIG Address Translation Service (ATS) specifies a way that uses an Address Translation Cache (ATC) in the Endpoint to reduce the latency.
The Linux kernel support for ATS capability will come soon.
Thanks,
Yu
Quoted 9 lines · Anthony Liguori
I don't think it's established that PV/VF will have less latency than using virtio-net. virtio-net requires a world switch to send a group of packets. The cost of this (if it stays in kernel) is only a few thousand cycles on the most modern processors.
Using VT-d means that for every DMA fetch that misses in the IOTLB, you potentially have to do four memory fetches to main memory. There will be additional packet latency using VT-d compared to native, it's just not known how much at this time.
If the IOTLB has intermediate TLB entries like the processor, we're talking just one or two fetches. That's a lot less than the cacheline bouncing that virtio and kvm interrupt injection incur right now.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 10 lines · Andi Kleen
Quoted 8 lines · Anthony Liguori
What we would rather do in KVM, is have the VFs appear in the host as standard network devices. We would then like to back our existing PV driver to this VF directly bypassing the host networking stack. A key feature here is being able to fill the VF's receive queue with guest memory instead of host kernel memory so that you can get zero-copy receive traffic. This will perform just as well as doing passthrough (at least) and avoid all that ugliness of dealing with SR-IOV in the guest.
Quoted 6 lines
But you shift a lot of ugliness into the host network stack again. Not sure that is a good trade off.
Also it would always require context switches and I believe one of the reasons for the PV/VF model is very low latency IO and having heavyweight switches to the host and back would be against that.
I don't think it's established that PV/VF will have less latency than using virtio-net. virtio-net requires a world switch to send a group of packets. The cost of this (if it stays in kernel) is only a few thousand cycles on the most modern processors.
Using VT-d means that for every DMA fetch that misses in the IOTLB, you potentially have to do four memory fetches to main memory. There will be additional packet latency using VT-d compared to native, it's just not known how much at this time.
Regards,
Anthony Liguori
Quoted 2 lines
-Andi
Quoted 16 lines · Anthony Liguori
Quoted 15 lines · Matthew Wilcox
[Anna, can you fix your word-wrapping please? Your lines appear to be infinitely long which is most unpleasant to reply to]
Quoted 9 lines · Anna
Quoted 3 lines
Where would the VF drivers have to be associated? On the "
pci_dev" level or on a higher one?A VF appears to the Linux OS as a standard (full, additional) PCI device. The driver is associated in the same way as for a normal PCI device. Ideally, you would use SR-IOV devices on a virtualized system, for example, using Xen. A VF can then be assigned to a guest domain as a full PCI device.
Quoted 7 lines
Quoted 5 lines
It's not clear thats the right solution. If the VF devices are _only_ going to be used by the guest, then arguably, we don't want to create
pci_devsfor them in the host. (I think it _is_ the right answer, but I want to make it clear there's multiple opinions on this).The VFs shouldn't be limited to being used by the guest.
Yes, VF driver running in the host is supported :-)
Quoted 2 lines
SR-IOV is actually an incredibly painful thing. You need to have a VF driver in the guest, do hardware pass through, have a PV driver stub in the guest that's hypervisor specific (a VF is not usable on it's own), have a device specific backend in the VMM, and if you want to do live migration, have another PV driver in the guest that you can do teaming with. Just a mess.
Actually not so mess. VF driver can be a plain PCI device driver and doesn't require any backend in the VMM, or hypervisor specific knowledge, if the hardware is properly designed. In this case PF driver controls hardware resource allocation for VFs and VF driver can work without any communication to PF driver or VMM.
Quoted 2 lines
What we would rather do in KVM, is have the VFs appear in the host as standard network devices. We would then like to back our existing PV driver to this VF directly bypassing the host networking stack. A key feature here is being able to fill the VF's receive queue with guest memory instead of host kernel memory so that you can get zero-copy receive traffic. This will perform just as well as doing passthrough (at least) and avoid all that ugliness of dealing with SR-IOV in the guest.
If the hardware supports both SR-IOV and IOMMU, I wouldn't suggest people to do so, because they will get better performance by directly assigning VF to the guest.
However, lots of low-end machines don't have SR-IOV and IOMMU support. They may have multi queue NIC, which uses built-in L2 switch to dispense packets to different DMA queue according to MAC address. They definitely can benefit a lot if there is software support for the DMA queue hooking virtio-net backend as you suggested.
Quoted 4 lines
This eliminates all of the mess of various drivers in the guest and all the associated baggage of doing hardware passthrough.
So IMHO, having VFs be usable in the host is absolutely critical because I think it's the only reasonable usage model.
Please don't worry, we have take this usage model as well as container model into account when designing SR-IOV framework for the kernel.
Quoted 8 lines
Regards,
Anthony Liguori
Signature · 3 lines
To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 16 lines · Anthony Liguori
Quoted 15 lines · Matthew Wilcox
[Anna, can you fix your word-wrapping please? Your lines appear to be infinitely long which is most unpleasant to reply to]
Quoted 9 lines · Anna
Quoted 3 lines
Where would the VF drivers have to be associated? On the "
pci_dev" level or on a higher one?A VF appears to the Linux OS as a standard (full, additional) PCI device. The driver is associated in the same way as for a normal PCI device. Ideally, you would use SR-IOV devices on a virtualized system, for example, using Xen. A VF can then be assigned to a guest domain as a full PCI device.
Quoted 7 lines
Quoted 5 lines
It's not clear thats the right solution. If the VF devices are _only_ going to be used by the guest, then arguably, we don't want to create
pci_devsfor them in the host. (I think it _is_ the right answer, but I want to make it clear there's multiple opinions on this).The VFs shouldn't be limited to being used by the guest.
Yes, VF driver running in the host is supported :-)
Quoted 7 lines
SR-IOV is actually an incredibly painful thing. You need to have a VF driver in the guest, do hardware pass through, have a PV driver stub in the guest that's hypervisor specific (a VF is not usable on it's own), have a device specific backend in the VMM, and if you want to do live migration, have another PV driver in the guest that you can do teaming with. Just a mess.
Actually not so mess. VF driver can be a plain PCI device driver and doesn't require any backend in the VMM, or hypervisor specific knowledge, if the hardware is properly designed. In this case PF driver controls hardware resource allocation for VFs and VF driver can work without any communication to PF driver or VMM.
Quoted 8 lines
What we would rather do in KVM, is have the VFs appear in the host as standard network devices. We would then like to back our existing PV driver to this VF directly bypassing the host networking stack. A key feature here is being able to fill the VF's receive queue with guest memory instead of host kernel memory so that you can get zero-copy receive traffic. This will perform just as well as doing passthrough (at least) and avoid all that ugliness of dealing with SR-IOV in the guest.
If the hardware supports both SR-IOV and IOMMU, I wouldn't suggest people to do so, because they will get better performance by directly assigning VF to the guest.
However, lots of low-end machines don't have SR-IOV and IOMMU support. They may have multi queue NIC, which uses built-in L2 switch to dispense packets to different DMA queue according to MAC address. They definitely can benefit a lot if there is software support for the DMA queue hooking virtio-net backend as you suggested.
Quoted 6 lines
This eliminates all of the mess of various drivers in the guest and all the associated baggage of doing hardware passthrough.
So IMHO, having VFs be usable in the host is absolutely critical because I think it's the only reasonable usage model.
Please don't worry, we have take this usage model as well as container model into account when designing SR-IOV framework for the kernel.
Quoted 8 lines
Regards,
Anthony Liguori
Signature · 3 lines
To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
RSS limitations
In reply to Anthony LiguoriHi Eddie,
I want to start using the new Intel 10Gige for several ISP services, looking through the documentation on the 10Gige cards, i can see that receive-side scaling (RSS) is supported, but RSS does not support searching for IP/PORT tupples inside PPPoE packets or MPLS packets.
This creates the issue, that i cannot load balance irq's across different CPU cores for PPPoE and MPLS framed Ethernet packets.
Is there a possibility Intel could extend the RSS system to support searching and hashing on the IP/PORT tupples inside PPPoE and MPLS? I assume it would be a fairly simple experimental firmware change.
Failing this, do you see that its possible for VMDq to be used to create virtual Ethernet cards each with its own MAC address visible by the switch connected to the physical Ethernet card?
Better still, could Intel have a "Linux Virtual Server" (LVS) protocol, that lets the host decide how to balance (depending on CPU load feedback for example).
I feel, the limited nature of RSS will limit the usefulness of 10Gige, Unless RSS is improved, or perhaps I've simply overlooked something.
Cheers,
Chris Gates
TPG Internet
Quoted 15 lines · Matthew Wilcox
[Anna, can you fix your word-wrapping please? Your lines appear to be infinitely long which is most unpleasant to reply to]
Quoted 9 lines · Anna
Quoted 3 lines
Where would the VF drivers have to be associated? On the "
pci_dev" level or on a higher one?A VF appears to the Linux OS as a standard (full, additional) PCI device. The driver is associated in the same way as for a normal PCI device. Ideally, you would use SR-IOV devices on a virtualized system, for example, using Xen. A VF can then be assigned to a guest domain as a full PCI device.
Quoted 5 lines
It's not clear thats the right solution. If the VF devices are _only_ going to be used by the guest, then arguably, we don't want to create
pci_devsfor them in the host. (I think it _is_ the right answer, but I want to make it clear there's multiple opinions on this).
The VFs shouldn't be limited to being used by the guest.
SR-IOV is actually an incredibly painful thing. You need to have a VF driver in the guest, do hardware pass through, have a PV driver stub in the guest that's hypervisor specific (a VF is not usable on it's own), have a device specific backend in the VMM, and if you want to do live migration, have another PV driver in the guest that you can do teaming with. Just a mess.
What we would rather do in KVM, is have the VFs appear in the host as standard network devices. We would then like to back our existing PV driver to this VF directly bypassing the host networking stack. A key feature here is being able to fill the VF's receive queue with guest memory instead of host kernel memory so that you can get zero-copy receive traffic. This will perform just as well as doing passthrough (at least) and avoid all that ugliness of dealing with SR-IOV in the guest.
This eliminates all of the mess of various drivers in the guest and all the associated baggage of doing hardware passthrough.
So IMHO, having VFs be usable in the host is absolutely critical because I think it's the only reasonable usage model.
Regards,
Anthony Liguori
[Anna, can you fix your word-wrapping please? Your lines appear to be infinitely long which is most unpleasant to reply to]
Quoted 8 lines · Anna
Quoted 2 lines
Where would the VF drivers have to be associated? On the "
pci_dev" level or on a higher one?A VF appears to the Linux OS as a standard (full, additional) PCI device. The driver is associated in the same way as for a normal PCI device. Ideally, you would use SR-IOV devices on a virtualized system, for example, using Xen. A VF can then be assigned to a guest domain as a full PCI device.
It's not clear thats the right solution. If the VF devices are _only_ going to be used by the guest, then arguably, we don't want to create pci_devs for them in the host. (I think it _is_ the right answer, but I want to make it clear there's multiple opinions on this).
Quoted 9 lines
Quoted 2 lines
Will all drivers that want to bind to a "VF" device need to be rewritten?
Currently, any vendor providing a SR-IOV device needs to provide a PF driver and a VF driver that runs on their hardware. A VF driver does not necessarily need to know much about SR-IOV but just run on the presented PCI device. You might want to have a communication channel between PF and VF driver though, for various reasons, if such a channel is not provided in hardware.
That is one model. Another model is to provide one driver that can handle both PF and VF devices. A third model is to provide, say, a Windows VF driver and a Xen PF driver and only support Windows-on-Xen. (This last would probably be an exercise in foot-shooting, but nevertheless, I've heard it mooted).
Quoted 4 lines
Quoted 3 lines
Yeah, that's what I'm worried/curious about. Without seeing the code for such a driver, how can we properly evaluate if this infrastructure is the correct one and the proper way to do all of this?
Quoted 9 lines
Yu's API allows a PF driver to register with the Linux PCI code and use it to activate VFs and allocate their resources. The PF driver needs to be modified to work with that API. While you can argue about how that API is supposed to look like, it is clear that such an API is required in some form. The PF driver needs to know when VFs are active as it might want to allocate further (device-specific) resources to VFs or initiate further (device-specific) configurations. While probably a lot of SR-IOV specific code has to be in the PF driver, there is also support required from the Linux PCI subsystem, which is to some extend provided by Yu's patches.
Everyone agrees that some support is necessary. The question is exactly what it looks like. I must confess to not having reviewed this latest patch series yet -- I'm a little burned out on patch review.
Signature · 4 lines
Matthew Wilcox Intel Open Source Technology Centre "Bill, look, we understand that you're interested in selling us this operating system, but compare it to ours. We can't possibly take such a retrograde step."
Quoted 14 lines · Greg KH
Quoted 5 lines · H L
I have not modified any existing drivers, but instead I threw together a bare-bones module enabling me to make a call to
pci_iov_register()and then poke at an SR-IOV adapter's /sys entries for which no driver was loaded.Quoted 7 lines
It appears from my perusal thus far that drivers using these new SR-IOV patches will require modification; i.e. the driver associated with the Physical Function (PF) will be required to make the
pci_iov_register()call along with the requisitenotify()function. Essentially this suggests to me a model for the PF driver to perform any "global actions" or setup on behalf of VFs before enabling them after which VF drivers could be associated.
Quoted 5 lines
Where would the VF drivers have to be associated? On the "
pci_dev" level or on a higher one?Will all drivers that want to bind to a "VF" device need to be rewritten?
The current model being implemented by my colleagues has separate drivers for the PF (aka native) and VF devices. I don't personally believe this is the correct path, but I'm reserving judgement until I see some code.
I don't think we really know what the One True Usage model is for VF devices. Chris Wright has some ideas, I have some ideas and Yu Zhao has some ideas. I bet there's other people who have other ideas too.
Signature · 4 lines
Matthew Wilcox Intel Open Source Technology Centre "Bill, look, we understand that you're interested in selling us this operating system, but compare it to ours. We can't possibly take such a retrograde step."
Quoted 18 lines · Matthew Wilcox
Quoted 17 lines · Greg KH
Quoted 5 lines · H L
I have not modified any existing drivers, but instead I threw together a bare-bones module enabling me to make a call to
pci_iov_register()and then poke at an SR-IOV adapter's /sys entries for which no driver was loaded.Quoted 7 lines
It appears from my perusal thus far that drivers using these new SR-IOV patches will require modification; i.e. the driver associated with the Physical Function (PF) will be required to make the
pci_iov_register()call along with the requisitenotify()function. Essentially this suggests to me a model for the PF driver to perform any "global actions" or setup on behalf of VFs before enabling them after which VF drivers could be associated.Where would the VF drivers have to be associated? On the "
pci_dev" level or on a higher one?
Quoted 7 lines
Quoted 2 lines
Will all drivers that want to bind to a "VF" device need to be rewritten?
The current model being implemented by my colleagues has separate drivers for the PF (aka native) and VF devices. I don't personally believe this is the correct path, but I'm reserving judgement until I see some code.
Hm, I would like to see that code before we can properly evaluate this interface. Especially as they are all tightly tied together.
Quoted 3 lines
I don't think we really know what the One True Usage model is for VF devices. Chris Wright has some ideas, I have some ideas and Yu Zhao has some ideas. I bet there's other people who have other ideas too.
I'd love to hear those ideas.
Rumor has it, there is some Xen code floating around to support this already, is that true?
thanks,
greg k-h
Quoted 30 lines · Greg KH
Quoted 18 lines · Matthew Wilcox
Quoted 17 lines · Greg KH
Quoted 5 lines · H L
I have not modified any existing drivers, but instead I threw together a bare-bones module enabling me to make a call to
pci_iov_register()and then poke at an SR-IOV adapter's /sys entries for which no driver was loaded.Quoted 7 lines
It appears from my perusal thus far that drivers using these new SR-IOV patches will require modification; i.e. the driver associated with the Physical Function (PF) will be required to make the
pci_iov_register()call along with the requisitenotify()function. Essentially this suggests to me a model for the PF driver to perform any "global actions" or setup on behalf of VFs before enabling them after which VF drivers could be associated.Where would the VF drivers have to be associated? On the "
pci_dev" level or on a higher one?Quoted 7 lines
Quoted 2 lines
Will all drivers that want to bind to a "VF" device need to be rewritten?
The current model being implemented by my colleagues has separate drivers for the PF (aka native) and VF devices. I don't personally believe this is the correct path, but I'm reserving judgement until I see some code.
Hm, I would like to see that code before we can properly evaluate this interface. Especially as they are all tightly tied together.
Quoted 8 lines
Quoted 3 lines
I don't think we really know what the One True Usage model is for VF devices. Chris Wright has some ideas, I have some ideas and Yu Zhao has some ideas. I bet there's other people who have other ideas too.
I'd love to hear those ideas.
Rumor has it, there is some Xen code floating around to support this already, is that true?
Xen patches were posted to xen-devel by Yu Zhao on the 29th of September [1]. Unfortunately the only responses that I can find are a) that the patches were mangled and b) they seem to include changes (by others) that have been merged into Linux. I have confirmed that both of these concerns are valid.
I have not yet examined the difference, if any, in the approach taken by Yu to SR-IOV in Linux and Xen. Unfortunately comparison is less than trivial due to the gaping gap in kernel versions between Linux-Xen (2.6.18.8) and Linux itself.
One approach that I was considering in order to familiarise myself with the code was to backport the v6 Linux patches (this thread) to Linux-Xen. I made a start on that, but again due to kernel version differences it is non-trivial.
[1] http://lists.xensource.com/archives/html/xen-devel/2008-09/msg00923.html
Signature · 3 lines
Simon Horman VA Linux Systems Japan K.K., Sydney, Australia Satellite Office H: www.vergenet.net/~horms/ W: www.valinux.co.jp/en
Quoted 30 lines · Greg KH
Quoted 18 lines · Matthew Wilcox
Quoted 17 lines · Greg KH
Quoted 5 lines · H L
I have not modified any existing drivers, but instead I threw together a bare-bones module enabling me to make a call to
pci_iov_register()and then poke at an SR-IOV adapter's /sys entries for which no driver was loaded.Quoted 7 lines
It appears from my perusal thus far that drivers using these new SR-IOV patches will require modification; i.e. the driver associated with the Physical Function (PF) will be required to make the
pci_iov_register()call along with the requisitenotify()function. Essentially this suggests to me a model for the PF driver to perform any "global actions" or setup on behalf of VFs before enabling them after which VF drivers could be associated.Where would the VF drivers have to be associated? On the "
pci_dev" level or on a higher one?Quoted 7 lines
Quoted 2 lines
Will all drivers that want to bind to a "VF" device need to be rewritten?
The current model being implemented by my colleagues has separate drivers for the PF (aka native) and VF devices. I don't personally believe this is the correct path, but I'm reserving judgement until I see some code.
Hm, I would like to see that code before we can properly evaluate this interface. Especially as they are all tightly tied together.
Quoted 8 lines
Quoted 3 lines
I don't think we really know what the One True Usage model is for VF devices. Chris Wright has some ideas, I have some ideas and Yu Zhao has some ideas. I bet there's other people who have other ideas too.
I'd love to hear those ideas.
Rumor has it, there is some Xen code floating around to support this already, is that true?
Xen patches were posted to xen-devel by Yu Zhao on the 29th of September [1]. Unfortunately the only responses that I can find are a) that the patches were mangled and b) they seem to include changes (by others) that have been merged into Linux. I have confirmed that both of these concerns are valid.
I have not yet examined the difference, if any, in the approach taken by Yu to SR-IOV in Linux and Xen. Unfortunately comparison is less than trivial due to the gaping gap in kernel versions between Linux-Xen (2.6.18.8) and Linux itself.
One approach that I was considering in order to familiarise myself with the code was to backport the v6 Linux patches (this thread) to Linux-Xen. I made a start on that, but again due to kernel version differences it is non-trivial.
[1] http://lists.xensource.com/archives/html/xen-devel/2008-09/msg00923.html
Signature · 3 lines
Simon Horman VA Linux Systems Japan K.K., Sydney, Australia Satellite Office H: www.vergenet.net/~horms/ W: www.valinux.co.jp/en
Quoted 9 lines · Greg KH
Quoted 4 lines · Matthew Wilcox
I don't think we really know what the One True Usage model is for VF devices. Chris Wright has some ideas, I have some ideas and Yu Zhao has some ideas. I bet there's other people who have other ideas too.
I'd love to hear those ideas.
We've been talking about avoiding hardware passthrough entirely and just backing a virtio-net backend driver by a dedicated VF in the host. That avoids a huge amount of guest-facing complexity, let's migration Just Work, and should give the same level of performance.
Regards,
Anthony Liguori
Quoted 11 lines
Rumor has it, there is some Xen code floating around to support this already, is that true?
thanks,
greg k-h
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 8 lines · Anthony Liguori
Quoted 7 lines · Greg KH
Quoted 4 lines · Matthew Wilcox
I don't think we really know what the One True Usage model is for VF devices. Chris Wright has some ideas, I have some ideas and Yu Zhao has some ideas. I bet there's other people who have other ideas too.
Quoted 7 lines
Quoted 2 lines
I'd love to hear those ideas.
We've been talking about avoiding hardware passthrough entirely and just backing a virtio-net backend driver by a dedicated VF in the host. That avoids a huge amount of guest-facing complexity, let's migration Just Work, and should give the same level of performance.
Does that involve this patch set? Or a different type of interface.
thanks,
greg k-h
Quoted 8 lines · Anthony Liguori
Quoted 7 lines · Greg KH
Quoted 4 lines · Matthew Wilcox
I don't think we really know what the One True Usage model is for VF devices. Chris Wright has some ideas, I have some ideas and Yu Zhao has some ideas. I bet there's other people who have other ideas too.
Quoted 7 lines
Quoted 2 lines
I'd love to hear those ideas.
We've been talking about avoiding hardware passthrough entirely and just backing a virtio-net backend driver by a dedicated VF in the host. That avoids a huge amount of guest-facing complexity, let's migration Just Work, and should give the same level of performance.
Does that involve this patch set? Or a different type of interface.
thanks,
greg k-h
Quoted 8 lines · Greg KH
Quoted 7 lines · Anthony Liguori
Quoted 6 lines · Greg KH
Quoted 4 lines · Matthew Wilcox
I don't think we really know what the One True Usage model is for VF devices. Chris Wright has some ideas, I have some ideas and Yu Zhao has some ideas. I bet there's other people who have other ideas too.
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
I'd love to hear those ideas.
We've been talking about avoiding hardware passthrough entirely and just backing a virtio-net backend driver by a dedicated VF in the host. That avoids a huge amount of guest-facing complexity, let's migration Just Work, and should give the same level of performance.
This can be commonly used not only with VF -- devices that have multiple DMA queues (e.g., Intel VMDq, Neterion Xframe) and even traditional devices can also take the advantage of this.
CC Rusty Russel in case he has more comments.
Quoted 2 lines
Does that involve this patch set? Or a different type of interface.
I think that is a different type of interface. We need to hook the DMA interface in the device driver to virtio-net backend so the hardware (normal device, VF, VMDq, etc.) can DMA data to/from the virtio-net backend.
Regards,
Yu
Quoted 8 lines · Greg KH
Quoted 7 lines · Anthony Liguori
Quoted 6 lines · Greg KH
Quoted 4 lines · Matthew Wilcox
I don't think we really know what the One True Usage model is for VF devices. Chris Wright has some ideas, I have some ideas and Yu Zhao has some ideas. I bet there's other people who have other ideas too.
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
I'd love to hear those ideas.
We've been talking about avoiding hardware passthrough entirely and just backing a virtio-net backend driver by a dedicated VF in the host. That avoids a huge amount of guest-facing complexity, let's migration Just Work, and should give the same level of performance.
This can be commonly used not only with VF -- devices that have multiple DMA queues (e.g., Intel VMDq, Neterion Xframe) and even traditional devices can also take the advantage of this.
CC Rusty Russel in case he has more comments.
Quoted 2 lines
Does that involve this patch set? Or a different type of interface.
I think that is a different type of interface. We need to hook the DMA interface in the device driver to virtio-net backend so the hardware (normal device, VF, VMDq, etc.) can DMA data to/from the virtio-net backend.
Regards,
Yu
Quoted 15 lines · Yu
Quoted 7 lines · Greg KH
Quoted 6 lines · Anthony Liguori
Quoted 5 lines · Greg KH
Quoted 3 lines · Matthew Wilcox
I don't think we really know what the One True Usage model is for VF devices. Chris Wright has some ideas, I have some ideas and Yu Zhao has some ideas. I bet there's other people who have other ideas too.
Quoted 6 lines
Quoted 6 lines
Quoted 1 line
I'd love to hear those ideas.
We've been talking about avoiding hardware passthrough entirely and just backing a virtio-net backend driver by a dedicated VF in the host. That avoids a huge amount of guest-facing complexity, let's migration Just Work, and should give the same level of performance.
Quoted 5 lines
This can be commonly used not only with VF -- devices that have multiple DMA queues (e.g., Intel VMDq, Neterion Xframe) and even traditional devices can also take the advantage of this.
CC Rusty Russel in case he has more comments.
Yes, even dumb devices could use this mechanism if you wanted to bind an entire device solely to one guest.
We don't have network infrastructure for this today, but my thought was to do something in dev_alloc_skb and dev_kfree_skb et al.
Cheers,
Rusty.
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linux-foundation.org/mailman/listinfo/virtualization
Quoted 20 lines · Rusty Russell
Quoted 19 lines · Yu
Quoted 7 lines · Greg KH
Quoted 6 lines · Anthony Liguori
Quoted 5 lines · Greg KH
Quoted 3 lines · Matthew Wilcox
I don't think we really know what the One True Usage model is for VF devices. Chris Wright has some ideas, I have some ideas and Yu Zhao has some ideas. I bet there's other people who have other ideas too.
Quoted 6 lines
Quoted 6 lines
Quoted 1 line
I'd love to hear those ideas.
We've been talking about avoiding hardware passthrough entirely and just backing a virtio-net backend driver by a dedicated VF in the host. That avoids a huge amount of guest-facing complexity, let's migration Just Work, and should give the same level of performance.
This can be commonly used not only with VF -- devices that have multiple DMA queues (e.g., Intel VMDq, Neterion Xframe) and even traditional devices can also take the advantage of this.
Quoted 7 lines
Quoted 1 line
CC Rusty Russel in case he has more comments.
Yes, even dumb devices could use this mechanism if you wanted to bind an entire device solely to one guest.
We don't have network infrastructure for this today, but my thought was to do something in
dev_alloc_skbanddev_kfree_skbet al.
Is there any discussion about this on the netdev? Any prototype available? If not, I'd like to create one and evaluate the performance of virtio-net solution again the hardware passthrough.
Thanks,
Yu
Quoted 20 lines · Rusty Russell
Quoted 19 lines · Yu
Quoted 7 lines · Greg KH
Quoted 6 lines · Anthony Liguori
Quoted 5 lines · Greg KH
Quoted 3 lines · Matthew Wilcox
I don't think we really know what the One True Usage model is for VF devices. Chris Wright has some ideas, I have some ideas and Yu Zhao has some ideas. I bet there's other people who have other ideas too.
Quoted 6 lines
Quoted 6 lines
Quoted 1 line
I'd love to hear those ideas.
We've been talking about avoiding hardware passthrough entirely and just backing a virtio-net backend driver by a dedicated VF in the host. That avoids a huge amount of guest-facing complexity, let's migration Just Work, and should give the same level of performance.
This can be commonly used not only with VF -- devices that have multiple DMA queues (e.g., Intel VMDq, Neterion Xframe) and even traditional devices can also take the advantage of this.
Quoted 5 lines
Quoted 1 line
CC Rusty Russel in case he has more comments.
Yes, even dumb devices could use this mechanism if you wanted to bind an entire device solely to one guest.
We don't have network infrastructure for this today, but my thought was to do something in
dev_alloc_skbanddev_kfree_skbet al.
Is there any discussion about this on the netdev? Any prototype available? If not, I'd like to create one and evaluate the performance of virtio-net solution again the hardware passthrough.
Thanks,
Yu
Quoted 8 lines · Greg KH
Quoted 5 lines
We've been talking about avoiding hardware passthrough entirely and just backing a virtio-net backend driver by a dedicated VF in the host. That avoids a huge amount of guest-facing complexity, let's migration Just Work, and should give the same level of performance.
Does that involve this patch set? Or a different type of interface.
So long as the VF is exposed as a standalone PCI device, it's the same interface. In fact you can take a random PCI card and expose it to a guest this way; it doesn't have to be SR-IOV. Of course, with a standard PCI card you won't get much sharing (a quad port NIC will be good for four guests).
We'll need other changes in the network stack, but these are orthogonal to SR-IOV.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 8 lines · Greg KH
Quoted 5 lines
We've been talking about avoiding hardware passthrough entirely and just backing a virtio-net backend driver by a dedicated VF in the host. That avoids a huge amount of guest-facing complexity, let's migration Just Work, and should give the same level of performance.
Does that involve this patch set? Or a different type of interface.
So long as the VF is exposed as a standalone PCI device, it's the same interface. In fact you can take a random PCI card and expose it to a guest this way; it doesn't have to be SR-IOV. Of course, with a standard PCI card you won't get much sharing (a quad port NIC will be good for four guests).
We'll need other changes in the network stack, but these are orthogonal to SR-IOV.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 4 lines · Anthony Liguori
We've been talking about avoiding hardware passthrough entirely and just backing a virtio-net backend driver by a dedicated VF in the host. That avoids a huge amount of guest-facing complexity, let's migration Just Work, and should give the same level of performance.
I don't believe that it will, and every benchmark I've seen or have done so far shows a significant performance gap between virtio and direct assignment, even on 1G ethernet. I am willing however to reserve judgement until someone implements your suggestion and actually measures it, preferably on 10G ethernet.
No doubt device assignment---and SR-IOV in particular---are complex, but I hardly think ignoring it as you seem to propose is the right approach.
Cheers,
Muli
Signature · 5 lines
The First Workshop on I/O Virtualization (WIOV '08)
Dec 2008, San Diego, CA, http://www.usenix.org/wiov08/
<->
SYSTOR 2009---The Israeli Experimental Systems Conference
http://www.haifa.il.ibm.com/conferences/systor2009/Quoted 6 lines · Muli Ben-Yehuda
Quoted 5 lines
We've been talking about avoiding hardware passthrough entirely and just backing a virtio-net backend driver by a dedicated VF in the host. That avoids a huge amount of guest-facing complexity, let's migration Just Work, and should give the same level of performance.
Quoted 6 lines
I don't believe that it will, and every benchmark I've seen or have done so far shows a significant performance gap between virtio and direct assignment, even on 1G ethernet. I am willing however to reserve judgement until someone implements your suggestion and actually measures it, preferably on 10G ethernet.
Right now virtio copies data, and has other inefficiencies. With a dedicated VF, we can eliminate the copies.
CPU utilization and latency will be worse. If we can limit the slowdowns to an acceptable amount, the simplicity and other advantages of VF-in-host may outweigh the performance degradation.
Quoted 3 lines
No doubt device assignment---and SR-IOV in particular---are complex, but I hardly think ignoring it as you seem to propose is the right approach.
I agree. We should hedge our bets and support both models.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 6 lines · Muli Ben-Yehuda
Quoted 5 lines
We've been talking about avoiding hardware passthrough entirely and just backing a virtio-net backend driver by a dedicated VF in the host. That avoids a huge amount of guest-facing complexity, let's migration Just Work, and should give the same level of performance.
Quoted 6 lines
I don't believe that it will, and every benchmark I've seen or have done so far shows a significant performance gap between virtio and direct assignment, even on 1G ethernet. I am willing however to reserve judgement until someone implements your suggestion and actually measures it, preferably on 10G ethernet.
Right now virtio copies data, and has other inefficiencies. With a dedicated VF, we can eliminate the copies.
CPU utilization and latency will be worse. If we can limit the slowdowns to an acceptable amount, the simplicity and other advantages of VF-in-host may outweigh the performance degradation.
Quoted 3 lines
No doubt device assignment---and SR-IOV in particular---are complex, but I hardly think ignoring it as you seem to propose is the right approach.
I agree. We should hedge our bets and support both models.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 4 lines · Anthony Liguori
We've been talking about avoiding hardware passthrough entirely and just backing a virtio-net backend driver by a dedicated VF in the host. That avoids a huge amount of guest-facing complexity, let's migration Just Work, and should give the same level of performance.
I don't believe that it will, and every benchmark I've seen or have done so far shows a significant performance gap between virtio and direct assignment, even on 1G ethernet. I am willing however to reserve judgement until someone implements your suggestion and actually measures it, preferably on 10G ethernet.
No doubt device assignment---and SR-IOV in particular---are complex, but I hardly think ignoring it as you seem to propose is the right approach.
Cheers,
Muli
Signature · 5 lines
The First Workshop on I/O Virtualization (WIOV '08)
Dec 2008, San Diego, CA, http://www.usenix.org/wiov08/
<->
SYSTOR 2009---The Israeli Experimental Systems Conference
http://www.haifa.il.ibm.com/conferences/systor2009/Quoted 9 lines · Greg KH
Quoted 4 lines · Matthew Wilcox
I don't think we really know what the One True Usage model is for VF devices. Chris Wright has some ideas, I have some ideas and Yu Zhao has some ideas. I bet there's other people who have other ideas too.
I'd love to hear those ideas.
We've been talking about avoiding hardware passthrough entirely and just backing a virtio-net backend driver by a dedicated VF in the host. That avoids a huge amount of guest-facing complexity, let's migration Just Work, and should give the same level of performance.
Regards,
Anthony Liguori
Quoted 11 lines
Rumor has it, there is some Xen code floating around to support this already, is that true?
thanks,
greg k-h
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html