PCI: Linux kernel SR-IOV support
Subsystems (5)
v6 cover PCI: Linux kernel SR-IOV support
Greetings,
Following patches are intended to support SR-IOV capability in the Linux kernel. With these patches, people can turn a PCI device with the capability into multiple ones from software perspective, which will benefit KVM and achieve other purposes such as QoS, security, and etc.
Changes from v5 to v6:
1, update ABI document to include SR-IOV sysfs entries (Greg KH) 2, fix two coding style problems (Ingo Molnar)
---
[PATCH 1/16 v6] PCI: remove unnecessary arg of pci_update_resource() [PATCH 2/16 v6] PCI: define PCI resource names in an 'enum' [PATCH 3/16 v6] PCI: export __pci_read_base
[PATCH 4/16 v6] PCI: make pci_alloc_child_bus() be able to handle NULL bridge [PATCH 5/16 v6] PCI: add a wrapper for resource_alignment() [PATCH 6/16 v6] PCI: add a new function to map BAR offset
[PATCH 7/16 v6] PCI: cleanup pcibios_allocate_resources()
[PATCH 8/16 v6] PCI: add boot options to reassign resources [PATCH 9/16 v6] PCI: add boot option to align MMIO resources [PATCH 10/16 v6] PCI: cleanup pci_bus_add_devices()
[PATCH 11/16 v6] PCI: split a new function from pci_bus_add_devices() [PATCH 12/16 v6] PCI: support the SR-IOV capability
[PATCH 13/16 v6] PCI: reserve bus range for SR-IOV device
[PATCH 14/16 v6] PCI: document for SR-IOV user and developer [PATCH 15/16 v6] PCI: document the SR-IOV sysfs entries
[PATCH 16/16 v6] PCI: document the new PCI boot parameters
---
Single Root I/O Virtualization (SR-IOV) capability defined by PCI-SIG is intended to enable multiple system software to share PCI hardware resources. PCI device that supports this capability can be extended to one Physical Functions plus multiple Virtual Functions. Physical Function, which could be considered as the "real" PCI device, reflects the hardware instance and manages all physical resources. Virtual Functions are associated with a Physical Function and shares physical resources with the Physical Function.Software can control allocation of Virtual Functions via registers encapsulated in the capability structure.
SR-IOV specification can be found at
http://www.pcisig.com/members/downloads/specifications/iov/sr-iov1.0_11Sep07.pdf
Devices that support SR-IOV are available from following vendors:
http://download.intel.com/design/network/ProdBrf/320025.pdf
http://www.netxen.com/products/chipsolutions/NX3031.html
http://www.neterion.com/products/x3100.html
PATCH v6 1/16 PCI: remove unnecessary arg of pci_update_resource()
This cleanup removes unnecessary argument 'struct resource *res' in pci_update_resource(), so it takes same arguments as other companion functions (pci_assign_resource(), etc.).
--- drivers/pci/pci.c | 4 ++-- drivers/pci/setup-res.c | 7 ++++--- include/linux/pci.h | 2 +- 3 files changed, 7 insertions(+), 6 deletions(-)
drivers/pci/pci.c +2 -2
@@ -376,8 +376,8 @@ pci_restore_bars(struct pci_dev *dev)return ; } -for (i =0 ; i < numres; i ++) -pci_update_resource (dev, &dev->resource[i], i); +for (i =0 ; i < numres; i++) +pci_update_resource (dev, i); }static struct pci_platform_pm_ops *pci_platform_pm;
drivers/pci/setup-res.c +4 -3
@@ -26,11 +26,12 @@#include "pci.h" -void pci_update_resource (struct pci_dev *dev,struct resource *res,int resno) +void pci_update_resource (struct pci_dev *dev,int resno) {struct pci_bus_region region;u32 new, check, mask;int reg; +struct resource *res = dev->resource + resno;/* * Ignore resources for unimplemented BARs and unused resource slots
@@ -162,7 +163,7 @@ int pci_assign_resource(struct pci_dev *dev, int resno) }else { res->flags &= ~IORESOURCE_STARTALIGN ;if (resno <PCI_BRIDGE_RESOURCES ) -pci_update_resource (dev, res, resno); +pci_update_resource (dev, resno); }return ret;
@@ -197,7 +198,7 @@ int pci_assign_resource_fixed(struct pci_dev *dev, int resno)dev_err (&dev->dev,"BAR %d: can't allocate %s resource %pR\n" , resno, res->flags &IORESOURCE_IO ?"I/O" :"mem" , res); }else if (resno <PCI_BRIDGE_RESOURCES ) { -pci_update_resource (dev, res, resno); +pci_update_resource (dev, resno); }return ret;
include/linux/pci.h +1 -1
@@ -626,7 +626,7 @@ int pcix_get_mmrbc(struct pci_dev *dev);int pcix_set_mmrbc (struct pci_dev *dev,int mmrbc);int pcie_get_readrq (struct pci_dev *dev);int pcie_set_readrq (struct pci_dev *dev,int rq); -void pci_update_resource (struct pci_dev *dev,struct resource *res,int resno); +void pci_update_resource (struct pci_dev *dev,int resno);int __must_check pci_assign_resource (struct pci_dev *dev,int i);int pci_select_bars (struct pci_dev *dev,unsigned long flags);
Signature · 1 line
1.5.6.4
PATCH v6 2/16 PCI: define PCI resource names in an 'enum'
In reply to Yu ZhaoThis patch moves all definitions of the PCI resource names to an 'enum', and also replaces some hard-coded resource variables with symbol names. This change eases introduction of device specific resources.
--- drivers/pci/pci-sysfs.c | 4 +++- drivers/pci/pci.c | 19 ++----------------- drivers/pci/probe.c | 2 +- drivers/pci/proc.c | 7 ++++--- include/linux/pci.h | 37 ++++++++++++++++++++++++------------- 5 files changed, 34 insertions(+), 35 deletions(-)
drivers/pci/pci-sysfs.c +3 -1
@@ -101,11 +101,13 @@ resource_show(struct device * dev, struct device_attribute *attr, char * buf)struct pci_dev * pci_dev =to_pci_dev (dev);char * str = buf;int i; -int max =7 ; +int max;resource_size_t start, end;if (pci_dev->subordinate) max =DEVICE_COUNT_RESOURCE ; +else + max =PCI_BRIDGE_RESOURCES ;for (i =0 ; i < max; i++) {struct resource *res = &pci_dev->resource[i];
drivers/pci/pci.c +2 -17
@@ -359,24 +359,9 @@ pci_find_parent_resource(const struct pci_dev *dev, struct resource *res)static void pci_restore_bars (struct pci_dev *dev) { -int i, numres; - -switch (dev->hdr_type) { -case PCI_HEADER_TYPE_NORMAL : - numres =6 ; -break ; -case PCI_HEADER_TYPE_BRIDGE : - numres =2 ; -break ; -case PCI_HEADER_TYPE_CARDBUS : - numres =1 ; -break ; -default : -/* Should never get here, but just in case... */ -return ; - } +int i; -for (i =0 ; i < numres; i++) +for (i =0 ; i <PCI_BRIDGE_RESOURCES ; i++)pci_update_resource (dev, i); }
drivers/pci/probe.c +1 -1
@@ -426,7 +426,7 @@ static struct pci_bus *pci_alloc_child_bus(struct pci_bus *parent, child->subordinate =0xff ;/* Set up default resource pointers and names.. */ -for (i =0 ; i <4 ; i++) { +for (i =0 ; i <PCI_BRIDGE_RES_NUM ; i++) { child->resource[i] = &bridge->resource[PCI_BRIDGE_RESOURCES +i]; child->resource[i]->name = child->name; }
drivers/pci/proc.c +4 -3
@@ -352,15 +352,16 @@ static int show_device(struct seq_file *m, void *v) dev->vendor, dev->device, dev->irq); -/* Here should be 7 and not PCI_NUM_RESOURCES as we need to preserve compatibility */ -for (i=0 ; i<7 ; i++) { + +/* only print standard and ROM resources to preserve compatibility */ +for (i =0 ; i <=PCI_ROM_RESOURCE ; i++) {resource_size_t start, end;pci_resource_to_user (dev, i, &dev->resource[i], &start, &end);seq_printf (m,"\t%16llx" , (unsigned long long )(start | (dev->resource[i].flags &PCI_REGION_FLAG_MASK ))); } -for (i=0 ; i<7 ; i++) { +for (i =0 ; i <=PCI_ROM_RESOURCE ; i++) {resource_size_t start, end;pci_resource_to_user (dev, i, &dev->resource[i], &start, &end); seq_printf(m, "\t%16llx",
include/linux/pci.h +24 -13
@@ -76,7 +76,30 @@ enum pci_mmap_state {#define PCI_DMA_FROMDEVICE 2#define PCI_DMA_NONE 3 -#define DEVICE_COUNT_RESOURCE 12 +/* +* For PCI devices, the region numbers are assigned this way: +*/ +enum { +/* #0-5: standard PCI regions */ +PCI_STD_RESOURCES , +PCI_STD_RESOURCES_END =5 , + +/* #6: expansion ROM */ +PCI_ROM_RESOURCE , + +/* address space assigned to buses behind the bridge */ +#ifndef PCI_BRIDGE_RES_NUM +#define PCI_BRIDGE_RES_NUM 4 +#endif +PCI_BRIDGE_RESOURCES , +PCI_BRIDGE_RES_END =PCI_BRIDGE_RESOURCES +PCI_BRIDGE_RES_NUM -1 , + +/* total resources associated with a PCI device */ +PCI_NUM_RESOURCES , + +/* preserve this for compatibility */ +DEVICE_COUNT_RESOURCE +};typedef int __bitwise pci_power_t ;
@@ -262,18 +285,6 @@ static inline void pci_add_saved_cap(struct pci_dev *pci_dev,hlist_add_head (&new_cap->next, &pci_dev->saved_cap_space); } -/* -* For PCI devices, the region numbers are assigned this way: -* -* 0-5 standard PCI regions -* 6 expansion ROM -* 7-10 bridges: address space assigned to buses behind the bridge -*/ - -#define PCI_ROM_RESOURCE 6 -#define PCI_BRIDGE_RESOURCES 7 -#define PCI_NUM_RESOURCES 11 -#ifndef PCI_BUS_NUM_RESOURCES #define PCI_BUS_NUM_RESOURCES 16#endif
Signature · 1 line
1.5.6.4
PATCH v6 3/16 PCI: export __pci_read_base
In reply to Yu ZhaoExport __pci_read_base() so it can be used by whole PCI subsystem.
--- drivers/pci/pci.h | 9 +++++++++ drivers/pci/probe.c | 20 +++++++++----------- 2 files changed, 18 insertions(+), 11 deletions(-)
drivers/pci/pci.h +9 -0
@@ -157,6 +157,15 @@ struct pci_slot_attribute { };#define to_pci_slot_attr (s) container_of(s, struct pci_slot_attribute, attr) +enum pci_bar_type { + pci_bar_unknown,/* Standard PCI BAR probe */ + pci_bar_io,/* An io port BAR */ + pci_bar_mem32,/* A 32-bit memory BAR */ + pci_bar_mem64,/* A 64-bit memory BAR */ +}; + +extern int __pci_read_base (struct pci_dev *dev,enum pci_bar_type type, +struct resource *res,unsigned int reg);extern void pci_enable_ari (struct pci_dev *dev);/** * pci_ari_enabled - query ARI forwarding status
drivers/pci/probe.c +9 -11
@@ -135,13 +135,6 @@ static u64 pci_size(u64 base, u64 maxbase, u64 mask)return size; } -enum pci_bar_type { - pci_bar_unknown,/* Standard PCI BAR probe */ - pci_bar_io,/* An io port BAR */ - pci_bar_mem32,/* A 32-bit memory BAR */ - pci_bar_mem64,/* A 64-bit memory BAR */ -}; -static inline enum pci_bar_type decode_bar (struct resource *res,u32 bar) {if ((bar &PCI_BASE_ADDRESS_SPACE ) ==PCI_BASE_ADDRESS_SPACE_IO ) {
@@ -156,11 +149,16 @@ static inline enum pci_bar_type decode_bar(struct resource *res, u32 bar)return pci_bar_mem32; } -/* -* If the type is not unknown, we assume that the lowest bit is 'enable'. -* Returns 1 if the BAR was 64-bit and 0 if it was 32-bit. +/** +* pci_read_base - read a PCI BAR +* @dev: the PCI device +* @type: type of the BAR +* @res: resource buffer to be filled in +* @pos: BAR position in the config space +* +* Returns 1 if the BAR is 64-bit, or 0 if 32-bit. */ -static int __pci_read_base (struct pci_dev *dev,enum pci_bar_type type, +int __pci_read_base (struct pci_dev *dev,enum pci_bar_type type,struct resource *res,unsigned int pos) {u32 l, sz, mask;
Signature · 1 line
1.5.6.4
PATCH v6 3/16 PCI: export __pci_read_base
In reply to Yu ZhaoExport __pci_read_base() so it can be used by whole PCI subsystem.
--- drivers/pci/pci.h | 9 +++++++++ drivers/pci/probe.c | 20 +++++++++----------- 2 files changed, 18 insertions(+), 11 deletions(-)
drivers/pci/pci.h +9 -0
@@ -157,6 +157,15 @@ struct pci_slot_attribute { };#define to_pci_slot_attr (s) container_of(s, struct pci_slot_attribute, attr) +enum pci_bar_type { + pci_bar_unknown,/* Standard PCI BAR probe */ + pci_bar_io,/* An io port BAR */ + pci_bar_mem32,/* A 32-bit memory BAR */ + pci_bar_mem64,/* A 64-bit memory BAR */ +}; + +extern int __pci_read_base (struct pci_dev *dev,enum pci_bar_type type, +struct resource *res,unsigned int reg);extern void pci_enable_ari (struct pci_dev *dev);/** * pci_ari_enabled - query ARI forwarding status
drivers/pci/probe.c +9 -11
@@ -135,13 +135,6 @@ static u64 pci_size(u64 base, u64 maxbase, u64 mask)return size; } -enum pci_bar_type { - pci_bar_unknown,/* Standard PCI BAR probe */ - pci_bar_io,/* An io port BAR */ - pci_bar_mem32,/* A 32-bit memory BAR */ - pci_bar_mem64,/* A 64-bit memory BAR */ -}; -static inline enum pci_bar_type decode_bar (struct resource *res,u32 bar) {if ((bar &PCI_BASE_ADDRESS_SPACE ) ==PCI_BASE_ADDRESS_SPACE_IO ) {
@@ -156,11 +149,16 @@ static inline enum pci_bar_type decode_bar(struct resource *res, u32 bar)return pci_bar_mem32; } -/* -* If the type is not unknown, we assume that the lowest bit is 'enable'. -* Returns 1 if the BAR was 64-bit and 0 if it was 32-bit. +/** +* pci_read_base - read a PCI BAR +* @dev: the PCI device +* @type: type of the BAR +* @res: resource buffer to be filled in +* @pos: BAR position in the config space +* +* Returns 1 if the BAR is 64-bit, or 0 if 32-bit. */ -static int __pci_read_base (struct pci_dev *dev,enum pci_bar_type type, +int __pci_read_base (struct pci_dev *dev,enum pci_bar_type type,struct resource *res,unsigned int pos) {u32 l, sz, mask;
Signature · 1 line
1.5.6.4
PATCH v6 4/16 PCI: make pci_alloc_child_bus() be able to handle NULL bridge
In reply to Yu ZhaoMake pci_alloc_child_bus() be able to allocate buses without bridge devices. Some SR-IOV devices can occupy more than one bus number, but there is no explicit bridges because that have internal routing mechanism.
--- drivers/pci/probe.c | 7 +++++-- 1 files changed, 5 insertions(+), 2 deletions(-)
drivers/pci/probe.c +5 -2
@@ -401,12 +401,10 @@ static struct pci_bus *pci_alloc_child_bus(struct pci_bus *parent,if (!child)return NULL ; - child->self = bridge; child->parent = parent; child->ops = parent->ops; child->sysdata = parent->sysdata; child->bus_flags = parent->bus_flags; - child->bridge =get_device (&bridge->dev);/* initialize some portions of the bus device, but don't register it * now as the parent is not properly set up yet. This device will get
@@ -423,6 +421,11 @@ static struct pci_bus *pci_alloc_child_bus(struct pci_bus *parent, child->primary = parent->secondary; child->subordinate =0xff ; +if (!bridge) +return child; + + child->self = bridge; + child->bridge =get_device (&bridge->dev);/* Set up default resource pointers and names.. */ for (i =0 ; i <PCI_BRIDGE_RES_NUM ; i++) { child->resource[i] = &bridge->resource[PCI_BRIDGE_RESOURCES +i];
Signature · 1 line
1.5.6.4
PATCH v6 5/16 PCI: add a wrapper for resource_alignment()
In reply to Yu ZhaoAdd a wrapper for resource_alignment() so it can handle device specific resource alignment.
--- drivers/pci/pci.c | 20 ++++++++++++++++++++ drivers/pci/pci.h | 1 + drivers/pci/setup-bus.c | 4 ++-- drivers/pci/setup-res.c | 7 ++++--- 4 files changed, 27 insertions(+), 5 deletions(-)
drivers/pci/pci.c +20 -0
@@ -1904,6 +1904,26 @@ int pci_select_bars(struct pci_dev *dev, unsigned long flags)return bars; } +/** +* pci_resource_alignment - get a PCI BAR resource alignment +* @dev: the PCI device +* @resno: the resource number +* +* Returns alignment size on success, or 0 on error. +*/ +int pci_resource_alignment (struct pci_dev *dev,int resno) +{ +resource_size_t align; +struct resource *res = dev->resource + resno; + + align =resource_alignment (res); +if (align) +return align; + +dev_err (&dev->dev,"alignment: invalid resource #%d\n" , resno); +return 0 ; +} +static void __devinit pci_no_domains (void ) {#ifdef CONFIG_PCI_DOMAINS
drivers/pci/pci.h +1 -0
@@ -166,6 +166,7 @@ enum pci_bar_type {extern int __pci_read_base (struct pci_dev *dev,enum pci_bar_type type,struct resource *res,unsigned int reg); +extern int pci_resource_alignment (struct pci_dev *dev,int resno);extern void pci_enable_ari (struct pci_dev *dev);/** * pci_ari_enabled - query ARI forwarding status
drivers/pci/setup-bus.c +2 -2
@@ -25,6 +25,7 @@#include <linux/ioport.h> #include <linux/cache.h> #include <linux/slab.h> +#include "pci.h" static void pbus_assign_resources_sorted (struct pci_bus *bus)
@@ -351,8 +352,7 @@ static int pbus_size_mem(struct pci_bus *bus, unsigned long mask, unsigned longif (r->parent || (r->flags & mask) != type)continue ; r_size =resource_size (r); -/* For bridges size != alignment */ - align =resource_alignment (r); + align =pci_resource_alignment (dev, i); order =__ffs (align) -20 ;if (order >11 ) { dev_warn(&dev->dev, "BAR %d bad alignment %llx: "
drivers/pci/setup-res.c +4 -3
@@ -133,7 +133,7 @@ int pci_assign_resource(struct pci_dev *dev, int resno) size =resource_size (res); min = (res->flags &IORESOURCE_IO ) ?PCIBIOS_MIN_IO :PCIBIOS_MIN_MEM ; - align =resource_alignment (res); + align =pci_resource_alignment (dev, resno);if (!align) { dev_err(&dev->dev, "BAR %d: can't allocate resource (bogus " "alignment) %pR flags %#lx\n",
@@ -224,7 +224,7 @@ void pdev_sort_resources(struct pci_dev *dev, struct resource_list *head)if (!(r->flags) || r->parent)continue ; - r_align =resource_alignment (r); + r_align =pci_resource_alignment (dev, i);if (!r_align) { dev_warn(&dev->dev, "BAR %d: bogus alignment " "%pR flags %#lx\n",
@@ -236,7 +236,8 @@ void pdev_sort_resources(struct pci_dev *dev, struct resource_list *head)struct resource_list *ln = list->next;if (ln) - align =resource_alignment (ln->res); + align =pci_resource_alignment (ln->dev, + ln->res - ln->dev->resource);if (r_align > align) { tmp =kmalloc (sizeof (*tmp),GFP_KERNEL );
Signature · 1 line
1.5.6.4
PATCH v6 6/16 PCI: add a new function to map BAR offset
In reply to Yu ZhaoAdd a function to map resource number to corresponding register so people can get the offset and type of device specific BARs.
--- drivers/pci/pci.c | 22 ++++++++++++++++++++++ drivers/pci/pci.h | 2 ++ drivers/pci/setup-res.c | 13 +++++-------- 3 files changed, 29 insertions(+), 8 deletions(-)
drivers/pci/pci.c +22 -0
@@ -1924,6 +1924,28 @@ int pci_resource_alignment(struct pci_dev *dev, int resno)return 0 ; } +/** +* pci_resource_bar - get position of the BAR associated with a resource +* @dev: the PCI device +* @resno: the resource number +* @type: the BAR type to be filled in +* +* Returns BAR position in config space, or 0 if the BAR is invalid. +*/ +int pci_resource_bar (struct pci_dev *dev,int resno,enum pci_bar_type *type) +{ +if (resno <PCI_ROM_RESOURCE ) { + *type = pci_bar_unknown; +return PCI_BASE_ADDRESS_0 +4 * resno; + }else if (resno ==PCI_ROM_RESOURCE ) { + *type = pci_bar_mem32; +return dev->rom_base_reg; + } + +dev_err (&dev->dev,"BAR: invalid resource #%d\n" , resno); +return 0 ; +} +static void __devinit pci_no_domains (void ) {#ifdef CONFIG_PCI_DOMAINS
drivers/pci/pci.h +2 -0
@@ -167,6 +167,8 @@ enum pci_bar_type {extern int __pci_read_base (struct pci_dev *dev,enum pci_bar_type type,struct resource *res,unsigned int reg);extern int pci_resource_alignment (struct pci_dev *dev,int resno); +extern int pci_resource_bar (struct pci_dev *dev,int resno, +enum pci_bar_type *type);extern void pci_enable_ari (struct pci_dev *dev);/** * pci_ari_enabled - query ARI forwarding status
drivers/pci/setup-res.c +5 -8
@@ -31,6 +31,7 @@ void pci_update_resource(struct pci_dev *dev, int resno)struct pci_bus_region region;u32 new, check, mask;int reg; +enum pci_bar_type type;struct resource *res = dev->resource + resno;/*
@@ -62,17 +63,13 @@ void pci_update_resource(struct pci_dev *dev, int resno)else mask = (u32 )PCI_BASE_ADDRESS_MEM_MASK ; -if (resno <6 ) { - reg =PCI_BASE_ADDRESS_0 +4 * resno; - }else if (resno ==PCI_ROM_RESOURCE ) { + reg =pci_resource_bar (dev, resno, &type); +if (!reg) +return ; +if (type != pci_bar_unknown) {if (!(res->flags &IORESOURCE_ROM_ENABLE ))return ; new |=PCI_ROM_ADDRESS_ENABLE ; - reg = dev->rom_base_reg; - }else { -/* Hmm, non-standard resource. */ - -return ;/* kill uninitialised var warning */ }pci_write_config_dword (dev, reg, new);
Signature · 1 line
1.5.6.4
PATCH v6 7/16 PCI: cleanup pcibios_allocate_resources()
In reply to Yu ZhaoThis cleanup makes pcibios_allocate_resources() easier to read.
--- arch/x86/pci/i386.c | 28 ++++++++++++++-------------- 1 files changed, 14 insertions(+), 14 deletions(-)
arch/x86/pci/i386.c +14 -14
@@ -147,7 +147,7 @@ static void __init pcibios_allocate_bus_resources(struct list_head *bus_list)static void __init pcibios_allocate_resources (int pass) {struct pci_dev *dev =NULL ; -int idx, disabled; +int idx, enabled;u16 command;struct resource *r, *pr;
@@ -160,22 +160,22 @@ static void __init pcibios_allocate_resources(int pass)if (!r->start)/* Address not assigned at all */ continue ;if (r->flags &IORESOURCE_IO ) - disabled = !(command &PCI_COMMAND_IO ); + enabled = command &PCI_COMMAND_IO ;else - disabled = !(command &PCI_COMMAND_MEMORY ); -if (pass == disabled) { -dev_dbg (&dev->dev,"resource %#08llx-%#08llx (f=%lx, d=%d, p=%d)\n" , + enabled = command &PCI_COMMAND_MEMORY ; +if (pass == enabled) +continue ; +dev_dbg (&dev->dev,"resource %#08llx-%#08llx (f=%lx, d=%d, p=%d)\n" , (unsigned long long ) r->start, (unsigned long long ) r->end, - r->flags, disabled, pass); - pr =pci_find_parent_resource (dev, r); -if (!pr ||request_resource (pr, r) <0 ) { -dev_err (&dev->dev,"BAR %d: can't allocate resource\n" , idx); -/* We'll assign a new address later */ - r->end -= r->start; - r->start =0 ; - } - } + r->flags, enabled, pass); + pr =pci_find_parent_resource (dev, r); +if (pr && !request_resource (pr, r)) +continue ; +dev_err (&dev->dev,"BAR %d: can't allocate resource\n" , idx); +/* We'll assign a new address later */ + r->end -= r->start; + r->start =0 ; }if (!pass) { r = &dev->resource[PCI_ROM_RESOURCE ];
Signature · 1 line
1.5.6.4
PATCH v6 8/16 PCI: add boot options to reassign resources
In reply to Yu ZhaoThis patch adds boot options so user can reassign device resources of all devices under a bus.
The boot options can be used as:
pci=assign-mmio=0000:01,assign-pio=0000:02
'[dddd:]bb' is the domain and bus number.
--- arch/x86/pci/common.c | 73 +++++++++++++++++++++++++++++++++++++++++++++++++ arch/x86/pci/i386.c | 10 ++++--- arch/x86/pci/pci.h | 3 ++ 3 files changed, 82 insertions(+), 4 deletions(-)
arch/x86/pci/common.c +73 -0
@@ -137,6 +137,72 @@ static void __devinit pcibios_fixup_device_resources(struct pci_dev *dev) } } +static char *pci_assign_pio; +static char *pci_assign_mmio; + +static int pcibios_bus_resource_needs_fixup (struct pci_bus *bus) +{ +int i; +int type =0 ; +int domain, busnr; + +if (!bus->self) +return 0 ; + +for (i =0 ; i <2 ; i++) { +char *str = i ? pci_assign_pio : pci_assign_mmio; + +while (str && *str) { +if (sscanf (str,"%04x:%02x" , &domain, &busnr) !=2 ) { +if (sscanf (str,"%02x" , &busnr) !=1 ) +break ; + domain =0 ; + } + +if (pci_domain_nr (bus) == domain && + bus->number == busnr) { + type |= i ?IORESOURCE_IO :IORESOURCE_MEM ; +break ; + } + + str =strchr (str,';' ); +if (str) + str++; + } + } + +return type; +} + +static void __devinit pcibios_fixup_bus_resources (struct pci_bus *bus) +{ +int i; +int type =pcibios_bus_resource_needs_fixup (bus); + +if (!type) +return ; + +for (i =0 ; i <PCI_BUS_NUM_RESOURCES ; i++) { +struct resource *res = bus->resource[i]; + +if (!res) +continue ; +if (res->flags & type) + res->flags =0 ; + } +} + +int pcibios_resource_needs_fixup (struct pci_dev *dev,int resno) +{ +struct pci_bus *bus; + +for (bus = dev->bus; bus && bus != pci_root_bus; bus = bus->parent) +if (pcibios_bus_resource_needs_fixup (bus)) +return 1 ; + +return 0 ; +} +/* * Called after each bus is probed, but before its children * are examined.
@@ -147,6 +213,7 @@ void __devinit pcibios_fixup_bus(struct pci_bus *b)struct pci_dev *dev;pci_read_bridge_bases (b); +pcibios_fixup_bus_resources (b);list_for_each_entry (dev, &b->devices, bus_list)pcibios_fixup_device_resources (dev); }
@@ -519,6 +586,12 @@ char * __devinit pcibios_setup(char *str) }else if (!strcmp (str,"skip_isa_align" )) { pci_probe |=PCI_CAN_SKIP_ISA_ALIGN ;return NULL ; + }else if (!strncmp (str,"assign-pio=" ,11 )) { + pci_assign_pio = str +11 ; +return NULL ; + }else if (!strncmp (str,"assign-mmio=" ,12 )) { + pci_assign_mmio = str +12 ; +return NULL ; }return str; }
arch/x86/pci/i386.c +6 -4
@@ -169,10 +169,12 @@ static void __init pcibios_allocate_resources(int pass) (unsigned long long ) r->start, (unsigned long long ) r->end, r->flags, enabled, pass); - pr =pci_find_parent_resource (dev, r); -if (pr && !request_resource (pr, r)) -continue ; -dev_err (&dev->dev,"BAR %d: can't allocate resource\n" , idx); +if (!pcibios_resource_needs_fixup (dev, idx)) { + pr =pci_find_parent_resource (dev, r); +if (pr && !request_resource (pr, r)) +continue ; +dev_err (&dev->dev,"BAR %d: can't allocate resource\n" , idx); + }/* We'll assign a new address later */ r->end -= r->start; r->start =0 ;
arch/x86/pci/pci.h +3 -0
@@ -117,6 +117,9 @@ extern int __init pcibios_init(void);extern int __init pci_mmcfg_arch_init (void );extern void __init pci_mmcfg_arch_free (void ); +/* pci-common.c */ +extern int pcibios_resource_needs_fixup (struct pci_dev *dev,int resno); +/* * AMD Fam10h CPUs are buggy, and cannot access MMIO config space * on their northbrige except through the * %eax register. As such, you MUST
Signature · 1 line
1.5.6.4
PATCH v6 9/16 PCI: add boot option to align MMIO resources
In reply to Yu ZhaoThis patch adds boot option to align MMIO resource for a device. The alignment is a bigger value between the PAGE_SIZE and the resource size.
The boot option can be used as:
pci=align-mmio=0000:01:02.3
'[0000:]01:02.3' is the domain, bus, device and function number of the device.
--- arch/x86/pci/common.c | 37 +++++++++++++++++++++++++++++++++++++ drivers/pci/pci.c | 20 ++++++++++++++++++-- include/linux/pci.h | 1 + 3 files changed, 56 insertions(+), 2 deletions(-)
arch/x86/pci/common.c +37 -0
@@ -139,6 +139,7 @@ static void __devinit pcibios_fixup_device_resources(struct pci_dev *dev)static char *pci_assign_pio;static char *pci_assign_mmio; +static char *pci_align_mmio;static int pcibios_bus_resource_needs_fixup (struct pci_bus *bus) {
@@ -192,6 +193,36 @@ static void __devinit pcibios_fixup_bus_resources(struct pci_bus *bus) } } +int pcibios_resource_alignment (struct pci_dev *dev,int resno) +{ +int domain, busnr, slot, func; +char *str = pci_align_mmio; + +if (dev->resource[resno].flags &IORESOURCE_IO ) +return 0 ; + +while (str && *str) { +if (sscanf (str,"%04x:%02x:%02x.%d" , + &domain, &busnr, &slot, &func) !=4 ) { +if (sscanf (str,"%02x:%02x.%d" , + &busnr, &slot, &func) !=3 ) +break ; + domain =0 ; + } + +if (pci_domain_nr (dev->bus) == domain && + dev->bus->number == busnr && + dev->devfn ==PCI_DEVFN (slot, func)) +return PAGE_SIZE ; + + str =strchr (str,';' ); +if (str) + str++; + } + +return 0 ; +} +int pcibios_resource_needs_fixup (struct pci_dev *dev,int resno) {struct pci_bus *bus;
@@ -200,6 +231,9 @@ int pcibios_resource_needs_fixup(struct pci_dev *dev, int resno)if (pcibios_bus_resource_needs_fixup (bus))return 1 ; +if (pcibios_resource_alignment (dev, resno)) +return 1 ; +return 0 ; }
@@ -592,6 +626,9 @@ char * __devinit pcibios_setup(char *str) }else if (!strncmp (str,"assign-mmio=" ,12 )) { pci_assign_mmio = str +12 ;return NULL ; + }else if (!strncmp (str,"align-mmio=" ,11 )) { + pci_align_mmio = str +11 ; +return NULL ; }return str; }
drivers/pci/pci.c +18 -2
@@ -1015,6 +1015,20 @@ int __attribute__ ((weak)) pcibios_set_pcie_reset_state(struct pci_dev *dev, }/** +* pcibios_resource_alignment - get resource alignment requirement +* @dev: the PCI device +* @resno: resource number +* +* Queries the resource alignment from PCI low level code. Returns positive +* if there is alignment requirement of the resource, or 0 otherwise. +*/ +int __attribute__ ((weak))pcibios_resource_alignment (struct pci_dev *dev, +int resno) +{ +return 0 ; +} + +/** * pci_set_pcie_reset_state - set reset state for device dev * @dev: the PCI-E device reset * @state: Reset state to enter into
@@ -1913,12 +1927,14 @@ int pci_select_bars(struct pci_dev *dev, unsigned long flags)*/ int pci_resource_alignment (struct pci_dev *dev,int resno) { -resource_size_t align; +resource_size_t align, bios_align;struct resource *res = dev->resource + resno; + bios_align =pcibios_resource_alignment (dev, resno); + align =resource_alignment (res);if (align) -return align; +return align > bios_align ? align : bios_align;dev_err (&dev->dev,"alignment: invalid resource #%d\n" , resno);return 0 ;
include/linux/pci.h +1 -0
@@ -1121,6 +1121,7 @@ int pcibios_add_platform_entries(struct pci_dev *dev);void pcibios_disable_device (struct pci_dev *dev);int pcibios_set_pcie_reset_state (struct pci_dev *dev,enum pcie_reset_state state); +int pcibios_resource_alignment (struct pci_dev *dev,int resno);#ifdef CONFIG_PCI_MMCONFIG extern void __init pci_mmcfg_early_init (void );
Signature · 1 line
1.5.6.4
PATCH v6 9/16 PCI: add boot option to align MMIO resources
In reply to Yu ZhaoThis patch adds boot option to align MMIO resource for a device. The alignment is a bigger value between the PAGE_SIZE and the resource size.
The boot option can be used as:
pci=align-mmio=0000:01:02.3
'[0000:]01:02.3' is the domain, bus, device and function number of the device.
--- arch/x86/pci/common.c | 37 +++++++++++++++++++++++++++++++++++++ drivers/pci/pci.c | 20 ++++++++++++++++++-- include/linux/pci.h | 1 + 3 files changed, 56 insertions(+), 2 deletions(-)
arch/x86/pci/common.c +37 -0
@@ -139,6 +139,7 @@ static void __devinit pcibios_fixup_device_resources(struct pci_dev *dev)static char *pci_assign_pio;static char *pci_assign_mmio; +static char *pci_align_mmio;static int pcibios_bus_resource_needs_fixup (struct pci_bus *bus) {
@@ -192,6 +193,36 @@ static void __devinit pcibios_fixup_bus_resources(struct pci_bus *bus) } } +int pcibios_resource_alignment (struct pci_dev *dev,int resno) +{ +int domain, busnr, slot, func; +char *str = pci_align_mmio; + +if (dev->resource[resno].flags &IORESOURCE_IO ) +return 0 ; + +while (str && *str) { +if (sscanf (str,"%04x:%02x:%02x.%d" , + &domain, &busnr, &slot, &func) !=4 ) { +if (sscanf (str,"%02x:%02x.%d" , + &busnr, &slot, &func) !=3 ) +break ; + domain =0 ; + } + +if (pci_domain_nr (dev->bus) == domain && + dev->bus->number == busnr && + dev->devfn ==PCI_DEVFN (slot, func)) +return PAGE_SIZE ; + + str =strchr (str,';' ); +if (str) + str++; + } + +return 0 ; +} +int pcibios_resource_needs_fixup (struct pci_dev *dev,int resno) {struct pci_bus *bus;
@@ -200,6 +231,9 @@ int pcibios_resource_needs_fixup(struct pci_dev *dev, int resno)if (pcibios_bus_resource_needs_fixup (bus))return 1 ; +if (pcibios_resource_alignment (dev, resno)) +return 1 ; +return 0 ; }
@@ -592,6 +626,9 @@ char * __devinit pcibios_setup(char *str) }else if (!strncmp (str,"assign-mmio=" ,12 )) { pci_assign_mmio = str +12 ;return NULL ; + }else if (!strncmp (str,"align-mmio=" ,11 )) { + pci_align_mmio = str +11 ; +return NULL ; }return str; }
drivers/pci/pci.c +18 -2
@@ -1015,6 +1015,20 @@ int __attribute__ ((weak)) pcibios_set_pcie_reset_state(struct pci_dev *dev, }/** +* pcibios_resource_alignment - get resource alignment requirement +* @dev: the PCI device +* @resno: resource number +* +* Queries the resource alignment from PCI low level code. Returns positive +* if there is alignment requirement of the resource, or 0 otherwise. +*/ +int __attribute__ ((weak))pcibios_resource_alignment (struct pci_dev *dev, +int resno) +{ +return 0 ; +} + +/** * pci_set_pcie_reset_state - set reset state for device dev * @dev: the PCI-E device reset * @state: Reset state to enter into
@@ -1913,12 +1927,14 @@ int pci_select_bars(struct pci_dev *dev, unsigned long flags)*/ int pci_resource_alignment (struct pci_dev *dev,int resno) { -resource_size_t align; +resource_size_t align, bios_align;struct resource *res = dev->resource + resno; + bios_align =pcibios_resource_alignment (dev, resno); + align =resource_alignment (res);if (align) -return align; +return align > bios_align ? align : bios_align;dev_err (&dev->dev,"alignment: invalid resource #%d\n" , resno);return 0 ;
include/linux/pci.h +1 -0
@@ -1121,6 +1121,7 @@ int pcibios_add_platform_entries(struct pci_dev *dev);void pcibios_disable_device (struct pci_dev *dev);int pcibios_set_pcie_reset_state (struct pci_dev *dev,enum pcie_reset_state state); +int pcibios_resource_alignment (struct pci_dev *dev,int resno);#ifdef CONFIG_PCI_MMCONFIG extern void __init pci_mmcfg_early_init (void );
Signature · 1 line
1.5.6.4
PATCH v6 10/16 PCI: cleanup pci_bus_add_devices()
In reply to Yu ZhaoThis cleanup makes pci_bus_add_devices() easier to read.
--- drivers/pci/bus.c | 56 +++++++++++++++++++++++++------------------------ drivers/pci/remove.c | 2 + 2 files changed, 31 insertions(+), 27 deletions(-)
drivers/pci/bus.c +29 -27
@@ -71,7 +71,7 @@ pci_bus_alloc_resource(struct pci_bus *bus, struct resource *res, }/** -* add a single device +* pci_bus_add_device - add a single device * @dev: device to add * * This adds a single pci device to the global
@@ -105,7 +105,7 @@ int pci_bus_add_device(struct pci_dev *dev)void pci_bus_add_devices (struct pci_bus *bus) {struct pci_dev *dev; -struct pci_bus *child_bus; +struct pci_bus *child;int retval;list_for_each_entry (dev, &bus->devices, bus_list) {
@@ -120,39 +120,41 @@ void pci_bus_add_devices(struct pci_bus *bus)list_for_each_entry (dev, &bus->devices, bus_list) {BUG_ON (!dev->is_added); + child = dev->subordinate;/* * If there is an unattached subordinate bus, attach * it and then scan for unattached PCI devices. */ -if (dev->subordinate) { -if (list_empty (&dev->subordinate->node)) { -down_write (&pci_bus_sem); -list_add_tail (&dev->subordinate->node, - &dev->bus->children); -up_write (&pci_bus_sem); - } -pci_bus_add_devices (dev->subordinate); - -/* register the bus with sysfs as the parent is now -* properly registered. */ - child_bus = dev->subordinate; -if (child_bus->is_added) -continue ; - child_bus->dev.parent = child_bus->bridge; - retval =device_register (&child_bus->dev); -if (retval) -dev_err (&dev->dev,"Error registering pci_bus," -" continuing...\n" ); -else { - child_bus->is_added =1 ; - retval =device_create_file (&child_bus->dev, - &dev_attr_cpuaffinity); - } +if (!child) +continue ; +if (list_empty (&child->node)) { +down_write (&pci_bus_sem); +list_add_tail (&child->node, + &dev->bus->children); +up_write (&pci_bus_sem); + } +pci_bus_add_devices (child); + +/* +* register the bus with sysfs as the parent is now +* properly registered. +*/ +if (child->is_added) +continue ; + child->dev.parent = child->bridge; + retval =device_register (&child->dev); +if (retval) +dev_err (&dev->dev,"Error registering pci_bus," +" continuing...\n" ); +else { + child->is_added =1 ; + retval =device_create_file (&child->dev, + &dev_attr_cpuaffinity);if (retval)dev_err (&dev->dev,"Error creating cpuaffinity" " file, continuing...\n" ); - retval =device_create_file (&child_bus->dev, + retval =device_create_file (&child->dev, &dev_attr_cpulistaffinity);if (retval) dev_err(&dev->dev,
drivers/pci/remove.c +2 -0
@@ -72,6 +72,8 @@ void pci_remove_bus(struct pci_bus *pci_bus)list_del (&pci_bus->node);up_write (&pci_bus_sem);pci_remove_legacy_files (pci_bus); +if (!pci_bus->is_added) +return ;device_remove_file (&pci_bus->dev, &dev_attr_cpuaffinity);device_remove_file (&pci_bus->dev, &dev_attr_cpulistaffinity);device_unregister (&pci_bus->dev);
Signature · 1 line
1.5.6.4
PATCH v6 11/16 PCI: split a new function from pci_bus_add_devices()
In reply to Yu ZhaoThis patch splits a new function from pci_bus_add_devices(). The new function can be used to register PCI bus to the device core and create its sysfs entries.
--- drivers/pci/bus.c | 47 ++++++++++++++++++++++++++++------------------- include/linux/pci.h | 1 + 2 files changed, 29 insertions(+), 19 deletions(-)
drivers/pci/bus.c +28 -19
@@ -91,6 +91,32 @@ int pci_bus_add_device(struct pci_dev *dev) }/** +* pci_bus_add_child - add a child bus +* @bus: bus to add +* +* This adds sysfs entries for a single bus +*/ +int pci_bus_add_child (struct pci_bus *bus) +{ +int retval; + +if (bus->bridge) + bus->dev.parent = bus->bridge; + + retval =device_register (&bus->dev); +if (retval) +return retval; + + bus->is_added =1 ; + + retval =device_create_file (&bus->dev, &dev_attr_cpuaffinity); +if (retval) +return retval; + +return device_create_file (&bus->dev, &dev_attr_cpulistaffinity); +} + +/** * pci_bus_add_devices - insert newly discovered PCI devices * @bus: bus to check for new devices *
@@ -141,26 +167,9 @@ void pci_bus_add_devices(struct pci_bus *bus)*/ if (child->is_added)continue ; - child->dev.parent = child->bridge; - retval =device_register (&child->dev); + retval =pci_bus_add_child (child);if (retval) -dev_err (&dev->dev,"Error registering pci_bus," -" continuing...\n" ); -else { - child->is_added =1 ; - retval =device_create_file (&child->dev, - &dev_attr_cpuaffinity); -if (retval) -dev_err (&dev->dev,"Error creating cpuaffinity" -" file, continuing...\n" ); - - retval =device_create_file (&child->dev, - &dev_attr_cpulistaffinity); -if (retval) -dev_err (&dev->dev, -"Error creating cpulistaffinity" -" file, continuing...\n" ); - } +dev_err (&dev->dev,"Error adding bus, continuing\n" ); } }
include/linux/pci.h +1 -0
@@ -528,6 +528,7 @@ struct pci_dev *pci_scan_single_device(struct pci_bus *bus, int devfn);void pci_device_add (struct pci_dev *dev,struct pci_bus *bus);unsigned int pci_scan_child_bus (struct pci_bus *bus);int __must_check pci_bus_add_device (struct pci_dev *dev); +int pci_bus_add_child (struct pci_bus *bus);void pci_read_bridge_bases (struct pci_bus *child);struct resource *pci_find_parent_resource (const struct pci_dev *dev,struct resource *res);
Signature · 1 line
1.5.6.4
PATCH v6 11/16 PCI: split a new function from pci_bus_add_devices()
In reply to Yu ZhaoThis patch splits a new function from pci_bus_add_devices(). The new function can be used to register PCI bus to the device core and create its sysfs entries.
--- drivers/pci/bus.c | 47 ++++++++++++++++++++++++++++------------------- include/linux/pci.h | 1 + 2 files changed, 29 insertions(+), 19 deletions(-)
drivers/pci/bus.c +28 -19
@@ -91,6 +91,32 @@ int pci_bus_add_device(struct pci_dev *dev) }/** +* pci_bus_add_child - add a child bus +* @bus: bus to add +* +* This adds sysfs entries for a single bus +*/ +int pci_bus_add_child (struct pci_bus *bus) +{ +int retval; + +if (bus->bridge) + bus->dev.parent = bus->bridge; + + retval =device_register (&bus->dev); +if (retval) +return retval; + + bus->is_added =1 ; + + retval =device_create_file (&bus->dev, &dev_attr_cpuaffinity); +if (retval) +return retval; + +return device_create_file (&bus->dev, &dev_attr_cpulistaffinity); +} + +/** * pci_bus_add_devices - insert newly discovered PCI devices * @bus: bus to check for new devices *
@@ -141,26 +167,9 @@ void pci_bus_add_devices(struct pci_bus *bus)*/ if (child->is_added)continue ; - child->dev.parent = child->bridge; - retval =device_register (&child->dev); + retval =pci_bus_add_child (child);if (retval) -dev_err (&dev->dev,"Error registering pci_bus," -" continuing...\n" ); -else { - child->is_added =1 ; - retval =device_create_file (&child->dev, - &dev_attr_cpuaffinity); -if (retval) -dev_err (&dev->dev,"Error creating cpuaffinity" -" file, continuing...\n" ); - - retval =device_create_file (&child->dev, - &dev_attr_cpulistaffinity); -if (retval) -dev_err (&dev->dev, -"Error creating cpulistaffinity" -" file, continuing...\n" ); - } +dev_err (&dev->dev,"Error adding bus, continuing\n" ); } }
include/linux/pci.h +1 -0
@@ -528,6 +528,7 @@ struct pci_dev *pci_scan_single_device(struct pci_bus *bus, int devfn);void pci_device_add (struct pci_dev *dev,struct pci_bus *bus);unsigned int pci_scan_child_bus (struct pci_bus *bus);int __must_check pci_bus_add_device (struct pci_dev *dev); +int pci_bus_add_child (struct pci_bus *bus);void pci_read_bridge_bases (struct pci_bus *child);struct resource *pci_find_parent_resource (const struct pci_dev *dev,struct resource *res);
Signature · 1 line
1.5.6.4
PATCH v6 12/16 PCI: support the SR-IOV capability
In reply to Yu ZhaoSupport Single Root I/O Virtualization (SR-IOV) capability.
--- drivers/pci/Kconfig | 12 + drivers/pci/Makefile | 2 + drivers/pci/iov.c | 592 ++++++++++++++++++++++++++++++++++++++++++++++ drivers/pci/pci-sysfs.c | 4 + drivers/pci/pci.c | 14 + drivers/pci/pci.h | 48 ++++ drivers/pci/probe.c | 4 + include/linux/pci.h | 39 +++ include/linux/pci_regs.h | 21 ++ 9 files changed, 736 insertions(+), 0 deletions(-) create mode 100644 drivers/pci/iov.c
drivers/pci/Kconfig +12 -0
@@ -50,3 +50,15 @@ config HT_IRQ This allows native hypertransport devices to use interrupts. If unsure say Y. + +config PCI_IOV + bool "PCI SR-IOV support" + depends on PCI + select PCI_MSI + default n + help + This option allows device drivers to enable Single Root I/O + Virtualization. Each Virtual Function's PCI configuration + space can be accessed using its own Bus, Device and Function + Number (Routing ID). Each Virtual Function also has PCI Memory + Space, which is used to map its own register set.
drivers/pci/Makefile +2 -0
@@ -55,3 +55,5 @@ obj-$(CONFIG_PCI_SYSCALL) += syscall.o ifeq ($(CONFIG_PCI_DEBUG),y) EXTRA_CFLAGS += -DDEBUG endif + +obj-$(CONFIG_PCI_IOV) += iov.o
drivers/pci/iov.c new file +592 -0
592 lines not on this page · raw message
drivers/pci/pci-sysfs.c +4 -0
@@ -847,6 +847,9 @@ static int pci_create_capabilities_sysfs(struct pci_dev *dev)/* Active State Power Management */ pcie_aspm_create_sysfs_dev_files (dev); +/* Single Root I/O Virtualization */ +pci_iov_create_sysfs (dev); +return 0 ; }
@@ -932,6 +935,7 @@ static void pci_remove_capabilities_sysfs(struct pci_dev *dev) }pcie_aspm_remove_sysfs_dev_files (dev); +pci_iov_remove_sysfs (dev); }/**
drivers/pci/pci.c +14 -0
@@ -1936,6 +1936,13 @@ int pci_resource_alignment(struct pci_dev *dev, int resno)if (align)return align > bios_align ? align : bios_align; +if (resno >PCI_ROM_RESOURCE && resno <PCI_BRIDGE_RESOURCES ) { +/* device specific resource */ + align =pci_iov_resource_align (dev, resno); +if (align) +return align > bios_align ? align : bios_align; + } +dev_err (&dev->dev,"alignment: invalid resource #%d\n" , resno);return 0 ; }
@@ -1950,12 +1957,19 @@ int pci_resource_alignment(struct pci_dev *dev, int resno)*/ int pci_resource_bar (struct pci_dev *dev,int resno,enum pci_bar_type *type) { +int reg; +if (resno <PCI_ROM_RESOURCE ) { *type = pci_bar_unknown;return PCI_BASE_ADDRESS_0 +4 * resno; }else if (resno ==PCI_ROM_RESOURCE ) { *type = pci_bar_mem32;return dev->rom_base_reg; + }else if (resno <PCI_BRIDGE_RESOURCES ) { +/* device specific resource */ + reg =pci_iov_resource_bar (dev, resno, type); +if (reg) +return reg; }dev_err (&dev->dev,"BAR: invalid resource #%d\n" , resno);
drivers/pci/pci.h +48 -0
52 lines not on this page · raw message
drivers/pci/probe.c +4 -0
@@ -779,6 +779,7 @@ static int pci_setup_device(struct pci_dev * dev)static void pci_release_capabilities (struct pci_dev *dev) {pci_vpd_release (dev); +pci_iov_release (dev); }/**
@@ -962,6 +963,9 @@ static void pci_init_capabilities(struct pci_dev *dev)/* Alternative Routing-ID Forwarding */ pci_enable_ari (dev); + +/* Single Root I/O Virtualization */ +pci_iov_init (dev); }void pci_device_add (struct pci_dev *dev,struct pci_bus *bus)
include/linux/pci.h +39 -0
@@ -87,6 +87,12 @@ enum {/* #6: expansion ROM */ PCI_ROM_RESOURCE, +/* device specific resources */ +#ifdef CONFIG_PCI_IOV + PCI_IOV_RESOURCES, + PCI_IOV_RESOURCES_END = PCI_IOV_RESOURCES + PCI_IOV_NUM_BAR - 1, +#endif +/* address space assigned to buses behind the bridge */ #ifndef PCI_BRIDGE_RES_NUM #define PCI_BRIDGE_RES_NUM 4
@@ -165,6 +171,7 @@ struct pci_cap_saved_state {struct pcie_link_state ;struct pci_vpd ; +struct pci_iov ;/* * The pci_dev structure is used to describe PCI devices.
@@ -253,6 +260,7 @@ struct pci_dev {struct list_head msi_list;#endif struct pci_vpd *vpd; +struct pci_iov *iov; };extern struct pci_dev *alloc_pci_dev (void );
@@ -1147,5 +1155,36 @@ static inline void * pci_ioremap_bar(struct pci_dev *pdev, int bar) }#endif +/* SR-IOV events masks */ +#define PCI_IOV_NUM_VIRTFN 0x0000FFFFU/* NumVFs to be set */ +/* SR-IOV events values */ +#define PCI_IOV_ENABLE 0x00010000U/* SR-IOV enable request */ +#define PCI_IOV_DISABLE 0x00020000U/* SR-IOV disable request */ +#define PCI_IOV_NUMVFS 0x00040000U/* SR-IOV disable request */ + +#ifdef CONFIG_PCI_IOV +extern int pci_iov_enable (struct pci_dev *dev,int numvfs); +extern void pci_iov_disable (struct pci_dev *dev); +extern int pci_iov_register (struct pci_dev *dev, +int (*callback)(struct pci_dev *dev,u32 event)); +extern void pci_iov_unregister (struct pci_dev *dev); +#else +static inline int pci_iov_enable (struct pci_dev *dev,int numvfs) +{ +return -EIO ; +} +static inline void pci_iov_disable (struct pci_dev *dev) +{ +} +static inline int pci_iov_register (struct pci_dev *dev, +int (*callback)(struct pci_dev *dev,u32 event)) +{ +return -EIO ; +} +static inline void pci_iov_unregister (struct pci_dev *dev) +{ +} +#endif /* CONFIG_PCI_IOV */ +#endif /* __KERNEL__ */ #endif /* LINUX_PCI_H */
include/linux/pci_regs.h +21 -0
@@ -363,6 +363,7 @@#define PCI_EXP_TYPE_UPSTREAM 0x5/* Upstream Port */ #define PCI_EXP_TYPE_DOWNSTREAM 0x6/* Downstream Port */ #define PCI_EXP_TYPE_PCI_BRIDGE 0x7/* PCI/PCI-X Bridge */ +#define PCI_EXP_TYPE_RC_END 0x9/* Root Complex Integrated Endpoint */ #define PCI_EXP_FLAGS_SLOT 0x0100/* Slot implemented */ #define PCI_EXP_FLAGS_IRQ 0x3e00/* Interrupt message number */ #define PCI_EXP_DEVCAP 4/* Device capabilities */
@@ -434,6 +435,7 @@#define PCI_EXT_CAP_ID_DSN 3#define PCI_EXT_CAP_ID_PWR 4#define PCI_EXT_CAP_ID_ARI 14 +#define PCI_EXT_CAP_ID_IOV 16/* Advanced Error Reporting */ #define PCI_ERR_UNCOR_STATUS 4/* Uncorrectable Error Status */
@@ -551,4 +553,23 @@#define PCI_ARI_CTRL_ACS 0x0002/* ACS Function Groups Enable */ #define PCI_ARI_CTRL_FG (x) (((x) >> 4) & 7)/* Function Group */ +/* Single Root I/O Virtualization */ +#define PCI_IOV_CAP 0x04/* SR-IOV Capabilities */ +#define PCI_IOV_CTRL 0x08/* SR-IOV Control */ +#define PCI_IOV_CTRL_VFE 0x01/* VF Enable */ +#define PCI_IOV_CTRL_MSE 0x08/* VF Memory Space Enable */ +#define PCI_IOV_CTRL_ARI 0x10/* ARI Capable Hierarchy */ +#define PCI_IOV_STATUS 0x0a/* SR-IOV Status */ +#define PCI_IOV_INITIAL_VF 0x0c/* Initial VFs */ +#define PCI_IOV_TOTAL_VF 0x0e/* Total VFs */ +#define PCI_IOV_NUM_VF 0x10/* Number of VFs */ +#define PCI_IOV_FUNC_LINK 0x12/* Function Dependency Link */ +#define PCI_IOV_VF_OFFSET 0x14/* First VF Offset */ +#define PCI_IOV_VF_STRIDE 0x16/* Following VF Stride */ +#define PCI_IOV_VF_DID 0x1a/* VF Device ID */ +#define PCI_IOV_SUP_PGSIZE 0x1c/* Supported Page Sizes */ +#define PCI_IOV_SYS_PGSIZE 0x20/* System Page Size */ +#define PCI_IOV_BAR_0 0x24/* VF BAR0 */ +#define PCI_IOV_NUM_BAR 6/* Number of VF BARs */ +#endif /* LINUX_PCI_REGS_H */
Signature · 1 line
1.5.6.4
PATCH v6 13/16 PCI: reserve bus range for SR-IOV device
In reply to Yu ZhaoReserve bus range for SR-IOV at device scanning stage.
--- drivers/pci/iov.c | 24 ++++++++++++++++++++++++ drivers/pci/pci.h | 5 +++++ drivers/pci/probe.c | 3 +++ 3 files changed, 32 insertions(+), 0 deletions(-)
drivers/pci/iov.c +24 -0
@@ -498,6 +498,30 @@ int pci_iov_resource_bar(struct pci_dev *dev, int resno, }/** +* pci_iov_bus_range - find bus range used by SR-IOV capability +* @bus: the PCI bus +* +* Returns max number of buses (exclude current one) used by Virtual +* Functions. +*/ +int pci_iov_bus_range (struct pci_bus *bus) +{ +int max =0 ; +u8 busnr, devfn; +struct pci_dev *dev; + +list_for_each_entry (dev, &bus->devices, bus_list) { +if (!dev->iov) +continue ; +vf_rid (dev, dev->iov->totalvfs -1 , &busnr, &devfn); +if (busnr > max) + max = busnr; + } + +return max ? max - bus->number :0 ; +} + +/** * pci_iov_register - register SR-IOV service * @dev: the PCI device * @callback: callback function for SR-IOV events
drivers/pci/pci.h +5 -0
@@ -204,6 +204,7 @@ void pci_iov_remove_sysfs(struct pci_dev *dev);extern int pci_iov_resource_align (struct pci_dev *dev,int resno);extern int pci_iov_resource_bar (struct pci_dev *dev,int resno,enum pci_bar_type *type); +extern int pci_iov_bus_range (struct pci_bus *bus);#else static inline int pci_iov_init (struct pci_dev *dev) {
@@ -227,6 +228,10 @@ static inline int pci_iov_resource_bar(struct pci_dev *dev, int resno, {return 0 ; } +extern inline int pci_iov_bus_range (struct pci_bus *bus) +{ +return 0 ; +}#endif /* CONFIG_PCI_IOV */ #endif /* DRIVERS_PCI_H */
drivers/pci/probe.c +3 -0
@@ -1068,6 +1068,9 @@ unsigned int __devinit pci_scan_child_bus(struct pci_bus *bus)for (devfn =0 ; devfn <0x100 ; devfn +=8 )pci_scan_slot (bus, devfn); +/* Reserve buses for SR-IOV capability. */ + max +=pci_iov_bus_range (bus); +/* * After performing arch-dependent fixup of the bus, look behind * all PCI-to-PCI bridges on this bus.
Signature · 1 line
1.5.6.4
PATCH v6 13/16 PCI: reserve bus range for SR-IOV device
In reply to Yu ZhaoReserve bus range for SR-IOV at device scanning stage.
--- drivers/pci/iov.c | 24 ++++++++++++++++++++++++ drivers/pci/pci.h | 5 +++++ drivers/pci/probe.c | 3 +++ 3 files changed, 32 insertions(+), 0 deletions(-)
drivers/pci/iov.c +24 -0
@@ -498,6 +498,30 @@ int pci_iov_resource_bar(struct pci_dev *dev, int resno, }/** +* pci_iov_bus_range - find bus range used by SR-IOV capability +* @bus: the PCI bus +* +* Returns max number of buses (exclude current one) used by Virtual +* Functions. +*/ +int pci_iov_bus_range (struct pci_bus *bus) +{ +int max =0 ; +u8 busnr, devfn; +struct pci_dev *dev; + +list_for_each_entry (dev, &bus->devices, bus_list) { +if (!dev->iov) +continue ; +vf_rid (dev, dev->iov->totalvfs -1 , &busnr, &devfn); +if (busnr > max) + max = busnr; + } + +return max ? max - bus->number :0 ; +} + +/** * pci_iov_register - register SR-IOV service * @dev: the PCI device * @callback: callback function for SR-IOV events
drivers/pci/pci.h +5 -0
@@ -204,6 +204,7 @@ void pci_iov_remove_sysfs(struct pci_dev *dev);extern int pci_iov_resource_align (struct pci_dev *dev,int resno);extern int pci_iov_resource_bar (struct pci_dev *dev,int resno,enum pci_bar_type *type); +extern int pci_iov_bus_range (struct pci_bus *bus);#else static inline int pci_iov_init (struct pci_dev *dev) {
@@ -227,6 +228,10 @@ static inline int pci_iov_resource_bar(struct pci_dev *dev, int resno, {return 0 ; } +extern inline int pci_iov_bus_range (struct pci_bus *bus) +{ +return 0 ; +}#endif /* CONFIG_PCI_IOV */ #endif /* DRIVERS_PCI_H */
drivers/pci/probe.c +3 -0
@@ -1068,6 +1068,9 @@ unsigned int __devinit pci_scan_child_bus(struct pci_bus *bus)for (devfn =0 ; devfn <0x100 ; devfn +=8 )pci_scan_slot (bus, devfn); +/* Reserve buses for SR-IOV capability. */ + max +=pci_iov_bus_range (bus); +/* * After performing arch-dependent fixup of the bus, look behind * all PCI-to-PCI bridges on this bus.
Signature · 1 line
1.5.6.4
PATCH v6 14/16 PCI: document for SR-IOV user and developer
In reply to Yu ZhaoCreate HOW-TO for SR-IOV user and driver developer.
--- Documentation/DocBook/kernel-api.tmpl | 1 + Documentation/PCI/pci-iov-howto.txt | 181 +++++++++++++++++++++++++++++++++ 2 files changed, 182 insertions(+), 0 deletions(-) create mode 100644 Documentation/PCI/pci-iov-howto.txt
Documentation/DocBook/kernel-api.tmpl +1 -0
@@ -251,6 +251,7 @@ X!Edrivers/pci/hotplug.c --> !Edrivers/pci/probe.c !Edrivers/pci/rom.c +!Edrivers/pci/iov.c </sect1> <sect1><title>PCI Hotplug Support Library</title> !Edrivers/pci/hotplug/pci_hotplug_core.c
Documentation/PCI/pci-iov-howto.txt new file +181 -0
181 lines not on this page · raw message
Signature · 1 line
1.5.6.4
PATCH v6 15/16 PCI: document the SR-IOV sysfs entries
In reply to Yu ZhaoDocument the SR-IOV sysfs entries.
--- Documentation/ABI/testing/sysfs-bus-pci | 33 +++++++++++++++++++++++++++++++ 1 files changed, 33 insertions(+), 0 deletions(-)
Documentation/ABI/testing/sysfs-bus-pci +33 -0
@@ -9,3 +9,36 @@ Description: that some devices may have malformatted data. If the underlying VPD has a writable section then the corresponding section of this file will be writable. + +What: /sys/bus/pci/devices/.../iov/enable +Date: October 2008 +Contact: Yu Zhao <[email protected]> +Description: + This file appears when a device has the SR-IOV capability. + It holds the status of the capability, and could be written + (0/1) to disable and enable the capability if the PF driver + supports this operation. + +What: /sys/bus/pci/devices/.../iov/initialvfs +Date: October 2008 +Contact: Yu Zhao <[email protected]> +Description: + This file appears when a device has the SR-IOV capability. + It holds the number of initial Virtual Functions (read-only). + +What: /sys/bus/pci/devices/.../iov/totalvfs +Date: October 2008 +Contact: Yu Zhao <[email protected]> +Description: + This file appears when a device has the SR-IOV capability. + It holds the number of total Virtual Functions (read-only). + + +What: /sys/bus/pci/devices/.../iov/numvfs +Date: October 2008 +Contact: Yu Zhao <[email protected]> +Description: + This file appears when a device has the SR-IOV capability. + It holds the number of available Virtual Functions, and + could be written (1 ~ InitialVFs) to change the number if + the PF driver supports this operation.
Signature · 1 line
1.5.6.4
PATCH v6 16/16 PCI: document the new PCI boot parameters
In reply to Yu ZhaoDocument the new PCI[x86] boot parameters.
--- Documentation/kernel-parameters.txt | 10 ++++++++++ 1 files changed, 10 insertions(+), 0 deletions(-)
Documentation/kernel-parameters.txt +10 -0
@@ -1677,6 +1677,16 @@ and is between 256 and 4096 characters. It is defined in the file cbmemsize=nn[KMG] The fixed amount of bus space which is reserved for the CardBus bridge's memory window. The default value is 64 megabytes. + assign-mmio=[dddd:]bb [X86] reassign memory resources of all + devices under bus [dddd:]bb (dddd is the domain + number and bb is the bus number). + assign-pio=[dddd:]bb [X86] reassign io port resources of all + devices under bus [dddd:]bb (dddd is the domain + number and bb is the bus number). + align-mmio=[dddd:]bb:dd.f [X86] relocate memory resources of a + device to minimum PAGE_SIZE alignment (dddd is + the domain number and bb, dd and f is the bus, + device and function number). pcie_aspm= [PCIE] Forcibly enable or disable PCIe Active State Power Management.
Signature · 1 line
1.5.6.4
PATCH v6 16/16 PCI: document the new PCI boot parameters
In reply to Yu ZhaoDocument the new PCI[x86] boot parameters.
--- Documentation/kernel-parameters.txt | 10 ++++++++++ 1 files changed, 10 insertions(+), 0 deletions(-)
Documentation/kernel-parameters.txt +10 -0
@@ -1677,6 +1677,16 @@ and is between 256 and 4096 characters. It is defined in the file cbmemsize=nn[KMG] The fixed amount of bus space which is reserved for the CardBus bridge's memory window. The default value is 64 megabytes. + assign-mmio=[dddd:]bb [X86] reassign memory resources of all + devices under bus [dddd:]bb (dddd is the domain + number and bb is the bus number). + assign-pio=[dddd:]bb [X86] reassign io port resources of all + devices under bus [dddd:]bb (dddd is the domain + number and bb is the bus number). + align-mmio=[dddd:]bb:dd.f [X86] relocate memory resources of a + device to minimum PAGE_SIZE alignment (dddd is + the domain number and bb, dd and f is the bus, + device and function number). pcie_aspm= [PCIE] Forcibly enable or disable PCIe Active State Power Management.
Signature · 1 line
1.5.6.4
PATCH v6 1/16 PCI: remove unnecessary arg of pci_update_resource()
In reply to Yu ZhaoThis cleanup removes unnecessary argument 'struct resource *res' in pci_update_resource(), so it takes same arguments as other companion functions (pci_assign_resource(), etc.).
--- drivers/pci/pci.c | 4 ++-- drivers/pci/setup-res.c | 7 ++++--- include/linux/pci.h | 2 +- 3 files changed, 7 insertions(+), 6 deletions(-)
drivers/pci/pci.c +2 -2
@@ -376,8 +376,8 @@ pci_restore_bars(struct pci_dev *dev)return ; } -for (i =0 ; i < numres; i ++) -pci_update_resource (dev, &dev->resource[i], i); +for (i =0 ; i < numres; i++) +pci_update_resource (dev, i); }static struct pci_platform_pm_ops *pci_platform_pm;
drivers/pci/setup-res.c +4 -3
@@ -26,11 +26,12 @@#include "pci.h" -void pci_update_resource (struct pci_dev *dev,struct resource *res,int resno) +void pci_update_resource (struct pci_dev *dev,int resno) {struct pci_bus_region region;u32 new, check, mask;int reg; +struct resource *res = dev->resource + resno;/* * Ignore resources for unimplemented BARs and unused resource slots
@@ -162,7 +163,7 @@ int pci_assign_resource(struct pci_dev *dev, int resno) }else { res->flags &= ~IORESOURCE_STARTALIGN ;if (resno <PCI_BRIDGE_RESOURCES ) -pci_update_resource (dev, res, resno); +pci_update_resource (dev, resno); }return ret;
@@ -197,7 +198,7 @@ int pci_assign_resource_fixed(struct pci_dev *dev, int resno)dev_err (&dev->dev,"BAR %d: can't allocate %s resource %pR\n" , resno, res->flags &IORESOURCE_IO ?"I/O" :"mem" , res); }else if (resno <PCI_BRIDGE_RESOURCES ) { -pci_update_resource (dev, res, resno); +pci_update_resource (dev, resno); }return ret;
include/linux/pci.h +1 -1
@@ -626,7 +626,7 @@ int pcix_get_mmrbc(struct pci_dev *dev);int pcix_set_mmrbc (struct pci_dev *dev,int mmrbc);int pcie_get_readrq (struct pci_dev *dev);int pcie_set_readrq (struct pci_dev *dev,int rq); -void pci_update_resource (struct pci_dev *dev,struct resource *res,int resno); +void pci_update_resource (struct pci_dev *dev,int resno);int __must_check pci_assign_resource (struct pci_dev *dev,int i);int pci_select_bars (struct pci_dev *dev,unsigned long flags);
Signature · 1 line
1.5.6.4
PATCH v6 2/16 PCI: define PCI resource names in an 'enum'
In reply to Yu ZhaoThis patch moves all definitions of the PCI resource names to an 'enum', and also replaces some hard-coded resource variables with symbol names. This change eases introduction of device specific resources.
--- drivers/pci/pci-sysfs.c | 4 +++- drivers/pci/pci.c | 19 ++----------------- drivers/pci/probe.c | 2 +- drivers/pci/proc.c | 7 ++++--- include/linux/pci.h | 37 ++++++++++++++++++++++++------------- 5 files changed, 34 insertions(+), 35 deletions(-)
drivers/pci/pci-sysfs.c +3 -1
@@ -101,11 +101,13 @@ resource_show(struct device * dev, struct device_attribute *attr, char * buf)struct pci_dev * pci_dev =to_pci_dev (dev);char * str = buf;int i; -int max =7 ; +int max;resource_size_t start, end;if (pci_dev->subordinate) max =DEVICE_COUNT_RESOURCE ; +else + max =PCI_BRIDGE_RESOURCES ;for (i =0 ; i < max; i++) {struct resource *res = &pci_dev->resource[i];
drivers/pci/pci.c +2 -17
@@ -359,24 +359,9 @@ pci_find_parent_resource(const struct pci_dev *dev, struct resource *res)static void pci_restore_bars (struct pci_dev *dev) { -int i, numres; - -switch (dev->hdr_type) { -case PCI_HEADER_TYPE_NORMAL : - numres =6 ; -break ; -case PCI_HEADER_TYPE_BRIDGE : - numres =2 ; -break ; -case PCI_HEADER_TYPE_CARDBUS : - numres =1 ; -break ; -default : -/* Should never get here, but just in case... */ -return ; - } +int i; -for (i =0 ; i < numres; i++) +for (i =0 ; i <PCI_BRIDGE_RESOURCES ; i++)pci_update_resource (dev, i); }
drivers/pci/probe.c +1 -1
@@ -426,7 +426,7 @@ static struct pci_bus *pci_alloc_child_bus(struct pci_bus *parent, child->subordinate =0xff ;/* Set up default resource pointers and names.. */ -for (i =0 ; i <4 ; i++) { +for (i =0 ; i <PCI_BRIDGE_RES_NUM ; i++) { child->resource[i] = &bridge->resource[PCI_BRIDGE_RESOURCES +i]; child->resource[i]->name = child->name; }
drivers/pci/proc.c +4 -3
@@ -352,15 +352,16 @@ static int show_device(struct seq_file *m, void *v) dev->vendor, dev->device, dev->irq); -/* Here should be 7 and not PCI_NUM_RESOURCES as we need to preserve compatibility */ -for (i=0 ; i<7 ; i++) { + +/* only print standard and ROM resources to preserve compatibility */ +for (i =0 ; i <=PCI_ROM_RESOURCE ; i++) {resource_size_t start, end;pci_resource_to_user (dev, i, &dev->resource[i], &start, &end);seq_printf (m,"\t%16llx" , (unsigned long long )(start | (dev->resource[i].flags &PCI_REGION_FLAG_MASK ))); } -for (i=0 ; i<7 ; i++) { +for (i =0 ; i <=PCI_ROM_RESOURCE ; i++) {resource_size_t start, end;pci_resource_to_user (dev, i, &dev->resource[i], &start, &end); seq_printf(m, "\t%16llx",
include/linux/pci.h +24 -13
@@ -76,7 +76,30 @@ enum pci_mmap_state {#define PCI_DMA_FROMDEVICE 2#define PCI_DMA_NONE 3 -#define DEVICE_COUNT_RESOURCE 12 +/* +* For PCI devices, the region numbers are assigned this way: +*/ +enum { +/* #0-5: standard PCI regions */ +PCI_STD_RESOURCES , +PCI_STD_RESOURCES_END =5 , + +/* #6: expansion ROM */ +PCI_ROM_RESOURCE , + +/* address space assigned to buses behind the bridge */ +#ifndef PCI_BRIDGE_RES_NUM +#define PCI_BRIDGE_RES_NUM 4 +#endif +PCI_BRIDGE_RESOURCES , +PCI_BRIDGE_RES_END =PCI_BRIDGE_RESOURCES +PCI_BRIDGE_RES_NUM -1 , + +/* total resources associated with a PCI device */ +PCI_NUM_RESOURCES , + +/* preserve this for compatibility */ +DEVICE_COUNT_RESOURCE +};typedef int __bitwise pci_power_t ;
@@ -262,18 +285,6 @@ static inline void pci_add_saved_cap(struct pci_dev *pci_dev,hlist_add_head (&new_cap->next, &pci_dev->saved_cap_space); } -/* -* For PCI devices, the region numbers are assigned this way: -* -* 0-5 standard PCI regions -* 6 expansion ROM -* 7-10 bridges: address space assigned to buses behind the bridge -*/ - -#define PCI_ROM_RESOURCE 6 -#define PCI_BRIDGE_RESOURCES 7 -#define PCI_NUM_RESOURCES 11 -#ifndef PCI_BUS_NUM_RESOURCES #define PCI_BUS_NUM_RESOURCES 16#endif
Signature · 1 line
1.5.6.4
PATCH v6 5/16 PCI: add a wrapper for resource_alignment()
In reply to Yu ZhaoAdd a wrapper for resource_alignment() so it can handle device specific resource alignment.
--- drivers/pci/pci.c | 20 ++++++++++++++++++++ drivers/pci/pci.h | 1 + drivers/pci/setup-bus.c | 4 ++-- drivers/pci/setup-res.c | 7 ++++--- 4 files changed, 27 insertions(+), 5 deletions(-)
drivers/pci/pci.c +20 -0
@@ -1904,6 +1904,26 @@ int pci_select_bars(struct pci_dev *dev, unsigned long flags)return bars; } +/** +* pci_resource_alignment - get a PCI BAR resource alignment +* @dev: the PCI device +* @resno: the resource number +* +* Returns alignment size on success, or 0 on error. +*/ +int pci_resource_alignment (struct pci_dev *dev,int resno) +{ +resource_size_t align; +struct resource *res = dev->resource + resno; + + align =resource_alignment (res); +if (align) +return align; + +dev_err (&dev->dev,"alignment: invalid resource #%d\n" , resno); +return 0 ; +} +static void __devinit pci_no_domains (void ) {#ifdef CONFIG_PCI_DOMAINS
drivers/pci/pci.h +1 -0
@@ -166,6 +166,7 @@ enum pci_bar_type {extern int __pci_read_base (struct pci_dev *dev,enum pci_bar_type type,struct resource *res,unsigned int reg); +extern int pci_resource_alignment (struct pci_dev *dev,int resno);extern void pci_enable_ari (struct pci_dev *dev);/** * pci_ari_enabled - query ARI forwarding status
drivers/pci/setup-bus.c +2 -2
@@ -25,6 +25,7 @@#include <linux/ioport.h> #include <linux/cache.h> #include <linux/slab.h> +#include "pci.h" static void pbus_assign_resources_sorted (struct pci_bus *bus)
@@ -351,8 +352,7 @@ static int pbus_size_mem(struct pci_bus *bus, unsigned long mask, unsigned longif (r->parent || (r->flags & mask) != type)continue ; r_size =resource_size (r); -/* For bridges size != alignment */ - align =resource_alignment (r); + align =pci_resource_alignment (dev, i); order =__ffs (align) -20 ;if (order >11 ) { dev_warn(&dev->dev, "BAR %d bad alignment %llx: "
drivers/pci/setup-res.c +4 -3
@@ -133,7 +133,7 @@ int pci_assign_resource(struct pci_dev *dev, int resno) size =resource_size (res); min = (res->flags &IORESOURCE_IO ) ?PCIBIOS_MIN_IO :PCIBIOS_MIN_MEM ; - align =resource_alignment (res); + align =pci_resource_alignment (dev, resno);if (!align) { dev_err(&dev->dev, "BAR %d: can't allocate resource (bogus " "alignment) %pR flags %#lx\n",
@@ -224,7 +224,7 @@ void pdev_sort_resources(struct pci_dev *dev, struct resource_list *head)if (!(r->flags) || r->parent)continue ; - r_align =resource_alignment (r); + r_align =pci_resource_alignment (dev, i);if (!r_align) { dev_warn(&dev->dev, "BAR %d: bogus alignment " "%pR flags %#lx\n",
@@ -236,7 +236,8 @@ void pdev_sort_resources(struct pci_dev *dev, struct resource_list *head)struct resource_list *ln = list->next;if (ln) - align =resource_alignment (ln->res); + align =pci_resource_alignment (ln->dev, + ln->res - ln->dev->resource);if (r_align > align) { tmp =kmalloc (sizeof (*tmp),GFP_KERNEL );
Signature · 1 line
1.5.6.4
PATCH v6 4/16 PCI: make pci_alloc_child_bus() be able to handle NULL bridge
In reply to Yu ZhaoMake pci_alloc_child_bus() be able to allocate buses without bridge devices. Some SR-IOV devices can occupy more than one bus number, but there is no explicit bridges because that have internal routing mechanism.
--- drivers/pci/probe.c | 7 +++++-- 1 files changed, 5 insertions(+), 2 deletions(-)
drivers/pci/probe.c +5 -2
@@ -401,12 +401,10 @@ static struct pci_bus *pci_alloc_child_bus(struct pci_bus *parent,if (!child)return NULL ; - child->self = bridge; child->parent = parent; child->ops = parent->ops; child->sysdata = parent->sysdata; child->bus_flags = parent->bus_flags; - child->bridge =get_device (&bridge->dev);/* initialize some portions of the bus device, but don't register it * now as the parent is not properly set up yet. This device will get
@@ -423,6 +421,11 @@ static struct pci_bus *pci_alloc_child_bus(struct pci_bus *parent, child->primary = parent->secondary; child->subordinate =0xff ; +if (!bridge) +return child; + + child->self = bridge; + child->bridge =get_device (&bridge->dev);/* Set up default resource pointers and names.. */ for (i =0 ; i <PCI_BRIDGE_RES_NUM ; i++) { child->resource[i] = &bridge->resource[PCI_BRIDGE_RESOURCES +i];
Signature · 1 line
1.5.6.4
PATCH v6 6/16 PCI: add a new function to map BAR offset
In reply to Yu ZhaoAdd a function to map resource number to corresponding register so people can get the offset and type of device specific BARs.
--- drivers/pci/pci.c | 22 ++++++++++++++++++++++ drivers/pci/pci.h | 2 ++ drivers/pci/setup-res.c | 13 +++++-------- 3 files changed, 29 insertions(+), 8 deletions(-)
drivers/pci/pci.c +22 -0
@@ -1924,6 +1924,28 @@ int pci_resource_alignment(struct pci_dev *dev, int resno)return 0 ; } +/** +* pci_resource_bar - get position of the BAR associated with a resource +* @dev: the PCI device +* @resno: the resource number +* @type: the BAR type to be filled in +* +* Returns BAR position in config space, or 0 if the BAR is invalid. +*/ +int pci_resource_bar (struct pci_dev *dev,int resno,enum pci_bar_type *type) +{ +if (resno <PCI_ROM_RESOURCE ) { + *type = pci_bar_unknown; +return PCI_BASE_ADDRESS_0 +4 * resno; + }else if (resno ==PCI_ROM_RESOURCE ) { + *type = pci_bar_mem32; +return dev->rom_base_reg; + } + +dev_err (&dev->dev,"BAR: invalid resource #%d\n" , resno); +return 0 ; +} +static void __devinit pci_no_domains (void ) {#ifdef CONFIG_PCI_DOMAINS
drivers/pci/pci.h +2 -0
@@ -167,6 +167,8 @@ enum pci_bar_type {extern int __pci_read_base (struct pci_dev *dev,enum pci_bar_type type,struct resource *res,unsigned int reg);extern int pci_resource_alignment (struct pci_dev *dev,int resno); +extern int pci_resource_bar (struct pci_dev *dev,int resno, +enum pci_bar_type *type);extern void pci_enable_ari (struct pci_dev *dev);/** * pci_ari_enabled - query ARI forwarding status
drivers/pci/setup-res.c +5 -8
@@ -31,6 +31,7 @@ void pci_update_resource(struct pci_dev *dev, int resno)struct pci_bus_region region;u32 new, check, mask;int reg; +enum pci_bar_type type;struct resource *res = dev->resource + resno;/*
@@ -62,17 +63,13 @@ void pci_update_resource(struct pci_dev *dev, int resno)else mask = (u32 )PCI_BASE_ADDRESS_MEM_MASK ; -if (resno <6 ) { - reg =PCI_BASE_ADDRESS_0 +4 * resno; - }else if (resno ==PCI_ROM_RESOURCE ) { + reg =pci_resource_bar (dev, resno, &type); +if (!reg) +return ; +if (type != pci_bar_unknown) {if (!(res->flags &IORESOURCE_ROM_ENABLE ))return ; new |=PCI_ROM_ADDRESS_ENABLE ; - reg = dev->rom_base_reg; - }else { -/* Hmm, non-standard resource. */ - -return ;/* kill uninitialised var warning */ }pci_write_config_dword (dev, reg, new);
Signature · 1 line
1.5.6.4
PATCH v6 7/16 PCI: cleanup pcibios_allocate_resources()
In reply to Yu ZhaoThis cleanup makes pcibios_allocate_resources() easier to read.
--- arch/x86/pci/i386.c | 28 ++++++++++++++-------------- 1 files changed, 14 insertions(+), 14 deletions(-)
arch/x86/pci/i386.c +14 -14
@@ -147,7 +147,7 @@ static void __init pcibios_allocate_bus_resources(struct list_head *bus_list)static void __init pcibios_allocate_resources (int pass) {struct pci_dev *dev =NULL ; -int idx, disabled; +int idx, enabled;u16 command;struct resource *r, *pr;
@@ -160,22 +160,22 @@ static void __init pcibios_allocate_resources(int pass)if (!r->start)/* Address not assigned at all */ continue ;if (r->flags &IORESOURCE_IO ) - disabled = !(command &PCI_COMMAND_IO ); + enabled = command &PCI_COMMAND_IO ;else - disabled = !(command &PCI_COMMAND_MEMORY ); -if (pass == disabled) { -dev_dbg (&dev->dev,"resource %#08llx-%#08llx (f=%lx, d=%d, p=%d)\n" , + enabled = command &PCI_COMMAND_MEMORY ; +if (pass == enabled) +continue ; +dev_dbg (&dev->dev,"resource %#08llx-%#08llx (f=%lx, d=%d, p=%d)\n" , (unsigned long long ) r->start, (unsigned long long ) r->end, - r->flags, disabled, pass); - pr =pci_find_parent_resource (dev, r); -if (!pr ||request_resource (pr, r) <0 ) { -dev_err (&dev->dev,"BAR %d: can't allocate resource\n" , idx); -/* We'll assign a new address later */ - r->end -= r->start; - r->start =0 ; - } - } + r->flags, enabled, pass); + pr =pci_find_parent_resource (dev, r); +if (pr && !request_resource (pr, r)) +continue ; +dev_err (&dev->dev,"BAR %d: can't allocate resource\n" , idx); +/* We'll assign a new address later */ + r->end -= r->start; + r->start =0 ; }if (!pass) { r = &dev->resource[PCI_ROM_RESOURCE ];
Signature · 1 line
1.5.6.4
PATCH v6 8/16 PCI: add boot options to reassign resources
In reply to Yu ZhaoThis patch adds boot options so user can reassign device resources of all devices under a bus.
The boot options can be used as:
pci=assign-mmio=0000:01,assign-pio=0000:02
'[dddd:]bb' is the domain and bus number.
--- arch/x86/pci/common.c | 73 +++++++++++++++++++++++++++++++++++++++++++++++++ arch/x86/pci/i386.c | 10 ++++--- arch/x86/pci/pci.h | 3 ++ 3 files changed, 82 insertions(+), 4 deletions(-)
arch/x86/pci/common.c +73 -0
@@ -137,6 +137,72 @@ static void __devinit pcibios_fixup_device_resources(struct pci_dev *dev) } } +static char *pci_assign_pio; +static char *pci_assign_mmio; + +static int pcibios_bus_resource_needs_fixup (struct pci_bus *bus) +{ +int i; +int type =0 ; +int domain, busnr; + +if (!bus->self) +return 0 ; + +for (i =0 ; i <2 ; i++) { +char *str = i ? pci_assign_pio : pci_assign_mmio; + +while (str && *str) { +if (sscanf (str,"%04x:%02x" , &domain, &busnr) !=2 ) { +if (sscanf (str,"%02x" , &busnr) !=1 ) +break ; + domain =0 ; + } + +if (pci_domain_nr (bus) == domain && + bus->number == busnr) { + type |= i ?IORESOURCE_IO :IORESOURCE_MEM ; +break ; + } + + str =strchr (str,';' ); +if (str) + str++; + } + } + +return type; +} + +static void __devinit pcibios_fixup_bus_resources (struct pci_bus *bus) +{ +int i; +int type =pcibios_bus_resource_needs_fixup (bus); + +if (!type) +return ; + +for (i =0 ; i <PCI_BUS_NUM_RESOURCES ; i++) { +struct resource *res = bus->resource[i]; + +if (!res) +continue ; +if (res->flags & type) + res->flags =0 ; + } +} + +int pcibios_resource_needs_fixup (struct pci_dev *dev,int resno) +{ +struct pci_bus *bus; + +for (bus = dev->bus; bus && bus != pci_root_bus; bus = bus->parent) +if (pcibios_bus_resource_needs_fixup (bus)) +return 1 ; + +return 0 ; +} +/* * Called after each bus is probed, but before its children * are examined.
@@ -147,6 +213,7 @@ void __devinit pcibios_fixup_bus(struct pci_bus *b)struct pci_dev *dev;pci_read_bridge_bases (b); +pcibios_fixup_bus_resources (b);list_for_each_entry (dev, &b->devices, bus_list)pcibios_fixup_device_resources (dev); }
@@ -519,6 +586,12 @@ char * __devinit pcibios_setup(char *str) }else if (!strcmp (str,"skip_isa_align" )) { pci_probe |=PCI_CAN_SKIP_ISA_ALIGN ;return NULL ; + }else if (!strncmp (str,"assign-pio=" ,11 )) { + pci_assign_pio = str +11 ; +return NULL ; + }else if (!strncmp (str,"assign-mmio=" ,12 )) { + pci_assign_mmio = str +12 ; +return NULL ; }return str; }
arch/x86/pci/i386.c +6 -4
@@ -169,10 +169,12 @@ static void __init pcibios_allocate_resources(int pass) (unsigned long long ) r->start, (unsigned long long ) r->end, r->flags, enabled, pass); - pr =pci_find_parent_resource (dev, r); -if (pr && !request_resource (pr, r)) -continue ; -dev_err (&dev->dev,"BAR %d: can't allocate resource\n" , idx); +if (!pcibios_resource_needs_fixup (dev, idx)) { + pr =pci_find_parent_resource (dev, r); +if (pr && !request_resource (pr, r)) +continue ; +dev_err (&dev->dev,"BAR %d: can't allocate resource\n" , idx); + }/* We'll assign a new address later */ r->end -= r->start; r->start =0 ;
arch/x86/pci/pci.h +3 -0
@@ -117,6 +117,9 @@ extern int __init pcibios_init(void);extern int __init pci_mmcfg_arch_init (void );extern void __init pci_mmcfg_arch_free (void ); +/* pci-common.c */ +extern int pcibios_resource_needs_fixup (struct pci_dev *dev,int resno); +/* * AMD Fam10h CPUs are buggy, and cannot access MMIO config space * on their northbrige except through the * %eax register. As such, you MUST
Signature · 1 line
1.5.6.4
PATCH v6 10/16 PCI: cleanup pci_bus_add_devices()
In reply to Yu ZhaoThis cleanup makes pci_bus_add_devices() easier to read.
--- drivers/pci/bus.c | 56 +++++++++++++++++++++++++------------------------ drivers/pci/remove.c | 2 + 2 files changed, 31 insertions(+), 27 deletions(-)
drivers/pci/bus.c +29 -27
@@ -71,7 +71,7 @@ pci_bus_alloc_resource(struct pci_bus *bus, struct resource *res, }/** -* add a single device +* pci_bus_add_device - add a single device * @dev: device to add * * This adds a single pci device to the global
@@ -105,7 +105,7 @@ int pci_bus_add_device(struct pci_dev *dev)void pci_bus_add_devices (struct pci_bus *bus) {struct pci_dev *dev; -struct pci_bus *child_bus; +struct pci_bus *child;int retval;list_for_each_entry (dev, &bus->devices, bus_list) {
@@ -120,39 +120,41 @@ void pci_bus_add_devices(struct pci_bus *bus)list_for_each_entry (dev, &bus->devices, bus_list) {BUG_ON (!dev->is_added); + child = dev->subordinate;/* * If there is an unattached subordinate bus, attach * it and then scan for unattached PCI devices. */ -if (dev->subordinate) { -if (list_empty (&dev->subordinate->node)) { -down_write (&pci_bus_sem); -list_add_tail (&dev->subordinate->node, - &dev->bus->children); -up_write (&pci_bus_sem); - } -pci_bus_add_devices (dev->subordinate); - -/* register the bus with sysfs as the parent is now -* properly registered. */ - child_bus = dev->subordinate; -if (child_bus->is_added) -continue ; - child_bus->dev.parent = child_bus->bridge; - retval =device_register (&child_bus->dev); -if (retval) -dev_err (&dev->dev,"Error registering pci_bus," -" continuing...\n" ); -else { - child_bus->is_added =1 ; - retval =device_create_file (&child_bus->dev, - &dev_attr_cpuaffinity); - } +if (!child) +continue ; +if (list_empty (&child->node)) { +down_write (&pci_bus_sem); +list_add_tail (&child->node, + &dev->bus->children); +up_write (&pci_bus_sem); + } +pci_bus_add_devices (child); + +/* +* register the bus with sysfs as the parent is now +* properly registered. +*/ +if (child->is_added) +continue ; + child->dev.parent = child->bridge; + retval =device_register (&child->dev); +if (retval) +dev_err (&dev->dev,"Error registering pci_bus," +" continuing...\n" ); +else { + child->is_added =1 ; + retval =device_create_file (&child->dev, + &dev_attr_cpuaffinity);if (retval)dev_err (&dev->dev,"Error creating cpuaffinity" " file, continuing...\n" ); - retval =device_create_file (&child_bus->dev, + retval =device_create_file (&child->dev, &dev_attr_cpulistaffinity);if (retval) dev_err(&dev->dev,
drivers/pci/remove.c +2 -0
@@ -72,6 +72,8 @@ void pci_remove_bus(struct pci_bus *pci_bus)list_del (&pci_bus->node);up_write (&pci_bus_sem);pci_remove_legacy_files (pci_bus); +if (!pci_bus->is_added) +return ;device_remove_file (&pci_bus->dev, &dev_attr_cpuaffinity);device_remove_file (&pci_bus->dev, &dev_attr_cpulistaffinity);device_unregister (&pci_bus->dev);
Signature · 1 line
1.5.6.4
PATCH v6 12/16 PCI: support the SR-IOV capability
In reply to Yu ZhaoSupport Single Root I/O Virtualization (SR-IOV) capability.
--- drivers/pci/Kconfig | 12 + drivers/pci/Makefile | 2 + drivers/pci/iov.c | 592 ++++++++++++++++++++++++++++++++++++++++++++++ drivers/pci/pci-sysfs.c | 4 + drivers/pci/pci.c | 14 + drivers/pci/pci.h | 48 ++++ drivers/pci/probe.c | 4 + include/linux/pci.h | 39 +++ include/linux/pci_regs.h | 21 ++ 9 files changed, 736 insertions(+), 0 deletions(-) create mode 100644 drivers/pci/iov.c
drivers/pci/Kconfig +12 -0
@@ -50,3 +50,15 @@ config HT_IRQ This allows native hypertransport devices to use interrupts. If unsure say Y. + +config PCI_IOV + bool "PCI SR-IOV support" + depends on PCI + select PCI_MSI + default n + help + This option allows device drivers to enable Single Root I/O + Virtualization. Each Virtual Function's PCI configuration + space can be accessed using its own Bus, Device and Function + Number (Routing ID). Each Virtual Function also has PCI Memory + Space, which is used to map its own register set.
drivers/pci/Makefile +2 -0
@@ -55,3 +55,5 @@ obj-$(CONFIG_PCI_SYSCALL) += syscall.o ifeq ($(CONFIG_PCI_DEBUG),y) EXTRA_CFLAGS += -DDEBUG endif + +obj-$(CONFIG_PCI_IOV) += iov.o
drivers/pci/iov.c new file +592 -0
592 lines not on this page · raw message
drivers/pci/pci-sysfs.c +4 -0
@@ -847,6 +847,9 @@ static int pci_create_capabilities_sysfs(struct pci_dev *dev)/* Active State Power Management */ pcie_aspm_create_sysfs_dev_files (dev); +/* Single Root I/O Virtualization */ +pci_iov_create_sysfs (dev); +return 0 ; }
@@ -932,6 +935,7 @@ static void pci_remove_capabilities_sysfs(struct pci_dev *dev) }pcie_aspm_remove_sysfs_dev_files (dev); +pci_iov_remove_sysfs (dev); }/**
drivers/pci/pci.c +14 -0
@@ -1936,6 +1936,13 @@ int pci_resource_alignment(struct pci_dev *dev, int resno)if (align)return align > bios_align ? align : bios_align; +if (resno >PCI_ROM_RESOURCE && resno <PCI_BRIDGE_RESOURCES ) { +/* device specific resource */ + align =pci_iov_resource_align (dev, resno); +if (align) +return align > bios_align ? align : bios_align; + } +dev_err (&dev->dev,"alignment: invalid resource #%d\n" , resno);return 0 ; }
@@ -1950,12 +1957,19 @@ int pci_resource_alignment(struct pci_dev *dev, int resno)*/ int pci_resource_bar (struct pci_dev *dev,int resno,enum pci_bar_type *type) { +int reg; +if (resno <PCI_ROM_RESOURCE ) { *type = pci_bar_unknown;return PCI_BASE_ADDRESS_0 +4 * resno; }else if (resno ==PCI_ROM_RESOURCE ) { *type = pci_bar_mem32;return dev->rom_base_reg; + }else if (resno <PCI_BRIDGE_RESOURCES ) { +/* device specific resource */ + reg =pci_iov_resource_bar (dev, resno, type); +if (reg) +return reg; }dev_err (&dev->dev,"BAR: invalid resource #%d\n" , resno);
drivers/pci/pci.h +48 -0
52 lines not on this page · raw message
drivers/pci/probe.c +4 -0
@@ -779,6 +779,7 @@ static int pci_setup_device(struct pci_dev * dev)static void pci_release_capabilities (struct pci_dev *dev) {pci_vpd_release (dev); +pci_iov_release (dev); }/**
@@ -962,6 +963,9 @@ static void pci_init_capabilities(struct pci_dev *dev)/* Alternative Routing-ID Forwarding */ pci_enable_ari (dev); + +/* Single Root I/O Virtualization */ +pci_iov_init (dev); }void pci_device_add (struct pci_dev *dev,struct pci_bus *bus)
include/linux/pci.h +39 -0
@@ -87,6 +87,12 @@ enum {/* #6: expansion ROM */ PCI_ROM_RESOURCE, +/* device specific resources */ +#ifdef CONFIG_PCI_IOV + PCI_IOV_RESOURCES, + PCI_IOV_RESOURCES_END = PCI_IOV_RESOURCES + PCI_IOV_NUM_BAR - 1, +#endif +/* address space assigned to buses behind the bridge */ #ifndef PCI_BRIDGE_RES_NUM #define PCI_BRIDGE_RES_NUM 4
@@ -165,6 +171,7 @@ struct pci_cap_saved_state {struct pcie_link_state ;struct pci_vpd ; +struct pci_iov ;/* * The pci_dev structure is used to describe PCI devices.
@@ -253,6 +260,7 @@ struct pci_dev {struct list_head msi_list;#endif struct pci_vpd *vpd; +struct pci_iov *iov; };extern struct pci_dev *alloc_pci_dev (void );
@@ -1147,5 +1155,36 @@ static inline void * pci_ioremap_bar(struct pci_dev *pdev, int bar) }#endif +/* SR-IOV events masks */ +#define PCI_IOV_NUM_VIRTFN 0x0000FFFFU/* NumVFs to be set */ +/* SR-IOV events values */ +#define PCI_IOV_ENABLE 0x00010000U/* SR-IOV enable request */ +#define PCI_IOV_DISABLE 0x00020000U/* SR-IOV disable request */ +#define PCI_IOV_NUMVFS 0x00040000U/* SR-IOV disable request */ + +#ifdef CONFIG_PCI_IOV +extern int pci_iov_enable (struct pci_dev *dev,int numvfs); +extern void pci_iov_disable (struct pci_dev *dev); +extern int pci_iov_register (struct pci_dev *dev, +int (*callback)(struct pci_dev *dev,u32 event)); +extern void pci_iov_unregister (struct pci_dev *dev); +#else +static inline int pci_iov_enable (struct pci_dev *dev,int numvfs) +{ +return -EIO ; +} +static inline void pci_iov_disable (struct pci_dev *dev) +{ +} +static inline int pci_iov_register (struct pci_dev *dev, +int (*callback)(struct pci_dev *dev,u32 event)) +{ +return -EIO ; +} +static inline void pci_iov_unregister (struct pci_dev *dev) +{ +} +#endif /* CONFIG_PCI_IOV */ +#endif /* __KERNEL__ */ #endif /* LINUX_PCI_H */
include/linux/pci_regs.h +21 -0
@@ -363,6 +363,7 @@#define PCI_EXP_TYPE_UPSTREAM 0x5/* Upstream Port */ #define PCI_EXP_TYPE_DOWNSTREAM 0x6/* Downstream Port */ #define PCI_EXP_TYPE_PCI_BRIDGE 0x7/* PCI/PCI-X Bridge */ +#define PCI_EXP_TYPE_RC_END 0x9/* Root Complex Integrated Endpoint */ #define PCI_EXP_FLAGS_SLOT 0x0100/* Slot implemented */ #define PCI_EXP_FLAGS_IRQ 0x3e00/* Interrupt message number */ #define PCI_EXP_DEVCAP 4/* Device capabilities */
@@ -434,6 +435,7 @@#define PCI_EXT_CAP_ID_DSN 3#define PCI_EXT_CAP_ID_PWR 4#define PCI_EXT_CAP_ID_ARI 14 +#define PCI_EXT_CAP_ID_IOV 16/* Advanced Error Reporting */ #define PCI_ERR_UNCOR_STATUS 4/* Uncorrectable Error Status */
@@ -551,4 +553,23 @@#define PCI_ARI_CTRL_ACS 0x0002/* ACS Function Groups Enable */ #define PCI_ARI_CTRL_FG (x) (((x) >> 4) & 7)/* Function Group */ +/* Single Root I/O Virtualization */ +#define PCI_IOV_CAP 0x04/* SR-IOV Capabilities */ +#define PCI_IOV_CTRL 0x08/* SR-IOV Control */ +#define PCI_IOV_CTRL_VFE 0x01/* VF Enable */ +#define PCI_IOV_CTRL_MSE 0x08/* VF Memory Space Enable */ +#define PCI_IOV_CTRL_ARI 0x10/* ARI Capable Hierarchy */ +#define PCI_IOV_STATUS 0x0a/* SR-IOV Status */ +#define PCI_IOV_INITIAL_VF 0x0c/* Initial VFs */ +#define PCI_IOV_TOTAL_VF 0x0e/* Total VFs */ +#define PCI_IOV_NUM_VF 0x10/* Number of VFs */ +#define PCI_IOV_FUNC_LINK 0x12/* Function Dependency Link */ +#define PCI_IOV_VF_OFFSET 0x14/* First VF Offset */ +#define PCI_IOV_VF_STRIDE 0x16/* Following VF Stride */ +#define PCI_IOV_VF_DID 0x1a/* VF Device ID */ +#define PCI_IOV_SUP_PGSIZE 0x1c/* Supported Page Sizes */ +#define PCI_IOV_SYS_PGSIZE 0x20/* System Page Size */ +#define PCI_IOV_BAR_0 0x24/* VF BAR0 */ +#define PCI_IOV_NUM_BAR 6/* Number of VF BARs */ +#endif /* LINUX_PCI_REGS_H */
Signature · 1 line
1.5.6.4
PATCH v6 14/16 PCI: document for SR-IOV user and developer
In reply to Yu ZhaoCreate HOW-TO for SR-IOV user and driver developer.
--- Documentation/DocBook/kernel-api.tmpl | 1 + Documentation/PCI/pci-iov-howto.txt | 181 +++++++++++++++++++++++++++++++++ 2 files changed, 182 insertions(+), 0 deletions(-) create mode 100644 Documentation/PCI/pci-iov-howto.txt
Documentation/DocBook/kernel-api.tmpl +1 -0
@@ -251,6 +251,7 @@ X!Edrivers/pci/hotplug.c --> !Edrivers/pci/probe.c !Edrivers/pci/rom.c +!Edrivers/pci/iov.c </sect1> <sect1><title>PCI Hotplug Support Library</title> !Edrivers/pci/hotplug/pci_hotplug_core.c
Documentation/PCI/pci-iov-howto.txt new file +181 -0
181 lines not on this page · raw message
Signature · 1 line
1.5.6.4
PATCH v6 15/16 PCI: document the SR-IOV sysfs entries
In reply to Yu ZhaoDocument the SR-IOV sysfs entries.
--- Documentation/ABI/testing/sysfs-bus-pci | 33 +++++++++++++++++++++++++++++++ 1 files changed, 33 insertions(+), 0 deletions(-)
Documentation/ABI/testing/sysfs-bus-pci +33 -0
@@ -9,3 +9,36 @@ Description: that some devices may have malformatted data. If the underlying VPD has a writable section then the corresponding section of this file will be writable. + +What: /sys/bus/pci/devices/.../iov/enable +Date: October 2008 +Contact: Yu Zhao <[email protected]> +Description: + This file appears when a device has the SR-IOV capability. + It holds the status of the capability, and could be written + (0/1) to disable and enable the capability if the PF driver + supports this operation. + +What: /sys/bus/pci/devices/.../iov/initialvfs +Date: October 2008 +Contact: Yu Zhao <[email protected]> +Description: + This file appears when a device has the SR-IOV capability. + It holds the number of initial Virtual Functions (read-only). + +What: /sys/bus/pci/devices/.../iov/totalvfs +Date: October 2008 +Contact: Yu Zhao <[email protected]> +Description: + This file appears when a device has the SR-IOV capability. + It holds the number of total Virtual Functions (read-only). + + +What: /sys/bus/pci/devices/.../iov/numvfs +Date: October 2008 +Contact: Yu Zhao <[email protected]> +Description: + This file appears when a device has the SR-IOV capability. + It holds the number of available Virtual Functions, and + could be written (1 ~ InitialVFs) to change the number if + the PF driver supports this operation.
Signature · 1 line
1.5.6.4
Re: [PATCH 2/16 v6] PCI: define PCI resource names in an 'enum'
In reply to Yu ZhaoQuoted 3 lines · Yu Zhao
This patch moves all definitions of the PCI resource names to an 'enum', and also replaces some hard-coded resource variables with symbol names. This change eases introduction of device specific resources.
Thanks for removing a bunch of magic numbers from the code.
Quoted 15 lines
static void
pci_restore_bars(struct pci_dev *dev)
{drivers/pci/pci.c +0
-12-int i, numres; - -switch (dev->hdr_type) { -case PCI_HEADER_TYPE_NORMAL : - numres =6 ; -break ; -case PCI_HEADER_TYPE_BRIDGE : - numres =2 ; -break ; -case PCI_HEADER_TYPE_CARDBUS : - numres =1 ; -break ;
drivers/pci/pci.c pci_find_parent_resource(const struct pci_dev *dev, struct resource *res) · line 366
Quoted 10 lines
drivers/pci/pci.c +2
-5-default : -/* Should never get here, but just in case... */ -return ; - } +int i; -for (i =0 ; i < numres; i++) +for (i =0 ; i <PCI_BRIDGE_RESOURCES ; i++)pci_update_resource (dev, i); }
The behavior of this function used to depend on dev->hdr_type. Now we don't look at hdr_type at all, so we do the same thing for all devices.
For example, for a CardBus device, we used to call pci_update_resource() only for BAR 0; now we call it for BARs 0-6.
Maybe this is safe, but I can't tell from the patch, so I think you should explain *why* it's safe in the changelog.
Quoted 7 lines
include/linux/pci.h +7
-0+/* +* For PCI devices, the region numbers are assigned this way: +*/ +enum { +/* #0-5: standard PCI regions */ +PCI_STD_RESOURCES , +PCI_STD_RESOURCES_END =5 ,
include/linux/pci.h enum pci_mmap_state { · line 95
Quoted 10 lines
include/linux/pci.h +10
-0+ +/* #6: expansion ROM */ + PCI_ROM_RESOURCE, + +/* address space assigned to buses behind the bridge */ +#ifndef PCI_BRIDGE_RES_NUM +#define PCI_BRIDGE_RES_NUM 4 +#endif +PCI_BRIDGE_RESOURCES , + PCI_BRIDGE_RES_END = PCI_BRIDGE_RESOURCES + PCI_BRIDGE_RES_NUM - 1,
Since you used "PCI_STD_RESOURCES_END" above, maybe you should use "PCI_BRIDGE_RESOURCES_END" instead of "PCI_BRIDGE_RES_END".
Bjorn
Re: [PATCH 2/16 v6] PCI: define PCI resource names in an 'enum'
In reply to Yu ZhaoQuoted 3 lines · Yu Zhao
This patch moves all definitions of the PCI resource names to an 'enum', and also replaces some hard-coded resource variables with symbol names. This change eases introduction of device specific resources.
Thanks for removing a bunch of magic numbers from the code.
Quoted 15 lines
static void
pci_restore_bars(struct pci_dev *dev)
{drivers/pci/pci.c +0
-12-int i, numres; - -switch (dev->hdr_type) { -case PCI_HEADER_TYPE_NORMAL : - numres =6 ; -break ; -case PCI_HEADER_TYPE_BRIDGE : - numres =2 ; -break ; -case PCI_HEADER_TYPE_CARDBUS : - numres =1 ; -break ;
drivers/pci/pci.c pci_find_parent_resource(const struct pci_dev *dev, struct resource *res) · line 366
Quoted 10 lines
drivers/pci/pci.c +2
-5-default : -/* Should never get here, but just in case... */ -return ; - } +int i; -for (i =0 ; i < numres; i++) +for (i =0 ; i <PCI_BRIDGE_RESOURCES ; i++)pci_update_resource (dev, i); }
The behavior of this function used to depend on dev->hdr_type. Now we don't look at hdr_type at all, so we do the same thing for all devices.
For example, for a CardBus device, we used to call pci_update_resource() only for BAR 0; now we call it for BARs 0-6.
Maybe this is safe, but I can't tell from the patch, so I think you should explain *why* it's safe in the changelog.
Quoted 7 lines
include/linux/pci.h +7
-0+/* +* For PCI devices, the region numbers are assigned this way: +*/ +enum { +/* #0-5: standard PCI regions */ +PCI_STD_RESOURCES , +PCI_STD_RESOURCES_END =5 ,
include/linux/pci.h enum pci_mmap_state { · line 95
Quoted 10 lines
include/linux/pci.h +10
-0+ +/* #6: expansion ROM */ + PCI_ROM_RESOURCE, + +/* address space assigned to buses behind the bridge */ +#ifndef PCI_BRIDGE_RES_NUM +#define PCI_BRIDGE_RES_NUM 4 +#endif +PCI_BRIDGE_RESOURCES , + PCI_BRIDGE_RES_END = PCI_BRIDGE_RESOURCES + PCI_BRIDGE_RES_NUM - 1,
Since you used "PCI_STD_RESOURCES_END" above, maybe you should use "PCI_BRIDGE_RESOURCES_END" instead of "PCI_BRIDGE_RES_END".
Bjorn
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
In reply to Yu ZhaoQuoted 27 lines · Yu Zhao
Document the new PCI[x86] boot parameters.
Cc Alex Chiang <[email protected]>Cc Grant Grundler <[email protected]>Cc Greg KH <[email protected]>Cc Ingo Molnar <[email protected]>Cc Jesse Barnes <[email protected]>Cc Matthew Wilcox <[email protected]>Cc Randy Dunlap <[email protected]>Cc Roland Dreier <[email protected]>Signed-off-by Yu Zhao <[email protected]>--- Documentation/kernel-parameters.txt | 10 ++++++++++ 1 files changed, 10 insertions(+), 0 deletions(-)Documentation/kernel-parameters.txt +3
-0@@ -1677,6 +1677,16 @@ and is between 256 and 4096 characters. It is defined in the file cbmemsize=nn[KMG] The fixed amount of bus space which is reserved for the CardBus bridge's memory window. The default value is 64 megabytes. + assign-mmio=[dddd:]bb [X86] reassign memory resources of all + devices under bus [dddd:]bb (dddd is the domain + number and bb is the bus number).
Documentation/kernel-parameters.txt and is between 256 and 4096 characters. It is defined in the file · line 1692
Quoted 10 lines
Documentation/kernel-parameters.txt +7
-0+ assign-pio=[dddd:]bb [X86] reassign io port resources of all + devices under bus [dddd:]bb (dddd is the domain + number and bb is the bus number). + align-mmio=[dddd:]bb:dd.f [X86] relocate memory resources of a + device to minimum PAGE_SIZE alignment (dddd is + the domain number and bb, dd and f is the bus, + device and function number). pcie_aspm= [PCIE] Forcibly enable or disable PCIe Active State Power Management.
I think it's nicer to have the documentation change included in the patch that implements the change. For example, I think this and patch 9/16 "add boot option to align ..." should be folded into a single patch. And similarly for the other documentation patches.
Bjorn
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
In reply to Yu ZhaoQuoted 27 lines · Yu Zhao
Document the new PCI[x86] boot parameters.
Cc Alex Chiang <[email protected]>Cc Grant Grundler <[email protected]>Cc Greg KH <[email protected]>Cc Ingo Molnar <[email protected]>Cc Jesse Barnes <[email protected]>Cc Matthew Wilcox <[email protected]>Cc Randy Dunlap <[email protected]>Cc Roland Dreier <[email protected]>Signed-off-by Yu Zhao <[email protected]>--- Documentation/kernel-parameters.txt | 10 ++++++++++ 1 files changed, 10 insertions(+), 0 deletions(-)Documentation/kernel-parameters.txt +3
-0@@ -1677,6 +1677,16 @@ and is between 256 and 4096 characters. It is defined in the file cbmemsize=nn[KMG] The fixed amount of bus space which is reserved for the CardBus bridge's memory window. The default value is 64 megabytes. + assign-mmio=[dddd:]bb [X86] reassign memory resources of all + devices under bus [dddd:]bb (dddd is the domain + number and bb is the bus number).
Documentation/kernel-parameters.txt and is between 256 and 4096 characters. It is defined in the file · line 1692
Quoted 10 lines
Documentation/kernel-parameters.txt +7
-0+ assign-pio=[dddd:]bb [X86] reassign io port resources of all + devices under bus [dddd:]bb (dddd is the domain + number and bb is the bus number). + align-mmio=[dddd:]bb:dd.f [X86] relocate memory resources of a + device to minimum PAGE_SIZE alignment (dddd is + the domain number and bb, dd and f is the bus, + device and function number). pcie_aspm= [PCIE] Forcibly enable or disable PCIe Active State Power Management.
I think it's nicer to have the documentation change included in the patch that implements the change. For example, I think this and patch 9/16 "add boot option to align ..." should be folded into a single patch. And similarly for the other documentation patches.
Bjorn
Re: [PATCH 9/16 v6] PCI: add boot option to align MMIO resources
In reply to Yu ZhaoQuoted 3 lines · Yu Zhao
This patch adds boot option to align MMIO resource for a device. The alignment is a bigger value between the
PAGE_SIZEand the resource size.
It looks like this forces alignment on PAGE_SIZE, not "a bigger value between the PAGE_SIZE and the resource size." Can you clarify the changelog to specify exactly what alignment this option forces?
Quoted 4 lines
The boot option can be used as:
pci=align-mmio=0000:01:02.3'[0000:]01:02.3' is the domain, bus, device and function number of the device.
I think you also support using multiple "align-mmio=DDDD:BB:dd.f" options separated by ";", but I had to read the code to figure that out. Can you give an example of this in the changelog and the kernel-parameters.txt patch?
Bjorn
Quoted 139 lines
Cc Alex Chiang <[email protected]>Cc Grant Grundler <[email protected]>Cc Greg KH <[email protected]>Cc Ingo Molnar <[email protected]>Cc Jesse Barnes <[email protected]>Cc Matthew Wilcox <[email protected]>Cc Randy Dunlap <[email protected]>Cc Roland Dreier <[email protected]>Signed-off-by Yu Zhao <[email protected]>--- arch/x86/pci/common.c | 37 +++++++++++++++++++++++++++++++++++++ drivers/pci/pci.c | 20 ++++++++++++++++++-- include/linux/pci.h | 1 + 3 files changed, 56 insertions(+), 2 deletions(-)arch/x86/pci/common.c +37
-0@@ -139,6 +139,7 @@ static void __devinit pcibios_fixup_device_resources(struct pci_dev *dev)static char *pci_assign_pio;static char *pci_assign_mmio; +static char *pci_align_mmio;static int pcibios_bus_resource_needs_fixup (struct pci_bus *bus) {@@ -192,6 +193,36 @@ static void __devinit pcibios_fixup_bus_resources(struct pci_bus *bus) } } +int pcibios_resource_alignment (struct pci_dev *dev,int resno) +{ +int domain, busnr, slot, func; +char *str = pci_align_mmio; + +if (dev->resource[resno].flags &IORESOURCE_IO ) +return 0 ; + +while (str && *str) { +if (sscanf (str,"%04x:%02x:%02x.%d" , + &domain, &busnr, &slot, &func) !=4 ) { +if (sscanf (str,"%02x:%02x.%d" , + &busnr, &slot, &func) !=3 ) +break ; + domain =0 ; + } + +if (pci_domain_nr (dev->bus) == domain && + dev->bus->number == busnr && + dev->devfn ==PCI_DEVFN (slot, func)) +return PAGE_SIZE ; + + str =strchr (str,';' ); +if (str) + str++; + } + +return 0 ; +} +int pcibios_resource_needs_fixup (struct pci_dev *dev,int resno) {struct pci_bus *bus;@@ -200,6 +231,9 @@ int pcibios_resource_needs_fixup(struct pci_dev *dev, int resno)if (pcibios_bus_resource_needs_fixup (bus))return 1 ; +if (pcibios_resource_alignment (dev, resno)) +return 1 ; +return 0 ; }@@ -592,6 +626,9 @@ char * __devinit pcibios_setup(char *str) }else if (!strncmp (str,"assign-mmio=" ,12 )) { pci_assign_mmio = str +12 ;return NULL ; + }else if (!strncmp (str,"align-mmio=" ,11 )) { + pci_align_mmio = str +11 ; +return NULL ; }return str; }drivers/pci/pci.c +18
-2@@ -1015,6 +1015,20 @@ int __attribute__ ((weak)) pcibios_set_pcie_reset_state(struct pci_dev *dev, }/** +* pcibios_resource_alignment - get resource alignment requirement +* @dev: the PCI device +* @resno: resource number +* +* Queries the resource alignment from PCI low level code. Returns positive +* if there is alignment requirement of the resource, or 0 otherwise. +*/ +int __attribute__ ((weak))pcibios_resource_alignment (struct pci_dev *dev, +int resno) +{ +return 0 ; +} + +/** * pci_set_pcie_reset_state - set reset state for device dev * @dev: the PCI-E device reset * @state: Reset state to enter into @@ -1913,12 +1927,14 @@ int pci_select_bars(struct pci_dev *dev, unsigned long flags)*/ int pci_resource_alignment (struct pci_dev *dev,int resno) { -resource_size_t align; +resource_size_t align, bios_align;struct resource *res = dev->resource + resno; + bios_align =pcibios_resource_alignment (dev, resno); + align =resource_alignment (res);if (align) -return align; +return align > bios_align ? align : bios_align;dev_err (&dev->dev,"alignment: invalid resource #%d\n" , resno);return 0 ;include/linux/pci.h +1
-0@@ -1121,6 +1121,7 @@ int pcibios_add_platform_entries(struct pci_dev *dev);void pcibios_disable_device (struct pci_dev *dev);int pcibios_set_pcie_reset_state (struct pci_dev *dev,enum pcie_reset_state state); +int pcibios_resource_alignment (struct pci_dev *dev,int resno);#ifdef CONFIG_PCI_MMCONFIG extern void __init pci_mmcfg_early_init (void );
Re: [PATCH 9/16 v6] PCI: add boot option to align MMIO resources
In reply to Yu ZhaoQuoted 3 lines · Yu Zhao
This patch adds boot option to align MMIO resource for a device. The alignment is a bigger value between the
PAGE_SIZEand the resource size.
It looks like this forces alignment on PAGE_SIZE, not "a bigger value between the PAGE_SIZE and the resource size." Can you clarify the changelog to specify exactly what alignment this option forces?
Quoted 4 lines
The boot option can be used as:
pci=align-mmio=0000:01:02.3'[0000:]01:02.3' is the domain, bus, device and function number of the device.
I think you also support using multiple "align-mmio=DDDD:BB:dd.f" options separated by ";", but I had to read the code to figure that out. Can you give an example of this in the changelog and the kernel-parameters.txt patch?
Bjorn
Quoted 139 lines
Cc Alex Chiang <[email protected]>Cc Grant Grundler <[email protected]>Cc Greg KH <[email protected]>Cc Ingo Molnar <[email protected]>Cc Jesse Barnes <[email protected]>Cc Matthew Wilcox <[email protected]>Cc Randy Dunlap <[email protected]>Cc Roland Dreier <[email protected]>Signed-off-by Yu Zhao <[email protected]>--- arch/x86/pci/common.c | 37 +++++++++++++++++++++++++++++++++++++ drivers/pci/pci.c | 20 ++++++++++++++++++-- include/linux/pci.h | 1 + 3 files changed, 56 insertions(+), 2 deletions(-)arch/x86/pci/common.c +37
-0@@ -139,6 +139,7 @@ static void __devinit pcibios_fixup_device_resources(struct pci_dev *dev)static char *pci_assign_pio;static char *pci_assign_mmio; +static char *pci_align_mmio;static int pcibios_bus_resource_needs_fixup (struct pci_bus *bus) {@@ -192,6 +193,36 @@ static void __devinit pcibios_fixup_bus_resources(struct pci_bus *bus) } } +int pcibios_resource_alignment (struct pci_dev *dev,int resno) +{ +int domain, busnr, slot, func; +char *str = pci_align_mmio; + +if (dev->resource[resno].flags &IORESOURCE_IO ) +return 0 ; + +while (str && *str) { +if (sscanf (str,"%04x:%02x:%02x.%d" , + &domain, &busnr, &slot, &func) !=4 ) { +if (sscanf (str,"%02x:%02x.%d" , + &busnr, &slot, &func) !=3 ) +break ; + domain =0 ; + } + +if (pci_domain_nr (dev->bus) == domain && + dev->bus->number == busnr && + dev->devfn ==PCI_DEVFN (slot, func)) +return PAGE_SIZE ; + + str =strchr (str,';' ); +if (str) + str++; + } + +return 0 ; +} +int pcibios_resource_needs_fixup (struct pci_dev *dev,int resno) {struct pci_bus *bus;@@ -200,6 +231,9 @@ int pcibios_resource_needs_fixup(struct pci_dev *dev, int resno)if (pcibios_bus_resource_needs_fixup (bus))return 1 ; +if (pcibios_resource_alignment (dev, resno)) +return 1 ; +return 0 ; }@@ -592,6 +626,9 @@ char * __devinit pcibios_setup(char *str) }else if (!strncmp (str,"assign-mmio=" ,12 )) { pci_assign_mmio = str +12 ;return NULL ; + }else if (!strncmp (str,"align-mmio=" ,11 )) { + pci_align_mmio = str +11 ; +return NULL ; }return str; }drivers/pci/pci.c +18
-2@@ -1015,6 +1015,20 @@ int __attribute__ ((weak)) pcibios_set_pcie_reset_state(struct pci_dev *dev, }/** +* pcibios_resource_alignment - get resource alignment requirement +* @dev: the PCI device +* @resno: resource number +* +* Queries the resource alignment from PCI low level code. Returns positive +* if there is alignment requirement of the resource, or 0 otherwise. +*/ +int __attribute__ ((weak))pcibios_resource_alignment (struct pci_dev *dev, +int resno) +{ +return 0 ; +} + +/** * pci_set_pcie_reset_state - set reset state for device dev * @dev: the PCI-E device reset * @state: Reset state to enter into @@ -1913,12 +1927,14 @@ int pci_select_bars(struct pci_dev *dev, unsigned long flags)*/ int pci_resource_alignment (struct pci_dev *dev,int resno) { -resource_size_t align; +resource_size_t align, bios_align;struct resource *res = dev->resource + resno; + bios_align =pcibios_resource_alignment (dev, resno); + align =resource_alignment (res);if (align) -return align; +return align > bios_align ? align : bios_align;dev_err (&dev->dev,"alignment: invalid resource #%d\n" , resno);return 0 ;include/linux/pci.h +1
-0@@ -1121,6 +1121,7 @@ int pcibios_add_platform_entries(struct pci_dev *dev);void pcibios_disable_device (struct pci_dev *dev);int pcibios_set_pcie_reset_state (struct pci_dev *dev,enum pcie_reset_state state); +int pcibios_resource_alignment (struct pci_dev *dev,int resno);#ifdef CONFIG_PCI_MMCONFIG extern void __init pci_mmcfg_early_init (void );
Re: [PATCH 8/16 v6] PCI: add boot options to reassign resources
In reply to Yu ZhaoQuoted 6 lines · Yu Zhao
This patch adds boot options so user can reassign device resources of all devices under a bus.
The boot options can be used as:
pci=assign-mmio=0000:01,assign-pio=0000:02'[dddd:]bb' is the domain and bus number.
I think this example is incorrect because you look for ";" to separate options, not ",".
Bjorn
Quoted 149 lines
Cc Alex Chiang <[email protected]>Cc Grant Grundler <[email protected]>Cc Greg KH <[email protected]>Cc Ingo Molnar <[email protected]>Cc Jesse Barnes <[email protected]>Cc Matthew Wilcox <[email protected]>Cc Randy Dunlap <[email protected]>Cc Roland Dreier <[email protected]>Signed-off-by Yu Zhao <[email protected]>--- arch/x86/pci/common.c | 73 +++++++++++++++++++++++++++++++++++++++++++++++++ arch/x86/pci/i386.c | 10 ++++--- arch/x86/pci/pci.h | 3 ++ 3 files changed, 82 insertions(+), 4 deletions(-)arch/x86/pci/common.c +73
-0@@ -137,6 +137,72 @@ static void __devinit pcibios_fixup_device_resources(struct pci_dev *dev) } } +static char *pci_assign_pio; +static char *pci_assign_mmio; + +static int pcibios_bus_resource_needs_fixup (struct pci_bus *bus) +{ +int i; +int type =0 ; +int domain, busnr; + +if (!bus->self) +return 0 ; + +for (i =0 ; i <2 ; i++) { +char *str = i ? pci_assign_pio : pci_assign_mmio; + +while (str && *str) { +if (sscanf (str,"%04x:%02x" , &domain, &busnr) !=2 ) { +if (sscanf (str,"%02x" , &busnr) !=1 ) +break ; + domain =0 ; + } + +if (pci_domain_nr (bus) == domain && + bus->number == busnr) { + type |= i ?IORESOURCE_IO :IORESOURCE_MEM ; +break ; + } + + str =strchr (str,';' ); +if (str) + str++; + } + } + +return type; +} + +static void __devinit pcibios_fixup_bus_resources (struct pci_bus *bus) +{ +int i; +int type =pcibios_bus_resource_needs_fixup (bus); + +if (!type) +return ; + +for (i =0 ; i <PCI_BUS_NUM_RESOURCES ; i++) { +struct resource *res = bus->resource[i]; + +if (!res) +continue ; +if (res->flags & type) + res->flags =0 ; + } +} + +int pcibios_resource_needs_fixup (struct pci_dev *dev,int resno) +{ +struct pci_bus *bus; + +for (bus = dev->bus; bus && bus != pci_root_bus; bus = bus->parent) +if (pcibios_bus_resource_needs_fixup (bus)) +return 1 ; + +return 0 ; +} +/* * Called after each bus is probed, but before its children * are examined. @@ -147,6 +213,7 @@ void __devinit pcibios_fixup_bus(struct pci_bus *b)struct pci_dev *dev;pci_read_bridge_bases (b); +pcibios_fixup_bus_resources (b);list_for_each_entry (dev, &b->devices, bus_list)pcibios_fixup_device_resources (dev); }@@ -519,6 +586,12 @@ char * __devinit pcibios_setup(char *str) }else if (!strcmp (str,"skip_isa_align" )) { pci_probe |=PCI_CAN_SKIP_ISA_ALIGN ;return NULL ; + }else if (!strncmp (str,"assign-pio=" ,11 )) { + pci_assign_pio = str +11 ; +return NULL ; + }else if (!strncmp (str,"assign-mmio=" ,12 )) { + pci_assign_mmio = str +12 ; +return NULL ; }return str; }arch/x86/pci/i386.c +6
-4@@ -169,10 +169,12 @@ static void __init pcibios_allocate_resources(int pass) (unsigned long long ) r->start, (unsigned long long ) r->end, r->flags, enabled, pass); - pr =pci_find_parent_resource (dev, r); -if (pr && !request_resource (pr, r)) -continue ; -dev_err (&dev->dev,"BAR %d: can't allocate resource\n" , idx); +if (!pcibios_resource_needs_fixup (dev, idx)) { + pr =pci_find_parent_resource (dev, r); +if (pr && !request_resource (pr, r)) +continue ; +dev_err (&dev->dev,"BAR %d: can't allocate resource\n" , idx); + }/* We'll assign a new address later */ r->end -= r->start; r->start =0 ;arch/x86/pci/pci.h +3
-0@@ -117,6 +117,9 @@ extern int __init pcibios_init(void);extern int __init pci_mmcfg_arch_init (void );extern void __init pci_mmcfg_arch_free (void ); +/* pci-common.c */ +extern int pcibios_resource_needs_fixup (struct pci_dev *dev,int resno); +/* * AMD Fam10h CPUs are buggy, and cannot access MMIO config space * on their northbrige except through the * %eax register. As such, you MUST
Re: [PATCH 8/16 v6] PCI: add boot options to reassign resources
In reply to Yu ZhaoQuoted 6 lines · Yu Zhao
This patch adds boot options so user can reassign device resources of all devices under a bus.
The boot options can be used as:
pci=assign-mmio=0000:01,assign-pio=0000:02'[dddd:]bb' is the domain and bus number.
I think this example is incorrect because you look for ";" to separate options, not ",".
Bjorn
Quoted 149 lines
Cc Alex Chiang <[email protected]>Cc Grant Grundler <[email protected]>Cc Greg KH <[email protected]>Cc Ingo Molnar <[email protected]>Cc Jesse Barnes <[email protected]>Cc Matthew Wilcox <[email protected]>Cc Randy Dunlap <[email protected]>Cc Roland Dreier <[email protected]>Signed-off-by Yu Zhao <[email protected]>--- arch/x86/pci/common.c | 73 +++++++++++++++++++++++++++++++++++++++++++++++++ arch/x86/pci/i386.c | 10 ++++--- arch/x86/pci/pci.h | 3 ++ 3 files changed, 82 insertions(+), 4 deletions(-)arch/x86/pci/common.c +73
-0@@ -137,6 +137,72 @@ static void __devinit pcibios_fixup_device_resources(struct pci_dev *dev) } } +static char *pci_assign_pio; +static char *pci_assign_mmio; + +static int pcibios_bus_resource_needs_fixup (struct pci_bus *bus) +{ +int i; +int type =0 ; +int domain, busnr; + +if (!bus->self) +return 0 ; + +for (i =0 ; i <2 ; i++) { +char *str = i ? pci_assign_pio : pci_assign_mmio; + +while (str && *str) { +if (sscanf (str,"%04x:%02x" , &domain, &busnr) !=2 ) { +if (sscanf (str,"%02x" , &busnr) !=1 ) +break ; + domain =0 ; + } + +if (pci_domain_nr (bus) == domain && + bus->number == busnr) { + type |= i ?IORESOURCE_IO :IORESOURCE_MEM ; +break ; + } + + str =strchr (str,';' ); +if (str) + str++; + } + } + +return type; +} + +static void __devinit pcibios_fixup_bus_resources (struct pci_bus *bus) +{ +int i; +int type =pcibios_bus_resource_needs_fixup (bus); + +if (!type) +return ; + +for (i =0 ; i <PCI_BUS_NUM_RESOURCES ; i++) { +struct resource *res = bus->resource[i]; + +if (!res) +continue ; +if (res->flags & type) + res->flags =0 ; + } +} + +int pcibios_resource_needs_fixup (struct pci_dev *dev,int resno) +{ +struct pci_bus *bus; + +for (bus = dev->bus; bus && bus != pci_root_bus; bus = bus->parent) +if (pcibios_bus_resource_needs_fixup (bus)) +return 1 ; + +return 0 ; +} +/* * Called after each bus is probed, but before its children * are examined. @@ -147,6 +213,7 @@ void __devinit pcibios_fixup_bus(struct pci_bus *b)struct pci_dev *dev;pci_read_bridge_bases (b); +pcibios_fixup_bus_resources (b);list_for_each_entry (dev, &b->devices, bus_list)pcibios_fixup_device_resources (dev); }@@ -519,6 +586,12 @@ char * __devinit pcibios_setup(char *str) }else if (!strcmp (str,"skip_isa_align" )) { pci_probe |=PCI_CAN_SKIP_ISA_ALIGN ;return NULL ; + }else if (!strncmp (str,"assign-pio=" ,11 )) { + pci_assign_pio = str +11 ; +return NULL ; + }else if (!strncmp (str,"assign-mmio=" ,12 )) { + pci_assign_mmio = str +12 ; +return NULL ; }return str; }arch/x86/pci/i386.c +6
-4@@ -169,10 +169,12 @@ static void __init pcibios_allocate_resources(int pass) (unsigned long long ) r->start, (unsigned long long ) r->end, r->flags, enabled, pass); - pr =pci_find_parent_resource (dev, r); -if (pr && !request_resource (pr, r)) -continue ; -dev_err (&dev->dev,"BAR %d: can't allocate resource\n" , idx); +if (!pcibios_resource_needs_fixup (dev, idx)) { + pr =pci_find_parent_resource (dev, r); +if (pr && !request_resource (pr, r)) +continue ; +dev_err (&dev->dev,"BAR %d: can't allocate resource\n" , idx); + }/* We'll assign a new address later */ r->end -= r->start; r->start =0 ;arch/x86/pci/pci.h +3
-0@@ -117,6 +117,9 @@ extern int __init pcibios_init(void);extern int __init pci_mmcfg_arch_init (void );extern void __init pci_mmcfg_arch_free (void ); +/* pci-common.c */ +extern int pcibios_resource_needs_fixup (struct pci_dev *dev,int resno); +/* * AMD Fam10h CPUs are buggy, and cannot access MMIO config space * on their northbrige except through the * %eax register. As such, you MUST
Re: [PATCH 2/16 v6] PCI: define PCI resource names in an 'enum'
In reply to Bjorn HelgaasQuoted 37 lines · Bjorn Helgaas
Quoted 3 lines · Yu Zhao
This patch moves all definitions of the PCI resource names to an 'enum', and also replaces some hard-coded resource variables with symbol names. This change eases introduction of device specific resources.
Thanks for removing a bunch of magic numbers from the code.
Quoted 15 lines
static void
pci_restore_bars(struct pci_dev *dev)
{drivers/pci/pci.c +0
-12-int i, numres; - -switch (dev->hdr_type) { -case PCI_HEADER_TYPE_NORMAL : - numres =6 ; -break ; -case PCI_HEADER_TYPE_BRIDGE : - numres =2 ; -break ; -case PCI_HEADER_TYPE_CARDBUS : - numres =1 ; -break ;drivers/pci/pci.c pci_find_parent_resource(const struct pci_dev *dev, struct resource *res) · line 366
Quoted 10 lines
drivers/pci/pci.c +2
-5-default : -/* Should never get here, but just in case... */ -return ; - } +int i; -for (i =0 ; i < numres; i++) +for (i =0 ; i <PCI_BRIDGE_RESOURCES ; i++)pci_update_resource (dev, i); }The behavior of this function used to depend on
dev->hdr_type. Now we don't look athdr_typeat all, so we do the same thing for all devices.
Quoted 5 lines
For example, for a CardBus device, we used to call
pci_update_resource()only for BAR 0; now we call it for BARs 0-6.Maybe this is safe, but I can't tell from the patch, so I think you should explain *why* it's safe in the changelog.
It's safe because pci_update_resource() will ignore unused resources. E.g., for a Cardbus, only BAR 0 is used and its 'flags' is set, then pci_update_resource() only updates it. BAR 1-6 are ignored since their 'flags' are 0.
I'll put more explanation in the changelog.
Thanks,
Yu
Re: [PATCH 8/16 v6] PCI: add boot options to reassign resources
In reply to Bjorn HelgaasQuoted 10 lines · Bjorn Helgaas
Quoted 6 lines · Yu Zhao
This patch adds boot options so user can reassign device resources of all devices under a bus.
The boot options can be used as:
pci=assign-mmio=0000:01,assign-pio=0000:02'[dddd:]bb' is the domain and bus number.
I think this example is incorrect because you look for ";" to separate options, not ",".
The semicolon is used to separate multiple parameters for assign-mmio and assign-pio. E.g., 'pci=assign-mmio=0000:01;0001:02;0004:03'. And the comma separates different parameters for 'pci='.
Re: [PATCH 8/16 v6] PCI: add boot options to reassign resources
In reply to Bjorn HelgaasQuoted 10 lines · Bjorn Helgaas
Quoted 6 lines · Yu Zhao
This patch adds boot options so user can reassign device resources of all devices under a bus.
The boot options can be used as:
pci=assign-mmio=0000:01,assign-pio=0000:02'[dddd:]bb' is the domain and bus number.
I think this example is incorrect because you look for ";" to separate options, not ",".
The semicolon is used to separate multiple parameters for assign-mmio and assign-pio. E.g., 'pci=assign-mmio=0000:01;0001:02;0004:03'. And the comma separates different parameters for 'pci='.
Re: [PATCH 2/16 v6] PCI: define PCI resource names in an 'enum'
In reply to Yu ZhaoQuoted 44 lines · Yu Zhao
Quoted 37 lines · Bjorn Helgaas
Quoted 3 lines · Yu Zhao
This patch moves all definitions of the PCI resource names to an 'enum', and also replaces some hard-coded resource variables with symbol names. This change eases introduction of device specific resources.
Thanks for removing a bunch of magic numbers from the code.
Quoted 15 lines
static void
pci_restore_bars(struct pci_dev *dev)
{drivers/pci/pci.c +0
-12-int i, numres; - -switch (dev->hdr_type) { -case PCI_HEADER_TYPE_NORMAL : - numres =6 ; -break ; -case PCI_HEADER_TYPE_BRIDGE : - numres =2 ; -break ; -case PCI_HEADER_TYPE_CARDBUS : - numres =1 ; -break ;drivers/pci/pci.c pci_find_parent_resource(const struct pci_dev *dev, struct resource *res) · line 366
Quoted 10 lines
drivers/pci/pci.c +2
-5-default : -/* Should never get here, but just in case... */ -return ; - } +int i; -for (i =0 ; i < numres; i++) +for (i =0 ; i <PCI_BRIDGE_RESOURCES ; i++)pci_update_resource (dev, i); }The behavior of this function used to depend on
dev->hdr_type. Now we don't look athdr_typeat all, so we do the same thing for all devices.Quoted 5 lines
For example, for a CardBus device, we used to call
pci_update_resource()only for BAR 0; now we call it for BARs 0-6.Maybe this is safe, but I can't tell from the patch, so I think you should explain *why* it's safe in the changelog.
Quoted 6 lines
It's safe because
pci_update_resource()will ignore unused resources. E.g., for a Cardbus, only BAR 0 is used and its 'flags' is set, thenpci_update_resource()only updates it. BAR 1-6 are ignored since their 'flags' are 0.I'll put more explanation in the changelog.
This is a logically separate change from merely substituting enum names for magic numbers, so you might even consider splitting it into a separate patch. Better bisection and all that, you know :-)
Bjorn
Re: [PATCH 2/16 v6] PCI: define PCI resource names in an 'enum'
In reply to Yu ZhaoQuoted 44 lines · Yu Zhao
Quoted 37 lines · Bjorn Helgaas
Quoted 3 lines · Yu Zhao
This patch moves all definitions of the PCI resource names to an 'enum', and also replaces some hard-coded resource variables with symbol names. This change eases introduction of device specific resources.
Thanks for removing a bunch of magic numbers from the code.
Quoted 15 lines
static void
pci_restore_bars(struct pci_dev *dev)
{drivers/pci/pci.c +0
-12-int i, numres; - -switch (dev->hdr_type) { -case PCI_HEADER_TYPE_NORMAL : - numres =6 ; -break ; -case PCI_HEADER_TYPE_BRIDGE : - numres =2 ; -break ; -case PCI_HEADER_TYPE_CARDBUS : - numres =1 ; -break ;drivers/pci/pci.c pci_find_parent_resource(const struct pci_dev *dev, struct resource *res) · line 366
Quoted 10 lines
drivers/pci/pci.c +2
-5-default : -/* Should never get here, but just in case... */ -return ; - } +int i; -for (i =0 ; i < numres; i++) +for (i =0 ; i <PCI_BRIDGE_RESOURCES ; i++)pci_update_resource (dev, i); }The behavior of this function used to depend on
dev->hdr_type. Now we don't look athdr_typeat all, so we do the same thing for all devices.Quoted 5 lines
For example, for a CardBus device, we used to call
pci_update_resource()only for BAR 0; now we call it for BARs 0-6.Maybe this is safe, but I can't tell from the patch, so I think you should explain *why* it's safe in the changelog.
Quoted 6 lines
It's safe because
pci_update_resource()will ignore unused resources. E.g., for a Cardbus, only BAR 0 is used and its 'flags' is set, thenpci_update_resource()only updates it. BAR 1-6 are ignored since their 'flags' are 0.I'll put more explanation in the changelog.
This is a logically separate change from merely substituting enum names for magic numbers, so you might even consider splitting it into a separate patch. Better bisection and all that, you know :-)
Bjorn
Re: [PATCH 9/16 v6] PCI: add boot option to align MMIO resources
In reply to Bjorn HelgaasQuoted 9 lines · Bjorn Helgaas
Quoted 3 lines · Yu Zhao
This patch adds boot option to align MMIO resource for a device. The alignment is a bigger value between the
PAGE_SIZEand the resource size.It looks like this forces alignment on
PAGE_SIZE, not "a bigger value between thePAGE_SIZEand the resource size." Can you clarify the changelog to specify exactly what alignment this option forces?
I guess following would explain your question.
Quoted 15 lines
Quoted 15 lines
int
pci_resource_alignment(struct pci_dev *dev, int resno){drivers/pci/pci.c +4
-2-resource_size_t align; +resource_size_t align, bios_align;struct resource *res = dev->resource + resno; + bios_align =pcibios_resource_alignment (dev, resno); + align =resource_alignment (res);if (align) -return align; +return align > bios_align ? align : bios_align;dev_err (&dev->dev,"alignment: invalid resource #%d\n" , resno);return 0 ;
Re: [PATCH 9/16 v6] PCI: add boot option to align MMIO resources
In reply to Bjorn HelgaasQuoted 9 lines · Bjorn Helgaas
Quoted 3 lines · Yu Zhao
This patch adds boot option to align MMIO resource for a device. The alignment is a bigger value between the
PAGE_SIZEand the resource size.It looks like this forces alignment on
PAGE_SIZE, not "a bigger value between thePAGE_SIZEand the resource size." Can you clarify the changelog to specify exactly what alignment this option forces?
I guess following would explain your question.
Quoted 15 lines
Quoted 15 lines
int
pci_resource_alignment(struct pci_dev *dev, int resno){drivers/pci/pci.c +4
-2-resource_size_t align; +resource_size_t align, bios_align;struct resource *res = dev->resource + resno; + bios_align =pcibios_resource_alignment (dev, resno); + align =resource_alignment (res);if (align) -return align; +return align > bios_align ? align : bios_align;dev_err (&dev->dev,"alignment: invalid resource #%d\n" , resno);return 0 ;
Re: [PATCH 2/16 v6] PCI: define PCI resource names in an 'enum'
In reply to Bjorn HelgaasQuoted 47 lines · Bjorn Helgaas
Quoted 46 lines · Yu Zhao
Quoted 35 lines · Bjorn Helgaas
Quoted 3 lines · Yu Zhao
This patch moves all definitions of the PCI resource names to an 'enum', and also replaces some hard-coded resource variables with symbol names. This change eases introduction of device specific resources.
Thanks for removing a bunch of magic numbers from the code.
Quoted 15 lines
static void
pci_restore_bars(struct pci_dev *dev)
{drivers/pci/pci.c +0
-12-int i, numres; - -switch (dev->hdr_type) { -case PCI_HEADER_TYPE_NORMAL : - numres =6 ; -break ; -case PCI_HEADER_TYPE_BRIDGE : - numres =2 ; -break ; -case PCI_HEADER_TYPE_CARDBUS : - numres =1 ; -break ;drivers/pci/pci.c pci_find_parent_resource(const struct pci_dev *dev, struct resource *res) · line 366
Quoted 10 lines
drivers/pci/pci.c +2
-5-default : -/* Should never get here, but just in case... */ -return ; - } +int i; -for (i =0 ; i < numres; i++) +for (i =0 ; i <PCI_BRIDGE_RESOURCES ; i++)pci_update_resource (dev, i); }The behavior of this function used to depend on
dev->hdr_type. Now we don't look athdr_typeat all, so we do the same thing for all devices.Quoted 5 lines
For example, for a CardBus device, we used to call
pci_update_resource()only for BAR 0; now we call it for BARs 0-6.Maybe this is safe, but I can't tell from the patch, so I think you should explain *why* it's safe in the changelog.
It's safe because
pci_update_resource()will ignore unused resources. E.g., for a Cardbus, only BAR 0 is used and its 'flags' is set, thenpci_update_resource()only updates it. BAR 1-6 are ignored since their 'flags' are 0.
Quoted 5 lines
Quoted 1 line
I'll put more explanation in the changelog.
This is a logically separate change from merely substituting enum names for magic numbers, so you might even consider splitting it into a separate patch. Better bisection and all that, you know :-)
Will do.
Thanks,
Yu
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
In reply to Bjorn HelgaasQuoted 17 lines · Bjorn Helgaas
Quoted 16 lines · Yu Zhao
Document the new PCI[x86] boot parameters.
Cc Alex Chiang <[email protected]>Cc Grant Grundler <[email protected]>Cc Greg KH <[email protected]>Cc Ingo Molnar <[email protected]>Cc Jesse Barnes <[email protected]>Cc Matthew Wilcox <[email protected]>Cc Randy Dunlap <[email protected]>Cc Roland Dreier <[email protected]>Signed-off-by Yu Zhao <[email protected]>--- Documentation/kernel-parameters.txt | 10 ++++++++++ 1 files changed, 10 insertions(+), 0 deletions(-)
Quoted 12 lines
Documentation/kernel-parameters.txt and is between 256 and 4096 characters. It is defined in the file · line 1683
Quoted 12 lines
Documentation/kernel-parameters.txt +4
-0@@ -1677,6 +1677,16 @@ and is between 256 and 4096 characters. It is defined in the file cbmemsize=nn[KMG] The fixed amount of bus space which is reserved for the CardBus bridge's memory window. The default value is 64 megabytes. + assign-mmio=[dddd:]bb [X86] reassign memory resources of all + devices under bus [dddd:]bb (dddd is the domain + number and bb is the bus number). + assign-pio=[dddd:]bb [X86] reassign io port resources of all
"io" in text should be "IO" or "I/O". (Small "io" is OK as a parameter placeholder.)
Quoted 5 lines
Documentation/kernel-parameters.txt and is between 256 and 4096 characters. It is defined in the file · line 1688
Quoted 5 lines
Documentation/kernel-parameters.txt +5
-0+ devices under bus [dddd:]bb (dddd is the domain + number and bb is the bus number). + align-mmio=[dddd:]bb:dd.f [X86] relocate memory resources of a + device to minimum PAGE_SIZE alignment (dddd is + the domain number and bb, dd and f is the bus,
are the bus,
Quoted 11 lines
Documentation/kernel-parameters.txt and is between 256 and 4096 characters. It is defined in the file · line 1692
Quoted 4 lines
Documentation/kernel-parameters.txt +1
-0+ device and function number). pcie_aspm= [PCIE] Forcibly enable or disable PCIe Active State Power Management.I think it's nicer to have the documentation change included in the patch that implements the change. For example, I think this and patch 9/16 "add boot option to align ..." should be folded into a single patch. And similarly for the other documentation patches.
Bjorn
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
In reply to Bjorn HelgaasQuoted 17 lines · Bjorn Helgaas
Quoted 16 lines · Yu Zhao
Document the new PCI[x86] boot parameters.
Cc Alex Chiang <[email protected]>Cc Grant Grundler <[email protected]>Cc Greg KH <[email protected]>Cc Ingo Molnar <[email protected]>Cc Jesse Barnes <[email protected]>Cc Matthew Wilcox <[email protected]>Cc Randy Dunlap <[email protected]>Cc Roland Dreier <[email protected]>Signed-off-by Yu Zhao <[email protected]>--- Documentation/kernel-parameters.txt | 10 ++++++++++ 1 files changed, 10 insertions(+), 0 deletions(-)
Quoted 12 lines
Documentation/kernel-parameters.txt and is between 256 and 4096 characters. It is defined in the file · line 1683
Quoted 12 lines
Documentation/kernel-parameters.txt +4
-0@@ -1677,6 +1677,16 @@ and is between 256 and 4096 characters. It is defined in the file cbmemsize=nn[KMG] The fixed amount of bus space which is reserved for the CardBus bridge's memory window. The default value is 64 megabytes. + assign-mmio=[dddd:]bb [X86] reassign memory resources of all + devices under bus [dddd:]bb (dddd is the domain + number and bb is the bus number). + assign-pio=[dddd:]bb [X86] reassign io port resources of all
"io" in text should be "IO" or "I/O". (Small "io" is OK as a parameter placeholder.)
Quoted 5 lines
Documentation/kernel-parameters.txt and is between 256 and 4096 characters. It is defined in the file · line 1688
Quoted 5 lines
Documentation/kernel-parameters.txt +5
-0+ devices under bus [dddd:]bb (dddd is the domain + number and bb is the bus number). + align-mmio=[dddd:]bb:dd.f [X86] relocate memory resources of a + device to minimum PAGE_SIZE alignment (dddd is + the domain number and bb, dd and f is the bus,
are the bus,
Quoted 11 lines
Documentation/kernel-parameters.txt and is between 256 and 4096 characters. It is defined in the file · line 1692
Quoted 4 lines
Documentation/kernel-parameters.txt +1
-0+ device and function number). pcie_aspm= [PCIE] Forcibly enable or disable PCIe Active State Power Management.I think it's nicer to have the documentation change included in the patch that implements the change. For example, I think this and patch 9/16 "add boot option to align ..." should be folded into a single patch. And similarly for the other documentation patches.
Bjorn
Re: [PATCH 7/16 v6] PCI: cleanup pcibios_allocate_resources()
In reply to Yinghai LuQuoted 31 lines · Yinghai Lu
Quoted 16 lines · Yu Zhao
arch/x86/pci/i386.c +1
-1@@ -147,7 +147,7 @@ static void __init pcibios_allocate_bus_resources(struct list_head *bus_list)static void __init pcibios_allocate_resources (int pass) {struct pci_dev *dev =NULL ; -int idx, disabled; +int idx, enabled;u16 command;struct resource *r, *pr;@@ -160,22 +160,22 @@ static void __init pcibios_allocate_resources(int pass)if (!r->start)/* Address not assigned at all */ continue ;arch/x86/pci/i386.c static void __init pcibios_allocate_resources(int pass) · line 167
Quoted 10 lines
arch/x86/pci/i386.c +4
-4if (r->flags &IORESOURCE_IO ) - disabled = !(command &PCI_COMMAND_IO ); + enabled = command &PCI_COMMAND_IO ;else - disabled = !(command &PCI_COMMAND_MEMORY ); -if (pass == disabled) { - dev_dbg(&dev->dev, "resource %#08llx-%#08llx (f=%lx, d=%d, p=%d)\n", + enabled = command &PCI_COMMAND_MEMORY ; +if (pass == enabled) +continue ;it seems you change the flow here for MMIO
becausePCI_COMMAND_MEMORYis 2.
Quoted 1 line
YH
Nice finding! Will change it back to 'disable' next version.
Thanks,
Yu
Re: [PATCH 7/16 v6] PCI: cleanup pcibios_allocate_resources()
In reply to Yu ZhaoQuoted 32 lines · Yu Zhao
This cleanup makes
pcibios_allocate_resources()easier to read.Cc Alex Chiang <[email protected]>Cc Grant Grundler <[email protected]>Cc Greg KH <[email protected]>Cc Ingo Molnar <[email protected]>Cc Jesse Barnes <[email protected]>Cc Matthew Wilcox <[email protected]>Cc Randy Dunlap <[email protected]>Cc Roland Dreier <[email protected]>Signed-off-by Yu Zhao <[email protected]>--- arch/x86/pci/i386.c | 28 ++++++++++++++-------------- 1 files changed, 14 insertions(+), 14 deletions(-)arch/x86/pci/i386.c +1
-1@@ -147,7 +147,7 @@ static void __init pcibios_allocate_bus_resources(struct list_head *bus_list)static void __init pcibios_allocate_resources (int pass) {struct pci_dev *dev =NULL ; -int idx, disabled; +int idx, enabled;u16 command;struct resource *r, *pr;@@ -160,22 +160,22 @@ static void __init pcibios_allocate_resources(int pass)if (!r->start)/* Address not assigned at all */ continue ;
arch/x86/pci/i386.c static void __init pcibios_allocate_resources(int pass) · line 167
Quoted 10 lines
arch/x86/pci/i386.c +4
-4if (r->flags &IORESOURCE_IO ) - disabled = !(command &PCI_COMMAND_IO ); + enabled = command &PCI_COMMAND_IO ;else - disabled = !(command &PCI_COMMAND_MEMORY ); -if (pass == disabled) { - dev_dbg(&dev->dev, "resource %#08llx-%#08llx (f=%lx, d=%d, p=%d)\n", + enabled = command &PCI_COMMAND_MEMORY ; +if (pass == enabled) +continue ;
it seems you change the flow here for MMIO
because PCI_COMMAND_MEMORY is 2.
YH
Re: [PATCH 7/16 v6] PCI: cleanup pcibios_allocate_resources()
In reply to Yu ZhaoQuoted 32 lines · Yu Zhao
This cleanup makes
pcibios_allocate_resources()easier to read.Cc Alex Chiang <[email protected]>Cc Grant Grundler <[email protected]>Cc Greg KH <[email protected]>Cc Ingo Molnar <[email protected]>Cc Jesse Barnes <[email protected]>Cc Matthew Wilcox <[email protected]>Cc Randy Dunlap <[email protected]>Cc Roland Dreier <[email protected]>Signed-off-by Yu Zhao <[email protected]>--- arch/x86/pci/i386.c | 28 ++++++++++++++-------------- 1 files changed, 14 insertions(+), 14 deletions(-)arch/x86/pci/i386.c +1
-1@@ -147,7 +147,7 @@ static void __init pcibios_allocate_bus_resources(struct list_head *bus_list)static void __init pcibios_allocate_resources (int pass) {struct pci_dev *dev =NULL ; -int idx, disabled; +int idx, enabled;u16 command;struct resource *r, *pr;@@ -160,22 +160,22 @@ static void __init pcibios_allocate_resources(int pass)if (!r->start)/* Address not assigned at all */ continue ;
arch/x86/pci/i386.c static void __init pcibios_allocate_resources(int pass) · line 167
Quoted 10 lines
arch/x86/pci/i386.c +4
-4if (r->flags &IORESOURCE_IO ) - disabled = !(command &PCI_COMMAND_IO ); + enabled = command &PCI_COMMAND_IO ;else - disabled = !(command &PCI_COMMAND_MEMORY ); -if (pass == disabled) { - dev_dbg(&dev->dev, "resource %#08llx-%#08llx (f=%lx, d=%d, p=%d)\n", + enabled = command &PCI_COMMAND_MEMORY ; +if (pass == enabled) +continue ;
it seems you change the flow here for MMIO
because PCI_COMMAND_MEMORY is 2.
YH
Re: [PATCH 7/16 v6] PCI: cleanup pcibios_allocate_resources()
Quoted 31 lines · Yinghai Lu
Quoted 16 lines · Yu Zhao
arch/x86/pci/i386.c +1
-1@@ -147,7 +147,7 @@ static void __init pcibios_allocate_bus_resources(struct list_head *bus_list)static void __init pcibios_allocate_resources (int pass) {struct pci_dev *dev =NULL ; -int idx, disabled; +int idx, enabled;u16 command;struct resource *r, *pr;@@ -160,22 +160,22 @@ static void __init pcibios_allocate_resources(int pass)if (!r->start)/* Address not assigned at all */ continue ;arch/x86/pci/i386.c static void __init pcibios_allocate_resources(int pass) · line 167
Quoted 10 lines
arch/x86/pci/i386.c +4
-4if (r->flags &IORESOURCE_IO ) - disabled = !(command &PCI_COMMAND_IO ); + enabled = command &PCI_COMMAND_IO ;else - disabled = !(command &PCI_COMMAND_MEMORY ); -if (pass == disabled) { - dev_dbg(&dev->dev, "resource %#08llx-%#08llx (f=%lx, d=%d, p=%d)\n", + enabled = command &PCI_COMMAND_MEMORY ; +if (pass == enabled) +continue ;it seems you change the flow here for MMIO
becausePCI_COMMAND_MEMORYis 2.
Quoted 1 line
YH
Nice finding! Will change it back to 'disable' next version.
Thanks,
Yu
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
In reply to Yu ZhaoQuoted 24 lines · Yu Zhao
Document the new PCI[x86] boot parameters.
Cc Alex Chiang <[email protected]>Cc Grant Grundler <[email protected]>Cc Greg KH <[email protected]>Cc Ingo Molnar <[email protected]>Cc Jesse Barnes <[email protected]>Cc Matthew Wilcox <[email protected]>Cc Randy Dunlap <[email protected]>Cc Roland Dreier <[email protected]>Signed-off-by Yu Zhao <[email protected]>--- Documentation/kernel-parameters.txt | 10 ++++++++++ 1 files changed, 10 insertions(+), 0 deletions(-)Documentation/kernel-parameters.txt +0
-0@@ -1677,6 +1677,16 @@ and is between 256 and 4096 characters. It is defined in the file cbmemsize=nn[KMG] The fixed amount of bus space which is reserved for the CardBus bridge's memory window. The default value is 64 megabytes.
Documentation/kernel-parameters.txt and is between 256 and 4096 characters. It is defined in the file · line 1689
Quoted 10 lines
Documentation/kernel-parameters.txt +10
-0+ assign-mmio=[dddd:]bb [X86] reassign memory resources of all + devices under bus [dddd:]bb (dddd is the domain + number and bb is the bus number). + assign-pio=[dddd:]bb [X86] reassign io port resources of all + devices under bus [dddd:]bb (dddd is the domain + number and bb is the bus number). + align-mmio=[dddd:]bb:dd.f [X86] relocate memory resources of a + device to minimum PAGE_SIZE alignment (dddd is + the domain number and bb, dd and f is the bus, + device and function number).
This seems like a big problem. How are we going to know to add these command line options for devices we haven't even seen/known about yet?
How do we know the bus ids aren't going to change between boots (hint, they are, pci bus ids change all the time...)
We need to be able to do this kind of thing dynamically, not fixed at boot time, which seems way to early to even know about this, right?
thanks,
greg k-h
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
In reply to Yu ZhaoQuoted 24 lines · Yu Zhao
Document the new PCI[x86] boot parameters.
Cc Alex Chiang <[email protected]>Cc Grant Grundler <[email protected]>Cc Greg KH <[email protected]>Cc Ingo Molnar <[email protected]>Cc Jesse Barnes <[email protected]>Cc Matthew Wilcox <[email protected]>Cc Randy Dunlap <[email protected]>Cc Roland Dreier <[email protected]>Signed-off-by Yu Zhao <[email protected]>--- Documentation/kernel-parameters.txt | 10 ++++++++++ 1 files changed, 10 insertions(+), 0 deletions(-)Documentation/kernel-parameters.txt +0
-0@@ -1677,6 +1677,16 @@ and is between 256 and 4096 characters. It is defined in the file cbmemsize=nn[KMG] The fixed amount of bus space which is reserved for the CardBus bridge's memory window. The default value is 64 megabytes.
Documentation/kernel-parameters.txt and is between 256 and 4096 characters. It is defined in the file · line 1689
Quoted 10 lines
Documentation/kernel-parameters.txt +10
-0+ assign-mmio=[dddd:]bb [X86] reassign memory resources of all + devices under bus [dddd:]bb (dddd is the domain + number and bb is the bus number). + assign-pio=[dddd:]bb [X86] reassign io port resources of all + devices under bus [dddd:]bb (dddd is the domain + number and bb is the bus number). + align-mmio=[dddd:]bb:dd.f [X86] relocate memory resources of a + device to minimum PAGE_SIZE alignment (dddd is + the domain number and bb, dd and f is the bus, + device and function number).
This seems like a big problem. How are we going to know to add these command line options for devices we haven't even seen/known about yet?
How do we know the bus ids aren't going to change between boots (hint, they are, pci bus ids change all the time...)
We need to be able to do this kind of thing dynamically, not fixed at boot time, which seems way to early to even know about this, right?
thanks,
greg k-h
Re: [PATCH 15/16 v6] PCI: document the SR-IOV sysfs entries
In reply to Yu ZhaoQuoted 21 lines · Yu Zhao
Document the SR-IOV sysfs entries.
Cc Alex Chiang <[email protected]>Cc Grant Grundler <[email protected]>Cc Greg KH <[email protected]>Cc Ingo Molnar <[email protected]>Cc Jesse Barnes <[email protected]>Cc Matthew Wilcox <[email protected]>Cc Randy Dunlap <[email protected]>Cc Roland Dreier <[email protected]>Signed-off-by Yu Zhao <[email protected]>--- Documentation/ABI/testing/sysfs-bus-pci | 33 +++++++++++++++++++++++++++++++ 1 files changed, 33 insertions(+), 0 deletions(-)Documentation/ABI/testing/sysfs-bus-pci +0
-0
Documentation/ABI/testing/sysfs-bus-pci Description: · line 13
Quoted 5 lines
Documentation/ABI/testing/sysfs-bus-pci +2
-0@@ -9,3 +9,36 @@ Description: that some devices may have malformatted data. If the underlying VPD has a writable section then the corresponding section of this file will be writable. + +What: /sys/bus/pci/devices/.../iov/enable
Are you sure this is still the correct location with your change to struct device?
thanks,
greg k-h
Re: [PATCH 15/16 v6] PCI: document the SR-IOV sysfs entries
In reply to Yu ZhaoQuoted 21 lines · Yu Zhao
Document the SR-IOV sysfs entries.
Cc Alex Chiang <[email protected]>Cc Grant Grundler <[email protected]>Cc Greg KH <[email protected]>Cc Ingo Molnar <[email protected]>Cc Jesse Barnes <[email protected]>Cc Matthew Wilcox <[email protected]>Cc Randy Dunlap <[email protected]>Cc Roland Dreier <[email protected]>Signed-off-by Yu Zhao <[email protected]>--- Documentation/ABI/testing/sysfs-bus-pci | 33 +++++++++++++++++++++++++++++++ 1 files changed, 33 insertions(+), 0 deletions(-)Documentation/ABI/testing/sysfs-bus-pci +0
-0
Documentation/ABI/testing/sysfs-bus-pci Description: · line 13
Quoted 5 lines
Documentation/ABI/testing/sysfs-bus-pci +2
-0@@ -9,3 +9,36 @@ Description: that some devices may have malformatted data. If the underlying VPD has a writable section then the corresponding section of this file will be writable. + +What: /sys/bus/pci/devices/.../iov/enable
Are you sure this is still the correct location with your change to struct device?
thanks,
greg k-h
Re: [PATCH 15/16 v6] PCI: document the SR-IOV sysfs entries
Quoted 17 lines · Greg KH
Quoted 16 lines · Yu Zhao
Document the SR-IOV sysfs entries.
Cc Alex Chiang <[email protected]>Cc Grant Grundler <[email protected]>Cc Greg KH <[email protected]>Cc Ingo Molnar <[email protected]>Cc Jesse Barnes <[email protected]>Cc Matthew Wilcox <[email protected]>Cc Randy Dunlap <[email protected]>Cc Roland Dreier <[email protected]>Signed-off-by Yu Zhao <[email protected]>--- Documentation/ABI/testing/sysfs-bus-pci | 33 +++++++++++++++++++++++++++++++ 1 files changed, 33 insertions(+), 0 deletions(-)
Quoted 13 lines
Documentation/ABI/testing/sysfs-bus-pci Description: · line 13
Quoted 10 lines
Documentation/ABI/testing/sysfs-bus-pci +2
-0@@ -9,3 +9,36 @@ Description: that some devices may have malformatted data. If the underlying VPD has a writable section then the corresponding section of this file will be writable. + +What: /sys/bus/pci/devices/.../iov/enableAre you sure this is still the correct location with your change to struct device?
Nevermind, this is correct.
But the bigger problem is that userspace doesn't know when these attributes show up. So tools like udev and HAL and others can't look for them as they never get notified, and they don't even know if they should be looking for them or not.
Is there any way to tie these attributes to the "main" pci device so that they get created before the device is announced to the world? Doing that would solve this issue.
thanks,
greg k-h
Re: [PATCH 15/16 v6] PCI: document the SR-IOV sysfs entries
In reply to Greg KHQuoted 17 lines · Greg KH
Quoted 16 lines · Yu Zhao
Document the SR-IOV sysfs entries.
Cc Alex Chiang <[email protected]>Cc Grant Grundler <[email protected]>Cc Greg KH <[email protected]>Cc Ingo Molnar <[email protected]>Cc Jesse Barnes <[email protected]>Cc Matthew Wilcox <[email protected]>Cc Randy Dunlap <[email protected]>Cc Roland Dreier <[email protected]>Signed-off-by Yu Zhao <[email protected]>--- Documentation/ABI/testing/sysfs-bus-pci | 33 +++++++++++++++++++++++++++++++ 1 files changed, 33 insertions(+), 0 deletions(-)
Quoted 13 lines
Documentation/ABI/testing/sysfs-bus-pci Description: · line 13
Quoted 10 lines
Documentation/ABI/testing/sysfs-bus-pci +2
-0@@ -9,3 +9,36 @@ Description: that some devices may have malformatted data. If the underlying VPD has a writable section then the corresponding section of this file will be writable. + +What: /sys/bus/pci/devices/.../iov/enableAre you sure this is still the correct location with your change to struct device?
Nevermind, this is correct.
But the bigger problem is that userspace doesn't know when these attributes show up. So tools like udev and HAL and others can't look for them as they never get notified, and they don't even know if they should be looking for them or not.
Is there any way to tie these attributes to the "main" pci device so that they get created before the device is announced to the world? Doing that would solve this issue.
thanks,
greg k-h
Quoted 7 lines · Yu Zhao
Greetings,
Following patches are intended to support SR-IOV capability in the Linux kernel. With these patches, people can turn a PCI device with the capability into multiple ones from software perspective, which will benefit KVM and achieve other purposes such as QoS, security, and etc.
Is there any actual users of this API around yet? How was it tested as there is no hardware to test on? Which drivers are going to have to be rewritten to take advantage of this new interface?
thanks,
greg k-h
Quoted 7 lines · Yu Zhao
Greetings,
Following patches are intended to support SR-IOV capability in the Linux kernel. With these patches, people can turn a PCI device with the capability into multiple ones from software perspective, which will benefit KVM and achieve other purposes such as QoS, security, and etc.
Is there any actual users of this API around yet? How was it tested as there is no hardware to test on? Which drivers are going to have to be rewritten to take advantage of this new interface?
thanks,
greg k-h
Greetings (from a new lurker to the list),
To your question Greg, "yes" and "sort of" ;-). I have started taking a look at these patches with a strong interest in understanding how they work. I've built a kernel with them and tried out a few things with real SR-IOV hardware.
--
Lance Hartmann
Quoted 14 lines · Greg KH
Is there any actual users of this API around yet? How was it tested as
there is no hardware to test on? Which drivers are going to have to be
rewritten to take advantage of this new interface?thanks,
greg k-h
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linux-foundation.org/mailman/listinfo/virtualization
Greetings (from a new lurker to the list),
To your question Greg, "yes" and "sort of" ;-). I have started taking a look at these patches with a strong interest in understanding how they work. I've built a kernel with them and tried out a few things with real SR-IOV hardware.
--
Lance Hartmann
Quoted 14 lines · Greg KH
Is there any actual users of this API around yet? How was it tested as
there is no hardware to test on? Which drivers are going to have to be
rewritten to take advantage of this new interface?thanks,
greg k-h
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linux-foundation.org/mailman/listinfo/virtualization
Quoted 2 lines · H L
Greetings (from a new lurker to the list),
Welcome!
Quoted 4 lines
To your question Greg, "yes" and "sort of" ;-). I have started taking a look at these patches with a strong interest in understanding how they work. I've built a kernel with them and tried out a few things with real SR-IOV hardware.
Did you have to modify individual drivers to take advantage of this code? It looks like the core code will run on this type of hardware, but there seems to be no real advantage until a driver is modified to use it, right?
Or am I missing some great advantage to having this code without modified drivers?
thanks,
greg k-h
Quoted 2 lines · H L
Greetings (from a new lurker to the list),
Welcome!
Quoted 4 lines
To your question Greg, "yes" and "sort of" ;-). I have started taking a look at these patches with a strong interest in understanding how they work. I've built a kernel with them and tried out a few things with real SR-IOV hardware.
Did you have to modify individual drivers to take advantage of this code? It looks like the core code will run on this type of hardware, but there seems to be no real advantage until a driver is modified to use it, right?
Or am I missing some great advantage to having this code without modified drivers?
thanks,
greg k-h
I have not modified any existing drivers, but instead I threw together a bare-bones module enabling me to make a call to pci_iov_register() and then poke at an SR-IOV adapter's /sys entries for which no driver was loaded.
It appears from my perusal thus far that drivers using these new SR-IOV patches will require modification; i.e. the driver associated with the Physical Function (PF) will be required to make the pci_iov_register() call along with the requisite notify() function. Essentially this suggests to me a model for the PF driver to perform any "global actions" or setup on behalf of VFs before enabling them after which VF drivers could be associated.
I have so far only seen Yu Zhao's "7-patch" set. I've not yet looked at his subsequently tendered "15-patch" set so I don't know what has changed. The hardware/firmware implementation for any given SR-IOV compatible device, will determine the extent of differences required between a PF driver and a VF driver.
--
Lance Hartmann
Quoted 30 lines · Greg KH
Date: Thursday, November 6, 2008, 9:43 AM
Quoted 2 lines · H L
Greetings (from a new lurker to the list),
Welcome!
Quoted 1 line
To your question Greg, "yes" and "sort
of" ;-). I have started taking
Quoted 1 line
a look at these patches with a strong interest in
understanding how
Quoted 1 line
they work. I've built a kernel with them and
tried out a few things
Quoted 1 line
with real SR-IOV hardware.
Did you have to modify individual drivers to take advantage of this
code? It looks like the core code will run on this type of hardware,
but there seems to be no real advantage until a driver is modified to
use it, right?Or am I missing some great advantage to having this code without
modified drivers?thanks,
greg k-h
I have not modified any existing drivers, but instead I threw together a bare-bones module enabling me to make a call to pci_iov_register() and then poke at an SR-IOV adapter's /sys entries for which no driver was loaded.
It appears from my perusal thus far that drivers using these new SR-IOV patches will require modification; i.e. the driver associated with the Physical Function (PF) will be required to make the pci_iov_register() call along with the requisite notify() function. Essentially this suggests to me a model for the PF driver to perform any "global actions" or setup on behalf of VFs before enabling them after which VF drivers could be associated.
I have so far only seen Yu Zhao's "7-patch" set. I've not yet looked at his subsequently tendered "15-patch" set so I don't know what has changed. The hardware/firmware implementation for any given SR-IOV compatible device, will determine the extent of differences required between a PF driver and a VF driver.
--
Lance Hartmann
Quoted 30 lines · Greg KH
Date: Thursday, November 6, 2008, 9:43 AM
Quoted 2 lines · H L
Greetings (from a new lurker to the list),
Welcome!
Quoted 1 line
To your question Greg, "yes" and "sort
of" ;-). I have started taking
Quoted 1 line
a look at these patches with a strong interest in
understanding how
Quoted 1 line
they work. I've built a kernel with them and
tried out a few things
Quoted 1 line
with real SR-IOV hardware.
Did you have to modify individual drivers to take advantage of this
code? It looks like the core code will run on this type of hardware,
but there seems to be no real advantage until a driver is modified to
use it, right?Or am I missing some great advantage to having this code without
modified drivers?thanks,
greg k-h
A: No.
Q: Should I include quotations after my reply?
Quoted 5 lines · H L
I have not modified any existing drivers, but instead I threw together a bare-bones module enabling me to make a call to
pci_iov_register()and then poke at an SR-IOV adapter's /sys entries for which no driver was loaded.
Quoted 7 lines
It appears from my perusal thus far that drivers using these new SR-IOV patches will require modification; i.e. the driver associated with the Physical Function (PF) will be required to make the
pci_iov_register()call along with the requisitenotify()function. Essentially this suggests to me a model for the PF driver to perform any "global actions" or setup on behalf of VFs before enabling them after which VF drivers could be associated.
Where would the VF drivers have to be associated? On the "pci_dev" level or on a higher one?
Will all drivers that want to bind to a "VF" device need to be rewritten?
Quoted 5 lines
I have so far only seen Yu Zhao's "7-patch" set. I've not yet looked at his subsequently tendered "15-patch" set so I don't know what has
changed. The hardware/firmware implementation for any given SR-IOV
compatible device, will determine the extent of differences required between a PF driver and a VF driver.
Yeah, that's what I'm worried/curious about. Without seeing the code for such a driver, how can we properly evaluate if this infrastructure is the correct one and the proper way to do all of this?
thanks,
greg k-h
git repository for SR-IOV development?
In reply to Greg KHHas anyone initiated or given consideration to the creation of a git repository (say, on kernel.org) for SR-IOV development?
--
Lance Hartmann
git repository for SR-IOV development?
In reply to Greg KHHas anyone initiated or given consideration to the creation of a git repository (say, on kernel.org) for SR-IOV development?
--
Lance Hartmann
A: No.
Q: Should I include quotations after my reply?
Quoted 5 lines · H L
I have not modified any existing drivers, but instead I threw together a bare-bones module enabling me to make a call to
pci_iov_register()and then poke at an SR-IOV adapter's /sys entries for which no driver was loaded.
Quoted 7 lines
It appears from my perusal thus far that drivers using these new SR-IOV patches will require modification; i.e. the driver associated with the Physical Function (PF) will be required to make the
pci_iov_register()call along with the requisitenotify()function. Essentially this suggests to me a model for the PF driver to perform any "global actions" or setup on behalf of VFs before enabling them after which VF drivers could be associated.
Where would the VF drivers have to be associated? On the "pci_dev" level or on a higher one?
Will all drivers that want to bind to a "VF" device need to be rewritten?
Quoted 5 lines
I have so far only seen Yu Zhao's "7-patch" set. I've not yet looked at his subsequently tendered "15-patch" set so I don't know what has
changed. The hardware/firmware implementation for any given SR-IOV
compatible device, will determine the extent of differences required between a PF driver and a VF driver.
Yeah, that's what I'm worried/curious about. Without seeing the code for such a driver, how can we properly evaluate if this infrastructure is the correct one and the proper way to do all of this?
thanks,
greg k-h
Re: git repository for SR-IOV development?
In reply to H LQuoted 3 lines · H L
Has anyone initiated or given consideration to the creation of a git repository (say, on kernel.org) for SR-IOV development?
Why? It's only a few patches, right? Why would it need a whole new git tree?
thanks,
greg k-h
Re: git repository for SR-IOV development?
In reply to H LQuoted 3 lines · H L
Has anyone initiated or given consideration to the creation of a git repository (say, on kernel.org) for SR-IOV development?
Why? It's only a few patches, right? Why would it need a whole new git tree?
thanks,
greg k-h
Quoted 15 lines · H L
Quoted 1 line
I have not modified any existing drivers, but instead I threw
together
Quoted 11 lines
a bare-bones module enabling me to make a call to
pci_iov_register()and then poke at an SR-IOV adapter's /sys entries for which no driver was loaded.It appears from my perusal thus far that drivers using these new SR-IOV patches will require modification; i.e. the driver associated with the Physical Function (PF) will be required to make the
pci_iov_register()call along with the requisitenotify()function. Essentially this suggests to me a model for the PF driver to perform any "global actions" or setup on behalf of VFs before enabling them after which VF drivers could be associated.
Quoted 2 lines
Where would the VF drivers have to be associated? On the "
pci_dev" level or on a higher one?
A VF appears to the Linux OS as a standard (full, additional) PCI device. The driver is associated in the same way as for a normal PCI device. Ideally, you would use SR-IOV devices on a virtualized system, for example, using Xen. A VF can then be assigned to a guest domain as a full PCI device.
Quoted 2 lines
Will all drivers that want to bind to a "VF" device need to be rewritten?
Currently, any vendor providing a SR-IOV device needs to provide a PF driver and a VF driver that runs on their hardware. A VF driver does not necessarily need to know much about SR-IOV but just run on the presented PCI device. You might want to have a communication channel between PF and VF driver though, for various reasons, if such a channel is not provided in hardware.
Quoted 9 lines
Quoted 5 lines
I have so far only seen Yu Zhao's "7-patch" set. I've not yet looked at his subsequently tendered "15-patch" set so I don't know what has
changed. The hardware/firmware implementation for any given SR-IOV
compatible device, will determine the extent of differences required between a PF driver and a VF driver.Yeah, that's what I'm worried/curious about. Without seeing the code for such a driver, how can we properly evaluate if this infrastructure is the correct one and the proper way to do all of this?
Yu's API allows a PF driver to register with the Linux PCI code and use it to activate VFs and allocate their resources. The PF driver needs to be modified to work with that API. While you can argue about how that API is supposed to look like, it is clear that such an API is required in some form. The PF driver needs to know when VFs are active as it might want to allocate further (device-specific) resources to VFs or initiate further (device-specific) configurations. While probably a lot of SR-IOV specific code has to be in the PF driver, there is also support required from the Linux PCI subsystem, which is to some extend provided by Yu's patches.
Anna
Quoted 15 lines · H L
Quoted 1 line
I have not modified any existing drivers, but instead I threw
together
Quoted 11 lines
a bare-bones module enabling me to make a call to
pci_iov_register()and then poke at an SR-IOV adapter's /sys entries for which no driver was loaded.It appears from my perusal thus far that drivers using these new SR-IOV patches will require modification; i.e. the driver associated with the Physical Function (PF) will be required to make the
pci_iov_register()call along with the requisitenotify()function. Essentially this suggests to me a model for the PF driver to perform any "global actions" or setup on behalf of VFs before enabling them after which VF drivers could be associated.
Quoted 2 lines
Where would the VF drivers have to be associated? On the "
pci_dev" level or on a higher one?
A VF appears to the Linux OS as a standard (full, additional) PCI device. The driver is associated in the same way as for a normal PCI device. Ideally, you would use SR-IOV devices on a virtualized system, for example, using Xen. A VF can then be assigned to a guest domain as a full PCI device.
Quoted 2 lines
Will all drivers that want to bind to a "VF" device need to be rewritten?
Currently, any vendor providing a SR-IOV device needs to provide a PF driver and a VF driver that runs on their hardware. A VF driver does not necessarily need to know much about SR-IOV but just run on the presented PCI device. You might want to have a communication channel between PF and VF driver though, for various reasons, if such a channel is not provided in hardware.
Quoted 9 lines
Quoted 5 lines
I have so far only seen Yu Zhao's "7-patch" set. I've not yet looked at his subsequently tendered "15-patch" set so I don't know what has
changed. The hardware/firmware implementation for any given SR-IOV
compatible device, will determine the extent of differences required between a PF driver and a VF driver.Yeah, that's what I'm worried/curious about. Without seeing the code for such a driver, how can we properly evaluate if this infrastructure is the correct one and the proper way to do all of this?
Yu's API allows a PF driver to register with the Linux PCI code and use it to activate VFs and allocate their resources. The PF driver needs to be modified to work with that API. While you can argue about how that API is supposed to look like, it is clear that such an API is required in some form. The PF driver needs to know when VFs are active as it might want to allocate further (device-specific) resources to VFs or initiate further (device-specific) configurations. While probably a lot of SR-IOV specific code has to be in the PF driver, there is also support required from the Linux PCI subsystem, which is to some extend provided by Yu's patches.
Anna
Quoted 14 lines · Greg KH
Quoted 5 lines · H L
I have not modified any existing drivers, but instead I threw together a bare-bones module enabling me to make a call to
pci_iov_register()and then poke at an SR-IOV adapter's /sys entries for which no driver was loaded.Quoted 7 lines
It appears from my perusal thus far that drivers using these new SR-IOV patches will require modification; i.e. the driver associated with the Physical Function (PF) will be required to make the
pci_iov_register()call along with the requisitenotify()function. Essentially this suggests to me a model for the PF driver to perform any "global actions" or setup on behalf of VFs before enabling them after which VF drivers could be associated.
Quoted 5 lines
Where would the VF drivers have to be associated? On the "
pci_dev" level or on a higher one?Will all drivers that want to bind to a "VF" device need to be rewritten?
The current model being implemented by my colleagues has separate drivers for the PF (aka native) and VF devices. I don't personally believe this is the correct path, but I'm reserving judgement until I see some code.
I don't think we really know what the One True Usage model is for VF devices. Chris Wright has some ideas, I have some ideas and Yu Zhao has some ideas. I bet there's other people who have other ideas too.
Signature · 4 lines
Matthew Wilcox Intel Open Source Technology Centre "Bill, look, we understand that you're interested in selling us this operating system, but compare it to ours. We can't possibly take such a retrograde step."
Quoted 14 lines · Greg KH
Quoted 5 lines · H L
I have not modified any existing drivers, but instead I threw together a bare-bones module enabling me to make a call to
pci_iov_register()and then poke at an SR-IOV adapter's /sys entries for which no driver was loaded.Quoted 7 lines
It appears from my perusal thus far that drivers using these new SR-IOV patches will require modification; i.e. the driver associated with the Physical Function (PF) will be required to make the
pci_iov_register()call along with the requisitenotify()function. Essentially this suggests to me a model for the PF driver to perform any "global actions" or setup on behalf of VFs before enabling them after which VF drivers could be associated.
Quoted 5 lines
Where would the VF drivers have to be associated? On the "
pci_dev" level or on a higher one?Will all drivers that want to bind to a "VF" device need to be rewritten?
The current model being implemented by my colleagues has separate drivers for the PF (aka native) and VF devices. I don't personally believe this is the correct path, but I'm reserving judgement until I see some code.
I don't think we really know what the One True Usage model is for VF devices. Chris Wright has some ideas, I have some ideas and Yu Zhao has some ideas. I bet there's other people who have other ideas too.
Signature · 4 lines
Matthew Wilcox Intel Open Source Technology Centre "Bill, look, we understand that you're interested in selling us this operating system, but compare it to ours. We can't possibly take such a retrograde step."
Quoted 18 lines · Matthew Wilcox
Quoted 17 lines · Greg KH
Quoted 5 lines · H L
I have not modified any existing drivers, but instead I threw together a bare-bones module enabling me to make a call to
pci_iov_register()and then poke at an SR-IOV adapter's /sys entries for which no driver was loaded.Quoted 7 lines
It appears from my perusal thus far that drivers using these new SR-IOV patches will require modification; i.e. the driver associated with the Physical Function (PF) will be required to make the
pci_iov_register()call along with the requisitenotify()function. Essentially this suggests to me a model for the PF driver to perform any "global actions" or setup on behalf of VFs before enabling them after which VF drivers could be associated.Where would the VF drivers have to be associated? On the "
pci_dev" level or on a higher one?
Quoted 7 lines
Quoted 2 lines
Will all drivers that want to bind to a "VF" device need to be rewritten?
The current model being implemented by my colleagues has separate drivers for the PF (aka native) and VF devices. I don't personally believe this is the correct path, but I'm reserving judgement until I see some code.
Hm, I would like to see that code before we can properly evaluate this interface. Especially as they are all tightly tied together.
Quoted 3 lines
I don't think we really know what the One True Usage model is for VF devices. Chris Wright has some ideas, I have some ideas and Yu Zhao has some ideas. I bet there's other people who have other ideas too.
I'd love to hear those ideas.
Rumor has it, there is some Xen code floating around to support this already, is that true?
thanks,
greg k-h
Quoted 18 lines · Matthew Wilcox
Quoted 17 lines · Greg KH
Quoted 5 lines · H L
I have not modified any existing drivers, but instead I threw together a bare-bones module enabling me to make a call to
pci_iov_register()and then poke at an SR-IOV adapter's /sys entries for which no driver was loaded.Quoted 7 lines
It appears from my perusal thus far that drivers using these new SR-IOV patches will require modification; i.e. the driver associated with the Physical Function (PF) will be required to make the
pci_iov_register()call along with the requisitenotify()function. Essentially this suggests to me a model for the PF driver to perform any "global actions" or setup on behalf of VFs before enabling them after which VF drivers could be associated.Where would the VF drivers have to be associated? On the "
pci_dev" level or on a higher one?
Quoted 7 lines
Quoted 2 lines
Will all drivers that want to bind to a "VF" device need to be rewritten?
The current model being implemented by my colleagues has separate drivers for the PF (aka native) and VF devices. I don't personally believe this is the correct path, but I'm reserving judgement until I see some code.
Hm, I would like to see that code before we can properly evaluate this interface. Especially as they are all tightly tied together.
Quoted 3 lines
I don't think we really know what the One True Usage model is for VF devices. Chris Wright has some ideas, I have some ideas and Yu Zhao has some ideas. I bet there's other people who have other ideas too.
I'd love to hear those ideas.
Rumor has it, there is some Xen code floating around to support this already, is that true?
thanks,
greg k-h
Quoted 15 lines · Anna
Quoted 15 lines · H L
Quoted 1 line
I have not modified any existing drivers, but instead I threw
together
Quoted 11 lines
a bare-bones module enabling me to make a call to
pci_iov_register()and then poke at an SR-IOV adapter's /sys entries for which no driver was loaded.It appears from my perusal thus far that drivers using these new SR-IOV patches will require modification; i.e. the driver associated with the Physical Function (PF) will be required to make the
pci_iov_register()call along with the requisitenotify()function. Essentially this suggests to me a model for the PF driver to perform any "global actions" or setup on behalf of VFs before enabling them after which VF drivers could be associated.
Quoted 8 lines
Quoted 2 lines
Where would the VF drivers have to be associated? On the "
pci_dev" level or on a higher one?A VF appears to the Linux OS as a standard (full, additional) PCI device. The driver is associated in the same way as for a normal PCI device. Ideally, you would use SR-IOV devices on a virtualized system, for example, using Xen. A VF can then be assigned to a guest domain as a full PCI device.
It's that "second" part that I'm worried about. How is that going to happen? Do you have any patches that show this kind of "assignment"?
Quoted 5 lines
Quoted 2 lines
Will all drivers that want to bind to a "VF" device need to be rewritten?
Currently, any vendor providing a SR-IOV device needs to provide a PF driver and a VF driver that runs on their hardware.
Are there any such drivers available yet?
Quoted 4 lines
A VF driver does not necessarily need to know much about SR-IOV but just run on the presented PCI device. You might want to have a communication channel between PF and VF driver though, for various reasons, if such a channel is not provided in hardware.
Agreed, but what does that channel look like in Linux?
I have some ideas of what I think it should look like, but if people already have code, I'd love to see that as well.
Quoted 10 lines
Quoted 9 lines
Quoted 5 lines
I have so far only seen Yu Zhao's "7-patch" set. I've not yet looked at his subsequently tendered "15-patch" set so I don't know what has
changed. The hardware/firmware implementation for any given SR-IOV
compatible device, will determine the extent of differences required between a PF driver and a VF driver.Yeah, that's what I'm worried/curious about. Without seeing the code for such a driver, how can we properly evaluate if this infrastructure is the correct one and the proper way to do all of this?
Quoted 5 lines
Yu's API allows a PF driver to register with the Linux PCI code and use it to activate VFs and allocate their resources. The PF driver needs to be modified to work with that API. While you can argue about how that API is supposed to look like, it is clear that such an API is required in some form.
I totally agree, I'm arguing about what that API looks like :)
I want to see some code...
thanks,
greg k-h
Quoted 15 lines · Anna
Quoted 15 lines · H L
Quoted 1 line
I have not modified any existing drivers, but instead I threw
together
Quoted 11 lines
a bare-bones module enabling me to make a call to
pci_iov_register()and then poke at an SR-IOV adapter's /sys entries for which no driver was loaded.It appears from my perusal thus far that drivers using these new SR-IOV patches will require modification; i.e. the driver associated with the Physical Function (PF) will be required to make the
pci_iov_register()call along with the requisitenotify()function. Essentially this suggests to me a model for the PF driver to perform any "global actions" or setup on behalf of VFs before enabling them after which VF drivers could be associated.
Quoted 8 lines
Quoted 2 lines
Where would the VF drivers have to be associated? On the "
pci_dev" level or on a higher one?A VF appears to the Linux OS as a standard (full, additional) PCI device. The driver is associated in the same way as for a normal PCI device. Ideally, you would use SR-IOV devices on a virtualized system, for example, using Xen. A VF can then be assigned to a guest domain as a full PCI device.
It's that "second" part that I'm worried about. How is that going to happen? Do you have any patches that show this kind of "assignment"?
Quoted 5 lines
Quoted 2 lines
Will all drivers that want to bind to a "VF" device need to be rewritten?
Currently, any vendor providing a SR-IOV device needs to provide a PF driver and a VF driver that runs on their hardware.
Are there any such drivers available yet?
Quoted 4 lines
A VF driver does not necessarily need to know much about SR-IOV but just run on the presented PCI device. You might want to have a communication channel between PF and VF driver though, for various reasons, if such a channel is not provided in hardware.
Agreed, but what does that channel look like in Linux?
I have some ideas of what I think it should look like, but if people already have code, I'd love to see that as well.
Quoted 10 lines
Quoted 9 lines
Quoted 5 lines
I have so far only seen Yu Zhao's "7-patch" set. I've not yet looked at his subsequently tendered "15-patch" set so I don't know what has
changed. The hardware/firmware implementation for any given SR-IOV
compatible device, will determine the extent of differences required between a PF driver and a VF driver.Yeah, that's what I'm worried/curious about. Without seeing the code for such a driver, how can we properly evaluate if this infrastructure is the correct one and the proper way to do all of this?
Quoted 5 lines
Yu's API allows a PF driver to register with the Linux PCI code and use it to activate VFs and allocate their resources. The PF driver needs to be modified to work with that API. While you can argue about how that API is supposed to look like, it is clear that such an API is required in some form.
I totally agree, I'm arguing about what that API looks like :)
I want to see some code...
thanks,
greg k-h
Quoted 18 lines · Greg KH
Quoted 1 line · H L
I have not modified any existing drivers, but instead
I threw together
Quoted 1 line
a bare-bones module enabling me to make a call to
pci_iov_register()Quoted 1 line
and then poke at an SR-IOV adapter's /sys entries
for which no driver
Quoted 3 lines
was loaded.
It appears from my perusal thus far that drivers using
these new
Quoted 1 line
SR-IOV patches will require modification; i.e. the
driver associated
Quoted 1 line
with the Physical Function (PF) will be required to
make the
Quoted 1 line
pci_iov_register()call along with the requisite
notify()function.Quoted 1 line
Essentially this suggests to me a model for the PF
Quoted 8 lines
driver to perform
Quoted 1 line
any "global actions" or setup on behalf of
VFs before enabling them
Quoted 1 line
after which VF drivers could be associated.
Where would the VF drivers have to be associated? On the "
pci_dev"
level or on a higher one?
I have not yet fully grocked Yu Zhao's model to answer this. That said, I would *hope* to find it on the "pci_dev" level.
Quoted 3 lines
Will all drivers that want to bind to a "VF" device need to be
rewritten?
Not necessarily, or perhaps minimally; depends on hardware/firmware and actions the driver wants to take. An example here might assist. Let's just say someone has created, oh, I don't know, maybe an SR-IOV NIC. Now, for 'general' I/O operations to pass network traffic back and forth there would ideally be no difference in the actions and therefore behavior of a PF driver and a VF driver. But, what do you do in the instance a VF wants to change link-speed? As that physical characteristic affects all VFs, how do you handle that? This is where the hardware/firmware implementation part comes to play. If a VF driver performs some actions to initiate the change in link speed, the logic in the adapter could be anything like:
1. Acknowledge the request as if it were really done, but effectively ignore it. The Independent Hardware Vendor (IHV) might dictate that if you want to change any "global" characteristics of an adapter, you may only do so via the PF driver. Granted, this, depending on the device class, may just not be acceptable.
2. Acknowledge the request and then trigger an interrupt to the PF driver to have it assist. The PF driver might then just set the new link-speed, or it could result in a PF driver communicating by some mechanism to all of the VF driver instances that this change of link-speed was requested.
3. Acknowledge the request and perform inner PF and VF communication of this event within the logic of the card (e.g. to "vote" on whether or not to perform this action) with interrupts and associated status delivered to all PF and VF drivers.
The list goes on.
Quoted 8 lines
Quoted 2 lines
I have so far only seen Yu Zhao's
"7-patch" set. I've not yet looked
Quoted 1 line
at his subsequently tendered "15-patch" set
so I don't know what has
Quoted 1 line
changed. The hardware/firmware implementation for
any given SR-IOV
Quoted 1 line
compatible device, will determine the extent of
Quoted 8 lines
differences required
Quoted 1 line
between a PF driver and a VF driver.
Yeah, that's what I'm worried/curious about. Without seeing the code
for such a driver, how can we properly evaluate if this infrastructure
is the correct one and the proper way to do all of this?
As the example above demonstrates, that's a tough question to answer. Ideally, in my view, there would only be one driver written per SR-IOV device and it would contain the logic to "do the right things" based on whether its running as a PF or VF with that determination easily accomplished by testing the existence of the SR-IOV extended capability. Then, in an effort to minimize (if not eliminate) the complexities of driver-to-driver actions for fielding "global events", contain as much of the logic as is possible within the adapter. Minimizing the efforts required for the device driver writers in my opinion paves the way to greater adoption of this technology.
Quoted 18 lines · Greg KH
Quoted 1 line · H L
I have not modified any existing drivers, but instead
I threw together
Quoted 1 line
a bare-bones module enabling me to make a call to
pci_iov_register()Quoted 1 line
and then poke at an SR-IOV adapter's /sys entries
for which no driver
Quoted 3 lines
was loaded.
It appears from my perusal thus far that drivers using
these new
Quoted 1 line
SR-IOV patches will require modification; i.e. the
driver associated
Quoted 1 line
with the Physical Function (PF) will be required to
make the
Quoted 1 line
pci_iov_register()call along with the requisite
notify()function.Quoted 1 line
Essentially this suggests to me a model for the PF
Quoted 8 lines
driver to perform
Quoted 1 line
any "global actions" or setup on behalf of
VFs before enabling them
Quoted 1 line
after which VF drivers could be associated.
Where would the VF drivers have to be associated? On the "
pci_dev"
level or on a higher one?
I have not yet fully grocked Yu Zhao's model to answer this. That said, I would *hope* to find it on the "pci_dev" level.
Quoted 3 lines
Will all drivers that want to bind to a "VF" device need to be
rewritten?
Not necessarily, or perhaps minimally; depends on hardware/firmware and actions the driver wants to take. An example here might assist. Let's just say someone has created, oh, I don't know, maybe an SR-IOV NIC. Now, for 'general' I/O operations to pass network traffic back and forth there would ideally be no difference in the actions and therefore behavior of a PF driver and a VF driver. But, what do you do in the instance a VF wants to change link-speed? As that physical characteristic affects all VFs, how do you handle that? This is where the hardware/firmware implementation part comes to play. If a VF driver performs some actions to initiate the change in link speed, the logic in the adapter could be anything like:
1. Acknowledge the request as if it were really done, but effectively ignore it. The Independent Hardware Vendor (IHV) might dictate that if you want to change any "global" characteristics of an adapter, you may only do so via the PF driver. Granted, this, depending on the device class, may just not be acceptable.
2. Acknowledge the request and then trigger an interrupt to the PF driver to have it assist. The PF driver might then just set the new link-speed, or it could result in a PF driver communicating by some mechanism to all of the VF driver instances that this change of link-speed was requested.
3. Acknowledge the request and perform inner PF and VF communication of this event within the logic of the card (e.g. to "vote" on whether or not to perform this action) with interrupts and associated status delivered to all PF and VF drivers.
The list goes on.
Quoted 8 lines
Quoted 2 lines
I have so far only seen Yu Zhao's
"7-patch" set. I've not yet looked
Quoted 1 line
at his subsequently tendered "15-patch" set
so I don't know what has
Quoted 1 line
changed. The hardware/firmware implementation for
any given SR-IOV
Quoted 1 line
compatible device, will determine the extent of
Quoted 8 lines
differences required
Quoted 1 line
between a PF driver and a VF driver.
Yeah, that's what I'm worried/curious about. Without seeing the code
for such a driver, how can we properly evaluate if this infrastructure
is the correct one and the proper way to do all of this?
As the example above demonstrates, that's a tough question to answer. Ideally, in my view, there would only be one driver written per SR-IOV device and it would contain the logic to "do the right things" based on whether its running as a PF or VF with that determination easily accomplished by testing the existence of the SR-IOV extended capability. Then, in an effort to minimize (if not eliminate) the complexities of driver-to-driver actions for fielding "global events", contain as much of the logic as is possible within the adapter. Minimizing the efforts required for the device driver writers in my opinion paves the way to greater adoption of this technology.
Quoted 26 lines · H L
Quoted 23 lines · Greg KH
Quoted 1 line · H L
I have not modified any existing drivers, but instead
I threw together
Quoted 1 line
a bare-bones module enabling me to make a call to
pci_iov_register()Quoted 1 line
and then poke at an SR-IOV adapter's /sys entries
for which no driver
Quoted 3 lines
was loaded.
It appears from my perusal thus far that drivers using
these new
Quoted 1 line
SR-IOV patches will require modification; i.e. the
driver associated
Quoted 1 line
with the Physical Function (PF) will be required to
make the
Quoted 1 line
pci_iov_register()call along with the requisite
notify()function.Quoted 1 line
Essentially this suggests to me a model for the PF
driver to perform
Quoted 1 line
any "global actions" or setup on behalf of
VFs before enabling them
Quoted 1 line
after which VF drivers could be associated.
Quoted 7 lines
Quoted 3 lines
Where would the VF drivers have to be associated? On the "
pci_dev"
level or on a higher one?I have not yet fully grocked Yu Zhao's model to answer this. That said, I would *hope* to find it on the "
pci_dev" level.
Me too.
Quoted 4 lines
Quoted 3 lines
Will all drivers that want to bind to a "VF" device need to be
rewritten?
Quoted 11 lines
Not necessarily, or perhaps minimally; depends on hardware/firmware and actions the driver wants to take. An example here might assist. Let's just say someone has created, oh, I don't know, maybe an SR-IOV NIC. Now, for 'general' I/O operations to pass network traffic back and forth there would ideally be no difference in the actions and therefore behavior of a PF driver and a VF driver. But, what do you do in the instance a VF wants to change link-speed? As that physical characteristic affects all VFs, how do you handle that? This is where the hardware/firmware implementation part comes to play. If a VF driver performs some actions to initiate the change in link speed, the logic in the adapter could be anything like:
<snip>
Yes, I agree that all of this needs to be done, somehow.
It's that "somehow" that I am interested in trying to see how it works out.
Quoted 18 lines
Quoted 8 lines
Quoted 2 lines
I have so far only seen Yu Zhao's
"7-patch" set. I've not yet looked
Quoted 1 line
at his subsequently tendered "15-patch" set
so I don't know what has
Quoted 1 line
changed. The hardware/firmware implementation for
any given SR-IOV
Quoted 1 line
compatible device, will determine the extent of
Quoted 8 lines
differences required
Quoted 1 line
between a PF driver and a VF driver.
Yeah, that's what I'm worried/curious about. Without seeing the code
for such a driver, how can we properly evaluate if this infrastructure
is the correct one and the proper way to do all of this?
Quoted 10 lines
As the example above demonstrates, that's a tough question to answer. Ideally, in my view, there would only be one driver written per SR-IOV device and it would contain the logic to "do the right things" based on whether its running as a PF or VF with that determination easily accomplished by testing the existence of the SR-IOV extended
capability. Then, in an effort to minimize (if not eliminate) the
complexities of driver-to-driver actions for fielding "global events", contain as much of the logic as is possible within the adapter. Minimizing the efforts required for the device driver writers in my opinion paves the way to greater adoption of this technology.
Yes, making things easier is the key here.
Perhaps some of this could be hidden with a new bus type for these kinds of devices? Or a "virtual" bus of pci devices that the original SR-IOV device creates that corrispond to the individual virtual PCI devices? If that were the case, then it might be a lot easier in the end.
thanks,
greg k-h
Quoted 26 lines · H L
Quoted 23 lines · Greg KH
Quoted 1 line · H L
I have not modified any existing drivers, but instead
I threw together
Quoted 1 line
a bare-bones module enabling me to make a call to
pci_iov_register()Quoted 1 line
and then poke at an SR-IOV adapter's /sys entries
for which no driver
Quoted 3 lines
was loaded.
It appears from my perusal thus far that drivers using
these new
Quoted 1 line
SR-IOV patches will require modification; i.e. the
driver associated
Quoted 1 line
with the Physical Function (PF) will be required to
make the
Quoted 1 line
pci_iov_register()call along with the requisite
notify()function.Quoted 1 line
Essentially this suggests to me a model for the PF
driver to perform
Quoted 1 line
any "global actions" or setup on behalf of
VFs before enabling them
Quoted 1 line
after which VF drivers could be associated.
Quoted 7 lines
Quoted 3 lines
Where would the VF drivers have to be associated? On the "
pci_dev"
level or on a higher one?I have not yet fully grocked Yu Zhao's model to answer this. That said, I would *hope* to find it on the "
pci_dev" level.
Me too.
Quoted 4 lines
Quoted 3 lines
Will all drivers that want to bind to a "VF" device need to be
rewritten?
Quoted 11 lines
Not necessarily, or perhaps minimally; depends on hardware/firmware and actions the driver wants to take. An example here might assist. Let's just say someone has created, oh, I don't know, maybe an SR-IOV NIC. Now, for 'general' I/O operations to pass network traffic back and forth there would ideally be no difference in the actions and therefore behavior of a PF driver and a VF driver. But, what do you do in the instance a VF wants to change link-speed? As that physical characteristic affects all VFs, how do you handle that? This is where the hardware/firmware implementation part comes to play. If a VF driver performs some actions to initiate the change in link speed, the logic in the adapter could be anything like:
<snip>
Yes, I agree that all of this needs to be done, somehow.
It's that "somehow" that I am interested in trying to see how it works out.
Quoted 18 lines
Quoted 8 lines
Quoted 2 lines
I have so far only seen Yu Zhao's
"7-patch" set. I've not yet looked
Quoted 1 line
at his subsequently tendered "15-patch" set
so I don't know what has
Quoted 1 line
changed. The hardware/firmware implementation for
any given SR-IOV
Quoted 1 line
compatible device, will determine the extent of
Quoted 8 lines
differences required
Quoted 1 line
between a PF driver and a VF driver.
Yeah, that's what I'm worried/curious about. Without seeing the code
for such a driver, how can we properly evaluate if this infrastructure
is the correct one and the proper way to do all of this?
Quoted 10 lines
As the example above demonstrates, that's a tough question to answer. Ideally, in my view, there would only be one driver written per SR-IOV device and it would contain the logic to "do the right things" based on whether its running as a PF or VF with that determination easily accomplished by testing the existence of the SR-IOV extended
capability. Then, in an effort to minimize (if not eliminate) the
complexities of driver-to-driver actions for fielding "global events", contain as much of the logic as is possible within the adapter. Minimizing the efforts required for the device driver writers in my opinion paves the way to greater adoption of this technology.
Yes, making things easier is the key here.
Perhaps some of this could be hidden with a new bus type for these kinds of devices? Or a "virtual" bus of pci devices that the original SR-IOV device creates that corrispond to the individual virtual PCI devices? If that were the case, then it might be a lot easier in the end.
thanks,
greg k-h
[Anna, can you fix your word-wrapping please? Your lines appear to be infinitely long which is most unpleasant to reply to]
Quoted 8 lines · Anna
Quoted 2 lines
Where would the VF drivers have to be associated? On the "
pci_dev" level or on a higher one?A VF appears to the Linux OS as a standard (full, additional) PCI device. The driver is associated in the same way as for a normal PCI device. Ideally, you would use SR-IOV devices on a virtualized system, for example, using Xen. A VF can then be assigned to a guest domain as a full PCI device.
It's not clear thats the right solution. If the VF devices are _only_ going to be used by the guest, then arguably, we don't want to create pci_devs for them in the host. (I think it _is_ the right answer, but I want to make it clear there's multiple opinions on this).
Quoted 9 lines
Quoted 2 lines
Will all drivers that want to bind to a "VF" device need to be rewritten?
Currently, any vendor providing a SR-IOV device needs to provide a PF driver and a VF driver that runs on their hardware. A VF driver does not necessarily need to know much about SR-IOV but just run on the presented PCI device. You might want to have a communication channel between PF and VF driver though, for various reasons, if such a channel is not provided in hardware.
That is one model. Another model is to provide one driver that can handle both PF and VF devices. A third model is to provide, say, a Windows VF driver and a Xen PF driver and only support Windows-on-Xen. (This last would probably be an exercise in foot-shooting, but nevertheless, I've heard it mooted).
Quoted 4 lines
Quoted 3 lines
Yeah, that's what I'm worried/curious about. Without seeing the code for such a driver, how can we properly evaluate if this infrastructure is the correct one and the proper way to do all of this?
Quoted 9 lines
Yu's API allows a PF driver to register with the Linux PCI code and use it to activate VFs and allocate their resources. The PF driver needs to be modified to work with that API. While you can argue about how that API is supposed to look like, it is clear that such an API is required in some form. The PF driver needs to know when VFs are active as it might want to allocate further (device-specific) resources to VFs or initiate further (device-specific) configurations. While probably a lot of SR-IOV specific code has to be in the PF driver, there is also support required from the Linux PCI subsystem, which is to some extend provided by Yu's patches.
Everyone agrees that some support is necessary. The question is exactly what it looks like. I must confess to not having reviewed this latest patch series yet -- I'm a little burned out on patch review.
Signature · 4 lines
Matthew Wilcox Intel Open Source Technology Centre "Bill, look, we understand that you're interested in selling us this operating system, but compare it to ours. We can't possibly take such a retrograde step."
[Anna, can you fix your word-wrapping please? Your lines appear to be infinitely long which is most unpleasant to reply to]
Quoted 8 lines · Anna
Quoted 2 lines
Where would the VF drivers have to be associated? On the "
pci_dev" level or on a higher one?A VF appears to the Linux OS as a standard (full, additional) PCI device. The driver is associated in the same way as for a normal PCI device. Ideally, you would use SR-IOV devices on a virtualized system, for example, using Xen. A VF can then be assigned to a guest domain as a full PCI device.
It's not clear thats the right solution. If the VF devices are _only_ going to be used by the guest, then arguably, we don't want to create pci_devs for them in the host. (I think it _is_ the right answer, but I want to make it clear there's multiple opinions on this).
Quoted 9 lines
Quoted 2 lines
Will all drivers that want to bind to a "VF" device need to be rewritten?
Currently, any vendor providing a SR-IOV device needs to provide a PF driver and a VF driver that runs on their hardware. A VF driver does not necessarily need to know much about SR-IOV but just run on the presented PCI device. You might want to have a communication channel between PF and VF driver though, for various reasons, if such a channel is not provided in hardware.
That is one model. Another model is to provide one driver that can handle both PF and VF devices. A third model is to provide, say, a Windows VF driver and a Xen PF driver and only support Windows-on-Xen. (This last would probably be an exercise in foot-shooting, but nevertheless, I've heard it mooted).
Quoted 4 lines
Quoted 3 lines
Yeah, that's what I'm worried/curious about. Without seeing the code for such a driver, how can we properly evaluate if this infrastructure is the correct one and the proper way to do all of this?
Quoted 9 lines
Yu's API allows a PF driver to register with the Linux PCI code and use it to activate VFs and allocate their resources. The PF driver needs to be modified to work with that API. While you can argue about how that API is supposed to look like, it is clear that such an API is required in some form. The PF driver needs to know when VFs are active as it might want to allocate further (device-specific) resources to VFs or initiate further (device-specific) configurations. While probably a lot of SR-IOV specific code has to be in the PF driver, there is also support required from the Linux PCI subsystem, which is to some extend provided by Yu's patches.
Everyone agrees that some support is necessary. The question is exactly what it looks like. I must confess to not having reviewed this latest patch series yet -- I'm a little burned out on patch review.
Signature · 4 lines
Matthew Wilcox Intel Open Source Technology Centre "Bill, look, we understand that you're interested in selling us this operating system, but compare it to ours. We can't possibly take such a retrograde step."
Re: git repository for SR-IOV development?
In reply to Greg KHQuoted 10 lines · Greg KH
Quoted 2 lines · H L
Has anyone initiated or given consideration to the
creation of a git
Quoted 1 line
repository (say, on kernel.org) for SR-IOV
development?
Why? It's only a few patches, right? Why would it need a whole new git
tree?
So as to minimize the time and effort patching a kernel, especially if the tree (and/or hash level) against which the patches were created fails to be specified on a mailing-list. Plus, there appears to be questions raised on how, precisely, the implementation should ultimately be modeled and especially given that, who knows at this point what number of patches will ultimately be submitted? I know I've built the "7-patch" one (painfully, by the way), and I'm aware there's another 15-patch set out there which I've not yet examined.
Re: git repository for SR-IOV development?
In reply to Greg KHQuoted 10 lines · Greg KH
Quoted 2 lines · H L
Has anyone initiated or given consideration to the
creation of a git
Quoted 1 line
repository (say, on kernel.org) for SR-IOV
development?
Why? It's only a few patches, right? Why would it need a whole new git
tree?
So as to minimize the time and effort patching a kernel, especially if the tree (and/or hash level) against which the patches were created fails to be specified on a mailing-list. Plus, there appears to be questions raised on how, precisely, the implementation should ultimately be modeled and especially given that, who knows at this point what number of patches will ultimately be submitted? I know I've built the "7-patch" one (painfully, by the way), and I'm aware there's another 15-patch set out there which I've not yet examined.
Quoted 30 lines
Subject: Re: [PATCH 0/16 v6] PCI: Linux kernel SR-IOV support
Quoted 4 lines · Anna
Quoted 4 lines · H L
Quoted 1 line
I have not modified any existing drivers, but instead I threw
together
Quoted 1 line
a bare-bones module enabling me to make a call to
pci_iov_register()Quoted 1 line
Quoted 1 line
Quoted 1 line
and then poke at an SR-IOV adapter's /sys entries for which no
driver
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
was loaded.
It appears from my perusal thus far that drivers using these new SR-IOV patches will require modification; i.e. the driver
associated
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
with the Physical Function (PF) will be required to make the
pci_iov_register()call along with the requisitenotify()function.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Essentially this suggests to me a model for the PF driver to
perform
Quoted 1 line
Quoted 1 line
Quoted 1 line
any "global actions" or setup on behalf of VFs before enabling
them
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
after which VF drivers could be associated.
Where would the VF drivers have to be associated? On the "
pci_dev" level or on a higher one?A VF appears to the Linux OS as a standard (full, additional) PCI device. The driver is associated in the same way as for a normal PCI device. Ideally, you would use SR-IOV devices on a virtualized
Quoted 7 lines
system,
Quoted 1 line
for example, using Xen. A VF can then be assigned to a guest domain
as
Quoted 1 line
a full PCI device.
It's that "second" part that I'm worried about. How is that going to happen? Do you have any patches that show this kind of "assignment"?
That depends on your setup. Using Xen, you could assign the VF to a guest domain like any other PCI device, e.g. using PCI pass-through. For VMware, KVM, there are standard ways to do that, too. I currently don't see why SR-IOV devices would need any specific, non-standard mechanism for device assignment.
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
Will all drivers that want to bind to a "VF" device need to be rewritten?
Currently, any vendor providing a SR-IOV device needs to provide a PF driver and a VF driver that runs on their hardware.
Are there any such drivers available yet?
I don't know.
Quoted 9 lines
Quoted 4 lines
A VF driver does not necessarily need to know much about SR-IOV but just run on the presented PCI device. You might want to have a communication channel between PF and VF driver though, for various reasons, if such a channel is not provided in hardware.
Agreed, but what does that channel look like in Linux?
I have some ideas of what I think it should look like, but if people already have code, I'd love to see that as well.
At this point I would guess that this code is vendor specific, as are the drivers. The issue I see is that most likely drivers will run in different environments, for example, in Xen the PF driver runs in a driver domain while a VF driver runs in a guest VM. So a communication channel would need to be either Xen specific, or vendor specific. Also, a guest using the VF might run Windows while the PF might be controlled under Linux.
Anna
Quoted 30 lines
Subject: Re: [PATCH 0/16 v6] PCI: Linux kernel SR-IOV support
Quoted 4 lines · Anna
Quoted 4 lines · H L
Quoted 1 line
I have not modified any existing drivers, but instead I threw
together
Quoted 1 line
a bare-bones module enabling me to make a call to
pci_iov_register()Quoted 1 line
Quoted 1 line
Quoted 1 line
and then poke at an SR-IOV adapter's /sys entries for which no
driver
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
was loaded.
It appears from my perusal thus far that drivers using these new SR-IOV patches will require modification; i.e. the driver
associated
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
with the Physical Function (PF) will be required to make the
pci_iov_register()call along with the requisitenotify()function.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Essentially this suggests to me a model for the PF driver to
perform
Quoted 1 line
Quoted 1 line
Quoted 1 line
any "global actions" or setup on behalf of VFs before enabling
them
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
after which VF drivers could be associated.
Where would the VF drivers have to be associated? On the "
pci_dev" level or on a higher one?A VF appears to the Linux OS as a standard (full, additional) PCI device. The driver is associated in the same way as for a normal PCI device. Ideally, you would use SR-IOV devices on a virtualized
Quoted 7 lines
system,
Quoted 1 line
for example, using Xen. A VF can then be assigned to a guest domain
as
Quoted 1 line
a full PCI device.
It's that "second" part that I'm worried about. How is that going to happen? Do you have any patches that show this kind of "assignment"?
That depends on your setup. Using Xen, you could assign the VF to a guest domain like any other PCI device, e.g. using PCI pass-through. For VMware, KVM, there are standard ways to do that, too. I currently don't see why SR-IOV devices would need any specific, non-standard mechanism for device assignment.
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
Will all drivers that want to bind to a "VF" device need to be rewritten?
Currently, any vendor providing a SR-IOV device needs to provide a PF driver and a VF driver that runs on their hardware.
Are there any such drivers available yet?
I don't know.
Quoted 9 lines
Quoted 4 lines
A VF driver does not necessarily need to know much about SR-IOV but just run on the presented PCI device. You might want to have a communication channel between PF and VF driver though, for various reasons, if such a channel is not provided in hardware.
Agreed, but what does that channel look like in Linux?
I have some ideas of what I think it should look like, but if people already have code, I'd love to see that as well.
At this point I would guess that this code is vendor specific, as are the drivers. The issue I see is that most likely drivers will run in different environments, for example, in Xen the PF driver runs in a driver domain while a VF driver runs in a guest VM. So a communication channel would need to be either Xen specific, or vendor specific. Also, a guest using the VF might run Windows while the PF might be controlled under Linux.
Anna
Quoted 95 lines
Subject: Re: [PATCH 0/16 v6] PCI: Linux kernel SR-IOV support
Quoted 26 lines · H L
Quoted 23 lines · Greg KH
Quoted 1 line · H L
I have not modified any existing drivers, but instead
I threw together
Quoted 1 line
a bare-bones module enabling me to make a call to
pci_iov_register()Quoted 1 line
and then poke at an SR-IOV adapter's /sys entries
for which no driver
Quoted 3 lines
was loaded.
It appears from my perusal thus far that drivers using
these new
Quoted 1 line
SR-IOV patches will require modification; i.e. the
driver associated
Quoted 1 line
with the Physical Function (PF) will be required to
make the
Quoted 1 line
pci_iov_register()call along with the requisite
notify()function.Quoted 1 line
Essentially this suggests to me a model for the PF
driver to perform
Quoted 1 line
any "global actions" or setup on behalf of
VFs before enabling them
Quoted 1 line
after which VF drivers could be associated.
Quoted 7 lines
Quoted 3 lines
Where would the VF drivers have to be associated? On the "
pci_dev"
level or on a higher one?I have not yet fully grocked Yu Zhao's model to answer this. That said, I would *hope* to find it on the "
pci_dev" level.Me too.
Quoted 4 lines
Quoted 3 lines
Will all drivers that want to bind to a "VF" device need to be
rewritten?Quoted 8 lines
Not necessarily, or perhaps minimally; depends on hardware/firmware and actions the driver wants to take. An example here might assist. Let's just say someone has created, oh, I don't know, maybe an SR-IOV NIC. Now, for 'general' I/O operations to pass network traffic back and forth there would ideally be no difference in the actions and therefore behavior of a PF driver and a VF driver. But, what do you do in the instance a VF wants to change link-speed? As that physical characteristic affects all VFs, how do you handle that? This is
where
Quoted 2 lines
the hardware/firmware implementation part comes to play. If a VF driver performs some actions to initiate the change in link speed,
the
Quoted 1 line
logic in the adapter could be anything like:
<snip>
Yes, I agree that all of this needs to be done, somehow.
It's that "somehow" that I am interested in trying to see how it works out.
Quoted 18 lines
Quoted 8 lines
Quoted 2 lines
I have so far only seen Yu Zhao's
"7-patch" set. I've not yet looked
Quoted 1 line
at his subsequently tendered "15-patch" set
so I don't know what has
Quoted 1 line
changed. The hardware/firmware implementation for
any given SR-IOV
Quoted 1 line
compatible device, will determine the extent of
Quoted 8 lines
differences required
Quoted 1 line
between a PF driver and a VF driver.
Yeah, that's what I'm worried/curious about. Without seeing the code
for such a driver, how can we properly evaluate if this infrastructure
is the correct one and the proper way to do all of this?Quoted 2 lines
As the example above demonstrates, that's a tough question to answer. Ideally, in my view, there would only be one driver written per SR-
IOV
Quoted 5 lines
device and it would contain the logic to "do the right things" based on whether its running as a PF or VF with that determination easily accomplished by testing the existence of the SR-IOV extended
capability. Then, in an effort to minimize (if not eliminate) the
complexities of driver-to-driver actions for fielding "globalevents",
Quoted 3 lines
contain as much of the logic as is possible within the adapter. Minimizing the efforts required for the device driver writers in my opinion paves the way to greater adoption of this technology.
Quoted 7 lines
Yes, making things easier is the key here.
Perhaps some of this could be hidden with a new bus type for these kinds
of devices? Or a "virtual" bus of pci devices that the original SR-IOV device creates that corrispond to the individual virtual PCI devices? If that were the case, then it might be a lot easier in the end.
I think a standard communication channel in Linux for SR-IOV devices would be a good start, and help to adopt the technology. Something like the virtual bus you are describing. It means that vendors do not need to write their own communication channel in the drivers. It would need to have well defined APIs though, as I guess that devices will have very different capabilities and hardware implementations for PFs and VFs, and so they might have very different events and information to propagate.
Anna
Quoted 30 lines · Greg KH
Quoted 18 lines · Matthew Wilcox
Quoted 17 lines · Greg KH
Quoted 5 lines · H L
I have not modified any existing drivers, but instead I threw together a bare-bones module enabling me to make a call to
pci_iov_register()and then poke at an SR-IOV adapter's /sys entries for which no driver was loaded.Quoted 7 lines
It appears from my perusal thus far that drivers using these new SR-IOV patches will require modification; i.e. the driver associated with the Physical Function (PF) will be required to make the
pci_iov_register()call along with the requisitenotify()function. Essentially this suggests to me a model for the PF driver to perform any "global actions" or setup on behalf of VFs before enabling them after which VF drivers could be associated.Where would the VF drivers have to be associated? On the "
pci_dev" level or on a higher one?Quoted 7 lines
Quoted 2 lines
Will all drivers that want to bind to a "VF" device need to be rewritten?
The current model being implemented by my colleagues has separate drivers for the PF (aka native) and VF devices. I don't personally believe this is the correct path, but I'm reserving judgement until I see some code.
Hm, I would like to see that code before we can properly evaluate this interface. Especially as they are all tightly tied together.
Quoted 8 lines
Quoted 3 lines
I don't think we really know what the One True Usage model is for VF devices. Chris Wright has some ideas, I have some ideas and Yu Zhao has some ideas. I bet there's other people who have other ideas too.
I'd love to hear those ideas.
Rumor has it, there is some Xen code floating around to support this already, is that true?
Xen patches were posted to xen-devel by Yu Zhao on the 29th of September [1]. Unfortunately the only responses that I can find are a) that the patches were mangled and b) they seem to include changes (by others) that have been merged into Linux. I have confirmed that both of these concerns are valid.
I have not yet examined the difference, if any, in the approach taken by Yu to SR-IOV in Linux and Xen. Unfortunately comparison is less than trivial due to the gaping gap in kernel versions between Linux-Xen (2.6.18.8) and Linux itself.
One approach that I was considering in order to familiarise myself with the code was to backport the v6 Linux patches (this thread) to Linux-Xen. I made a start on that, but again due to kernel version differences it is non-trivial.
[1] http://lists.xensource.com/archives/html/xen-devel/2008-09/msg00923.html
Signature · 3 lines
Simon Horman VA Linux Systems Japan K.K., Sydney, Australia Satellite Office H: www.vergenet.net/~horms/ W: www.valinux.co.jp/en
Quoted 30 lines · Greg KH
Quoted 18 lines · Matthew Wilcox
Quoted 17 lines · Greg KH
Quoted 5 lines · H L
I have not modified any existing drivers, but instead I threw together a bare-bones module enabling me to make a call to
pci_iov_register()and then poke at an SR-IOV adapter's /sys entries for which no driver was loaded.Quoted 7 lines
It appears from my perusal thus far that drivers using these new SR-IOV patches will require modification; i.e. the driver associated with the Physical Function (PF) will be required to make the
pci_iov_register()call along with the requisitenotify()function. Essentially this suggests to me a model for the PF driver to perform any "global actions" or setup on behalf of VFs before enabling them after which VF drivers could be associated.Where would the VF drivers have to be associated? On the "
pci_dev" level or on a higher one?Quoted 7 lines
Quoted 2 lines
Will all drivers that want to bind to a "VF" device need to be rewritten?
The current model being implemented by my colleagues has separate drivers for the PF (aka native) and VF devices. I don't personally believe this is the correct path, but I'm reserving judgement until I see some code.
Hm, I would like to see that code before we can properly evaluate this interface. Especially as they are all tightly tied together.
Quoted 8 lines
Quoted 3 lines
I don't think we really know what the One True Usage model is for VF devices. Chris Wright has some ideas, I have some ideas and Yu Zhao has some ideas. I bet there's other people who have other ideas too.
I'd love to hear those ideas.
Rumor has it, there is some Xen code floating around to support this already, is that true?
Xen patches were posted to xen-devel by Yu Zhao on the 29th of September [1]. Unfortunately the only responses that I can find are a) that the patches were mangled and b) they seem to include changes (by others) that have been merged into Linux. I have confirmed that both of these concerns are valid.
I have not yet examined the difference, if any, in the approach taken by Yu to SR-IOV in Linux and Xen. Unfortunately comparison is less than trivial due to the gaping gap in kernel versions between Linux-Xen (2.6.18.8) and Linux itself.
One approach that I was considering in order to familiarise myself with the code was to backport the v6 Linux patches (this thread) to Linux-Xen. I made a start on that, but again due to kernel version differences it is non-trivial.
[1] http://lists.xensource.com/archives/html/xen-devel/2008-09/msg00923.html
Signature · 3 lines
Simon Horman VA Linux Systems Japan K.K., Sydney, Australia Satellite Office H: www.vergenet.net/~horms/ W: www.valinux.co.jp/en
Quoted 15 lines · Matthew Wilcox
[Anna, can you fix your word-wrapping please? Your lines appear to be infinitely long which is most unpleasant to reply to]
Quoted 9 lines · Anna
Quoted 3 lines
Where would the VF drivers have to be associated? On the "
pci_dev" level or on a higher one?A VF appears to the Linux OS as a standard (full, additional) PCI device. The driver is associated in the same way as for a normal PCI device. Ideally, you would use SR-IOV devices on a virtualized system, for example, using Xen. A VF can then be assigned to a guest domain as a full PCI device.
Quoted 5 lines
It's not clear thats the right solution. If the VF devices are _only_ going to be used by the guest, then arguably, we don't want to create
pci_devsfor them in the host. (I think it _is_ the right answer, but I want to make it clear there's multiple opinions on this).
The VFs shouldn't be limited to being used by the guest.
SR-IOV is actually an incredibly painful thing. You need to have a VF driver in the guest, do hardware pass through, have a PV driver stub in the guest that's hypervisor specific (a VF is not usable on it's own), have a device specific backend in the VMM, and if you want to do live migration, have another PV driver in the guest that you can do teaming with. Just a mess.
What we would rather do in KVM, is have the VFs appear in the host as standard network devices. We would then like to back our existing PV driver to this VF directly bypassing the host networking stack. A key feature here is being able to fill the VF's receive queue with guest memory instead of host kernel memory so that you can get zero-copy receive traffic. This will perform just as well as doing passthrough (at least) and avoid all that ugliness of dealing with SR-IOV in the guest.
This eliminates all of the mess of various drivers in the guest and all the associated baggage of doing hardware passthrough.
So IMHO, having VFs be usable in the host is absolutely critical because I think it's the only reasonable usage model.
Regards,
Anthony Liguori
Quoted 15 lines · Matthew Wilcox
[Anna, can you fix your word-wrapping please? Your lines appear to be infinitely long which is most unpleasant to reply to]
Quoted 9 lines · Anna
Quoted 3 lines
Where would the VF drivers have to be associated? On the "
pci_dev" level or on a higher one?A VF appears to the Linux OS as a standard (full, additional) PCI device. The driver is associated in the same way as for a normal PCI device. Ideally, you would use SR-IOV devices on a virtualized system, for example, using Xen. A VF can then be assigned to a guest domain as a full PCI device.
Quoted 5 lines
It's not clear thats the right solution. If the VF devices are _only_ going to be used by the guest, then arguably, we don't want to create
pci_devsfor them in the host. (I think it _is_ the right answer, but I want to make it clear there's multiple opinions on this).
The VFs shouldn't be limited to being used by the guest.
SR-IOV is actually an incredibly painful thing. You need to have a VF driver in the guest, do hardware pass through, have a PV driver stub in the guest that's hypervisor specific (a VF is not usable on it's own), have a device specific backend in the VMM, and if you want to do live migration, have another PV driver in the guest that you can do teaming with. Just a mess.
What we would rather do in KVM, is have the VFs appear in the host as standard network devices. We would then like to back our existing PV driver to this VF directly bypassing the host networking stack. A key feature here is being able to fill the VF's receive queue with guest memory instead of host kernel memory so that you can get zero-copy receive traffic. This will perform just as well as doing passthrough (at least) and avoid all that ugliness of dealing with SR-IOV in the guest.
This eliminates all of the mess of various drivers in the guest and all the associated baggage of doing hardware passthrough.
So IMHO, having VFs be usable in the host is absolutely critical because I think it's the only reasonable usage model.
Regards,
Anthony Liguori
Quoted 9 lines · Greg KH
Quoted 4 lines · Matthew Wilcox
I don't think we really know what the One True Usage model is for VF devices. Chris Wright has some ideas, I have some ideas and Yu Zhao has some ideas. I bet there's other people who have other ideas too.
I'd love to hear those ideas.
We've been talking about avoiding hardware passthrough entirely and just backing a virtio-net backend driver by a dedicated VF in the host. That avoids a huge amount of guest-facing complexity, let's migration Just Work, and should give the same level of performance.
Regards,
Anthony Liguori
Quoted 11 lines
Rumor has it, there is some Xen code floating around to support this already, is that true?
thanks,
greg k-h
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 9 lines · Greg KH
Quoted 4 lines · Matthew Wilcox
I don't think we really know what the One True Usage model is for VF devices. Chris Wright has some ideas, I have some ideas and Yu Zhao has some ideas. I bet there's other people who have other ideas too.
I'd love to hear those ideas.
We've been talking about avoiding hardware passthrough entirely and just backing a virtio-net backend driver by a dedicated VF in the host. That avoids a huge amount of guest-facing complexity, let's migration Just Work, and should give the same level of performance.
Regards,
Anthony Liguori
Quoted 11 lines
Rumor has it, there is some Xen code floating around to support this already, is that true?
thanks,
greg k-h
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: git repository for SR-IOV development?
In reply to H LQuoted 14 lines · H L
Quoted 10 lines · Greg KH
Quoted 2 lines · H L
Has anyone initiated or given consideration to the
creation of a git
Quoted 1 line
repository (say, on kernel.org) for SR-IOV
development?
Why? It's only a few patches, right? Why would it need a whole new git
tree?
Quoted 8 lines
So as to minimize the time and effort patching a kernel, especially if the tree (and/or hash level) against which the patches were created fails to be specified on a mailing-list. Plus, there appears to be questions raised on how, precisely, the implementation should ultimately be modeled and especially given that, who knows at this point what number of patches will ultimately be submitted? I know I've built the "7-patch" one (painfully, by the way), and I'm aware there's another 15-patch set out there which I've not yet examined.
FWIW, the v6 patch series (this thread) applied to both 2.6.28-rc3 and the current Linus tree after a minor tweak to the first patch, as below.
Signature · 31 lines
Simon Horman VA Linux Systems Japan K.K., Sydney, Australia Satellite Office H: www.vergenet.net/~horms/ W: www.valinux.co.jp/en From: Yu Zhao <[email protected]> [PATCH 1/16 v6] PCI: remove unnecessary arg of pci_update_resource() This cleanup removes unnecessary argument 'struct resource *res' in pci_update_resource(), so it takes same arguments as other companion functions (pci_assign_resource(), etc.). Cc: Alex Chiang <[email protected]> Cc: Grant Grundler <[email protected]> Cc: Greg KH <[email protected]> Cc: Ingo Molnar <[email protected]> Cc: Jesse Barnes <[email protected]> Cc: Matthew Wilcox <[email protected]> Cc: Randy Dunlap <[email protected]> Cc: Roland Dreier <[email protected]> Signed-off-by: Yu Zhao <[email protected]> Upported-by: Simon Horman <[email protected]> --- drivers/pci/pci.c | 4 ++-- drivers/pci/setup-res.c | 7 ++++--- include/linux/pci.h | 2 +- 3 files changed, 7 insertions(+), 6 deletions(-) * Fri, 07 Nov 2008 09:05:18 +1100, Simon Horman - Minor rediff of include/linux/pci.h section to apply to 2.6.28-rc3
drivers/pci/pci.c +2 -2
@@ -376,8 +376,8 @@ pci_restore_bars(struct pci_dev *dev)return ; } -for (i =0 ; i < numres; i ++) -pci_update_resource (dev, &dev->resource[i], i); +for (i =0 ; i < numres; i++) +pci_update_resource (dev, i); }static struct pci_platform_pm_ops *pci_platform_pm;
drivers/pci/setup-res.c +4 -3
@@ -26,11 +26,12 @@#include "pci.h" -void pci_update_resource (struct pci_dev *dev,struct resource *res,int resno) +void pci_update_resource (struct pci_dev *dev,int resno) {struct pci_bus_region region;u32 new, check, mask;int reg; +struct resource *res = dev->resource + resno;/* * Ignore resources for unimplemented BARs and unused resource slots
@@ -162,7 +163,7 @@ int pci_assign_resource(struct pci_dev *dev, int resno) }else { res->flags &= ~IORESOURCE_STARTALIGN ;if (resno <PCI_BRIDGE_RESOURCES ) -pci_update_resource (dev, res, resno); +pci_update_resource (dev, resno); }return ret;
@@ -197,7 +198,7 @@ int pci_assign_resource_fixed(struct pci_dev *dev, int resno)dev_err (&dev->dev,"BAR %d: can't allocate %s resource %pR\n" , resno, res->flags &IORESOURCE_IO ?"I/O" :"mem" , res); }else if (resno <PCI_BRIDGE_RESOURCES ) { -pci_update_resource (dev, res, resno); +pci_update_resource (dev, resno); }return ret;
include/linux/pci.h +1 -1
@@ -626,7 +626,7 @@ int pcix_get_mmrbc(struct pci_dev *dev);int pcie_set_readrq (struct pci_dev *dev,int rq);int pci_reset_function (struct pci_dev *dev);int pci_execute_reset_function (struct pci_dev *dev); -void pci_update_resource (struct pci_dev *dev,struct resource *res,int resno); +void pci_update_resource (struct pci_dev *dev,int resno);int __must_check pci_assign_resource (struct pci_dev *dev,int i);int pci_select_bars (struct pci_dev *dev,unsigned long flags);
Signature · 1 line
1.5.6.4
Re: git repository for SR-IOV development?
In reply to H LQuoted 14 lines · H L
Quoted 10 lines · Greg KH
Quoted 2 lines · H L
Has anyone initiated or given consideration to the
creation of a git
Quoted 1 line
repository (say, on kernel.org) for SR-IOV
development?
Why? It's only a few patches, right? Why would it need a whole new git
tree?
Quoted 8 lines
So as to minimize the time and effort patching a kernel, especially if the tree (and/or hash level) against which the patches were created fails to be specified on a mailing-list. Plus, there appears to be questions raised on how, precisely, the implementation should ultimately be modeled and especially given that, who knows at this point what number of patches will ultimately be submitted? I know I've built the "7-patch" one (painfully, by the way), and I'm aware there's another 15-patch set out there which I've not yet examined.
FWIW, the v6 patch series (this thread) applied to both 2.6.28-rc3 and the current Linus tree after a minor tweak to the first patch, as below.
Signature · 31 lines
Simon Horman VA Linux Systems Japan K.K., Sydney, Australia Satellite Office H: www.vergenet.net/~horms/ W: www.valinux.co.jp/en From: Yu Zhao <[email protected]> [PATCH 1/16 v6] PCI: remove unnecessary arg of pci_update_resource() This cleanup removes unnecessary argument 'struct resource *res' in pci_update_resource(), so it takes same arguments as other companion functions (pci_assign_resource(), etc.). Cc: Alex Chiang <[email protected]> Cc: Grant Grundler <[email protected]> Cc: Greg KH <[email protected]> Cc: Ingo Molnar <[email protected]> Cc: Jesse Barnes <[email protected]> Cc: Matthew Wilcox <[email protected]> Cc: Randy Dunlap <[email protected]> Cc: Roland Dreier <[email protected]> Signed-off-by: Yu Zhao <[email protected]> Upported-by: Simon Horman <[email protected]> --- drivers/pci/pci.c | 4 ++-- drivers/pci/setup-res.c | 7 ++++--- include/linux/pci.h | 2 +- 3 files changed, 7 insertions(+), 6 deletions(-) * Fri, 07 Nov 2008 09:05:18 +1100, Simon Horman - Minor rediff of include/linux/pci.h section to apply to 2.6.28-rc3
drivers/pci/pci.c +2 -2
@@ -376,8 +376,8 @@ pci_restore_bars(struct pci_dev *dev)return ; } -for (i =0 ; i < numres; i ++) -pci_update_resource (dev, &dev->resource[i], i); +for (i =0 ; i < numres; i++) +pci_update_resource (dev, i); }static struct pci_platform_pm_ops *pci_platform_pm;
drivers/pci/setup-res.c +4 -3
@@ -26,11 +26,12 @@#include "pci.h" -void pci_update_resource (struct pci_dev *dev,struct resource *res,int resno) +void pci_update_resource (struct pci_dev *dev,int resno) {struct pci_bus_region region;u32 new, check, mask;int reg; +struct resource *res = dev->resource + resno;/* * Ignore resources for unimplemented BARs and unused resource slots
@@ -162,7 +163,7 @@ int pci_assign_resource(struct pci_dev *dev, int resno) }else { res->flags &= ~IORESOURCE_STARTALIGN ;if (resno <PCI_BRIDGE_RESOURCES ) -pci_update_resource (dev, res, resno); +pci_update_resource (dev, resno); }return ret;
@@ -197,7 +198,7 @@ int pci_assign_resource_fixed(struct pci_dev *dev, int resno)dev_err (&dev->dev,"BAR %d: can't allocate %s resource %pR\n" , resno, res->flags &IORESOURCE_IO ?"I/O" :"mem" , res); }else if (resno <PCI_BRIDGE_RESOURCES ) { -pci_update_resource (dev, res, resno); +pci_update_resource (dev, resno); }return ret;
include/linux/pci.h +1 -1
@@ -626,7 +626,7 @@ int pcix_get_mmrbc(struct pci_dev *dev);int pcie_set_readrq (struct pci_dev *dev,int rq);int pci_reset_function (struct pci_dev *dev);int pci_execute_reset_function (struct pci_dev *dev); -void pci_update_resource (struct pci_dev *dev,struct resource *res,int resno); +void pci_update_resource (struct pci_dev *dev,int resno);int __must_check pci_assign_resource (struct pci_dev *dev,int i);int pci_select_bars (struct pci_dev *dev,unsigned long flags);
Signature · 1 line
1.5.6.4
Quoted 5 lines · Anthony Liguori
Quoted 4 lines
It's not clear thats the right solution. If the VF devices are _only_ going to be used by the guest, then arguably, we don't want to create
pci_devsfor them in the host. (I think it _is_ the right answer, but I want to make it clear there's multiple opinions on this).
Quoted 8 lines
The VFs shouldn't be limited to being used by the guest.
SR-IOV is actually an incredibly painful thing. You need to have a VF driver in the guest, do hardware pass through, have a PV driver stub in the guest that's hypervisor specific (a VF is not usable on it's own), have a device specific backend in the VMM, and if you want to do live migration, have another PV driver in the guest that you can do teaming with. Just a mess.
Not to mention that you basically have to statically allocate them up front.
Quoted 7 lines
What we would rather do in KVM, is have the VFs appear in the host as standard network devices. We would then like to back our existing PV driver to this VF directly bypassing the host networking stack. A key feature here is being able to fill the VF's receive queue with guest memory instead of host kernel memory so that you can get zero-copy receive traffic. This will perform just as well as doing passthrough (at least) and avoid all that ugliness of dealing with SR-IOV in the guest.
This argues for ignoring the SR-IOV mess completely. Just have the host driver expose multiple 'ethN' devices.
Quoted 13 lines
This eliminates all of the mess of various drivers in the guest and all the associated baggage of doing hardware passthrough.
So IMHO, having VFs be usable in the host is absolutely critical because I think it's the only reasonable usage model.
Regards,
Anthony Liguori
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Signature · 4 lines
Matthew Wilcox Intel Open Source Technology Centre "Bill, look, we understand that you're interested in selling us this operating system, but compare it to ours. We can't possibly take such a retrograde step."
Quoted 5 lines · Anthony Liguori
Quoted 4 lines
It's not clear thats the right solution. If the VF devices are _only_ going to be used by the guest, then arguably, we don't want to create
pci_devsfor them in the host. (I think it _is_ the right answer, but I want to make it clear there's multiple opinions on this).
Quoted 8 lines
The VFs shouldn't be limited to being used by the guest.
SR-IOV is actually an incredibly painful thing. You need to have a VF driver in the guest, do hardware pass through, have a PV driver stub in the guest that's hypervisor specific (a VF is not usable on it's own), have a device specific backend in the VMM, and if you want to do live migration, have another PV driver in the guest that you can do teaming with. Just a mess.
Not to mention that you basically have to statically allocate them up front.
Quoted 7 lines
What we would rather do in KVM, is have the VFs appear in the host as standard network devices. We would then like to back our existing PV driver to this VF directly bypassing the host networking stack. A key feature here is being able to fill the VF's receive queue with guest memory instead of host kernel memory so that you can get zero-copy receive traffic. This will perform just as well as doing passthrough (at least) and avoid all that ugliness of dealing with SR-IOV in the guest.
This argues for ignoring the SR-IOV mess completely. Just have the host driver expose multiple 'ethN' devices.
Quoted 13 lines
This eliminates all of the mess of various drivers in the guest and all the associated baggage of doing hardware passthrough.
So IMHO, having VFs be usable in the host is absolutely critical because I think it's the only reasonable usage model.
Regards,
Anthony Liguori
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Signature · 4 lines
Matthew Wilcox Intel Open Source Technology Centre "Bill, look, we understand that you're interested in selling us this operating system, but compare it to ours. We can't possibly take such a retrograde step."
Quoted 27 lines · * Greg KH ([email protected]) wrote:
Quoted 18 lines · Matthew Wilcox
Quoted 17 lines · Greg KH
Quoted 5 lines · H L
I have not modified any existing drivers, but instead I threw together a bare-bones module enabling me to make a call to
pci_iov_register()and then poke at an SR-IOV adapter's /sys entries for which no driver was loaded.Quoted 7 lines
It appears from my perusal thus far that drivers using these new SR-IOV patches will require modification; i.e. the driver associated with the Physical Function (PF) will be required to make the
pci_iov_register()call along with the requisitenotify()function. Essentially this suggests to me a model for the PF driver to perform any "global actions" or setup on behalf of VFs before enabling them after which VF drivers could be associated.Where would the VF drivers have to be associated? On the "
pci_dev" level or on a higher one?Quoted 7 lines
Quoted 2 lines
Will all drivers that want to bind to a "VF" device need to be rewritten?
The current model being implemented by my colleagues has separate drivers for the PF (aka native) and VF devices. I don't personally believe this is the correct path, but I'm reserving judgement until I see some code.
Quoted 8 lines
Hm, I would like to see that code before we can properly evaluate this interface. Especially as they are all tightly tied together.
Quoted 3 lines
I don't think we really know what the One True Usage model is for VF devices. Chris Wright has some ideas, I have some ideas and Yu Zhao has some ideas. I bet there's other people who have other ideas too.
I'd love to hear those ideas.
First there's the question of how to represent the VF on the host. Ideally (IMO) this would show up as a normal interface so that normal tools can configure the interface. This is not exactly how the first round of patches were designed.
Second there's the question of reserving the BDF on the host such that we don't have two drivers (one in the host and one in a guest) trying to drive the same device (an issue that shows up for device assignment as well as VF assignment).
Third there's the question of whether the VF can be used in the host at all.
Fourth there's the question of whether the VF and PF drivers are the same or separate.
The typical usecase is assigning the VF to the guest directly, so there's only enough functionality in the host side to allocate a VF, configure it, and assign it (and propagate AER). This is with separate PF and VF driver.
As Anthony mentioned, we are interested in allowing the host to use the VF. This could be useful for containers as well as dedicating a VF (a set of device resources) to a guest w/out passing it through.
thanks,
-chris
Quoted 27 lines · * Greg KH ([email protected]) wrote:
Quoted 18 lines · Matthew Wilcox
Quoted 17 lines · Greg KH
Quoted 5 lines · H L
I have not modified any existing drivers, but instead I threw together a bare-bones module enabling me to make a call to
pci_iov_register()and then poke at an SR-IOV adapter's /sys entries for which no driver was loaded.Quoted 7 lines
It appears from my perusal thus far that drivers using these new SR-IOV patches will require modification; i.e. the driver associated with the Physical Function (PF) will be required to make the
pci_iov_register()call along with the requisitenotify()function. Essentially this suggests to me a model for the PF driver to perform any "global actions" or setup on behalf of VFs before enabling them after which VF drivers could be associated.Where would the VF drivers have to be associated? On the "
pci_dev" level or on a higher one?Quoted 7 lines
Quoted 2 lines
Will all drivers that want to bind to a "VF" device need to be rewritten?
The current model being implemented by my colleagues has separate drivers for the PF (aka native) and VF devices. I don't personally believe this is the correct path, but I'm reserving judgement until I see some code.
Quoted 8 lines
Hm, I would like to see that code before we can properly evaluate this interface. Especially as they are all tightly tied together.
Quoted 3 lines
I don't think we really know what the One True Usage model is for VF devices. Chris Wright has some ideas, I have some ideas and Yu Zhao has some ideas. I bet there's other people who have other ideas too.
I'd love to hear those ideas.
First there's the question of how to represent the VF on the host. Ideally (IMO) this would show up as a normal interface so that normal tools can configure the interface. This is not exactly how the first round of patches were designed.
Second there's the question of reserving the BDF on the host such that we don't have two drivers (one in the host and one in a guest) trying to drive the same device (an issue that shows up for device assignment as well as VF assignment).
Third there's the question of whether the VF can be used in the host at all.
Fourth there's the question of whether the VF and PF drivers are the same or separate.
The typical usecase is assigning the VF to the guest directly, so there's only enough functionality in the host side to allocate a VF, configure it, and assign it (and propagate AER). This is with separate PF and VF driver.
As Anthony mentioned, we are interested in allowing the host to use the VF. This could be useful for containers as well as dedicating a VF (a set of device resources) to a guest w/out passing it through.
thanks,
-chris
Quoted 12 lines
What we would rather do in KVM, is have the VFs appear in the host as standard network devices. We would then like to back our existing PV driver to this VF directly bypassing the host networking stack. A key feature here is being able to fill the VF's receive queue with guest memory instead of host kernel memory so that you can get zero-copy
receive traffic. This will perform just as well as doing passthrough (at least) and avoid all that ugliness of dealing with SR-IOV in the guest.
Anthony:
This is already addressed by VMDq solution(or so called netchannel2), right? Qing He is debugging the KVM side patch and pretty much close to end.
For this single purpose, we don't need SR-IOV. BTW at least Intel SR-IOV NIC also supports VMDq, so you can achieve this by simply use "native" VMDq enabled driver here, plus the work we are debugging now.
Thx, eddie
Quoted 12 lines
What we would rather do in KVM, is have the VFs appear in the host as standard network devices. We would then like to back our existing PV driver to this VF directly bypassing the host networking stack. A key feature here is being able to fill the VF's receive queue with guest memory instead of host kernel memory so that you can get zero-copy
receive traffic. This will perform just as well as doing passthrough (at least) and avoid all that ugliness of dealing with SR-IOV in the guest.
Anthony:
This is already addressed by VMDq solution(or so called netchannel2), right? Qing He is debugging the KVM side patch and pretty much close to end.
For this single purpose, we don't need SR-IOV. BTW at least Intel SR-IOV NIC also supports VMDq, so you can achieve this by simply use "native" VMDq enabled driver here, plus the work we are debugging now.
Thx, eddie
Re: git repository for SR-IOV development?
In reply to H LQuoted 14 lines · H L
Quoted 10 lines · Greg KH
Quoted 2 lines · H L
Has anyone initiated or given consideration to the
creation of a git
Quoted 1 line
repository (say, on kernel.org) for SR-IOV
development?
Why? It's only a few patches, right? Why would it need a whole new git
tree?
Quoted 8 lines
So as to minimize the time and effort patching a kernel, especially if the tree (and/or hash level) against which the patches were created fails to be specified on a mailing-list. Plus, there appears to be questions raised on how, precisely, the implementation should ultimately be modeled and especially given that, who knows at this point what number of patches will ultimately be submitted? I know I've built the "7-patch" one (painfully, by the way), and I'm aware there's another 15-patch set out there which I've not yet examined.
It's a mere 7 or 15 patches, you don't need a whole git tree for something small like that.
Especially as there only seems to be one developer doing real work...
thanks,
greg k-h
Re: git repository for SR-IOV development?
In reply to H LQuoted 14 lines · H L
Quoted 10 lines · Greg KH
Quoted 2 lines · H L
Has anyone initiated or given consideration to the
creation of a git
Quoted 1 line
repository (say, on kernel.org) for SR-IOV
development?
Why? It's only a few patches, right? Why would it need a whole new git
tree?
Quoted 8 lines
So as to minimize the time and effort patching a kernel, especially if the tree (and/or hash level) against which the patches were created fails to be specified on a mailing-list. Plus, there appears to be questions raised on how, precisely, the implementation should ultimately be modeled and especially given that, who knows at this point what number of patches will ultimately be submitted? I know I've built the "7-patch" one (painfully, by the way), and I'm aware there's another 15-patch set out there which I've not yet examined.
It's a mere 7 or 15 patches, you don't need a whole git tree for something small like that.
Especially as there only seems to be one developer doing real work...
thanks,
greg k-h
Quoted 22 lines · Anthony Liguori
Quoted 15 lines · Matthew Wilcox
[Anna, can you fix your word-wrapping please? Your lines appear to be infinitely long which is most unpleasant to reply to]
Quoted 9 lines · Anna
Quoted 3 lines
Where would the VF drivers have to be associated? On the "
pci_dev" level or on a higher one?A VF appears to the Linux OS as a standard (full, additional) PCI device. The driver is associated in the same way as for a normal PCI device. Ideally, you would use SR-IOV devices on a virtualized system, for example, using Xen. A VF can then be assigned to a guest domain as a full PCI device.
Quoted 5 lines
It's not clear thats the right solution. If the VF devices are _only_ going to be used by the guest, then arguably, we don't want to create
pci_devsfor them in the host. (I think it _is_ the right answer, but I want to make it clear there's multiple opinions on this).
Quoted 8 lines
The VFs shouldn't be limited to being used by the guest.
SR-IOV is actually an incredibly painful thing. You need to have a VF driver in the guest, do hardware pass through, have a PV driver stub in the guest that's hypervisor specific (a VF is not usable on it's own), have a device specific backend in the VMM, and if you want to do live migration, have another PV driver in the guest that you can do teaming with. Just a mess.
Actually "a PV driver stub in the guest" _was_ correct; I admit that I stated so at a virt mini summit more than a half year ago ;-). But the things have changed, and such a stub is no longer required (at least in our implementation). The major benefit of VF drivers now is that they are VMM-agnostic.
Quoted 10 lines
What we would rather do in KVM, is have the VFs appear in the host as standard network devices. We would then like to back our existing PV driver to this VF directly bypassing the host networking stack. A key feature here is being able to fill the VF's receive queue with guest memory instead of host kernel memory so that you can get zero-copy receive traffic. This will perform just as well as doing passthrough (at
least) and avoid all that ugliness of dealing with SR-IOV in the guest.
Quoted 5 lines
This eliminates all of the mess of various drivers in the guest and all the associated baggage of doing hardware passthrough.
So IMHO, having VFs be usable in the host is absolutely critical because I think it's the only reasonable usage model.
As Eddie said, VMDq is better for this model, and the feature is already available today. It is much simpler because it was designed for such purposes. It does not require hardware pass-through (e.g. VT-d) or VFs as a PCI device, either.
Quoted 8 lines
Regards,
Anthony Liguori
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected] More majordomo info at
http://vger.kernel.org/majordomo-info.html
.
Jun Nakajima | Intel Open Source Technology Center
Quoted 22 lines · Anthony Liguori
Quoted 15 lines · Matthew Wilcox
[Anna, can you fix your word-wrapping please? Your lines appear to be infinitely long which is most unpleasant to reply to]
Quoted 9 lines · Anna
Quoted 3 lines
Where would the VF drivers have to be associated? On the "
pci_dev" level or on a higher one?A VF appears to the Linux OS as a standard (full, additional) PCI device. The driver is associated in the same way as for a normal PCI device. Ideally, you would use SR-IOV devices on a virtualized system, for example, using Xen. A VF can then be assigned to a guest domain as a full PCI device.
Quoted 5 lines
It's not clear thats the right solution. If the VF devices are _only_ going to be used by the guest, then arguably, we don't want to create
pci_devsfor them in the host. (I think it _is_ the right answer, but I want to make it clear there's multiple opinions on this).
Quoted 8 lines
The VFs shouldn't be limited to being used by the guest.
SR-IOV is actually an incredibly painful thing. You need to have a VF driver in the guest, do hardware pass through, have a PV driver stub in the guest that's hypervisor specific (a VF is not usable on it's own), have a device specific backend in the VMM, and if you want to do live migration, have another PV driver in the guest that you can do teaming with. Just a mess.
Actually "a PV driver stub in the guest" _was_ correct; I admit that I stated so at a virt mini summit more than a half year ago ;-). But the things have changed, and such a stub is no longer required (at least in our implementation). The major benefit of VF drivers now is that they are VMM-agnostic.
Quoted 10 lines
What we would rather do in KVM, is have the VFs appear in the host as standard network devices. We would then like to back our existing PV driver to this VF directly bypassing the host networking stack. A key feature here is being able to fill the VF's receive queue with guest memory instead of host kernel memory so that you can get zero-copy receive traffic. This will perform just as well as doing passthrough (at
least) and avoid all that ugliness of dealing with SR-IOV in the guest.
Quoted 5 lines
This eliminates all of the mess of various drivers in the guest and all the associated baggage of doing hardware passthrough.
So IMHO, having VFs be usable in the host is absolutely critical because I think it's the only reasonable usage model.
As Eddie said, VMDq is better for this model, and the feature is already available today. It is much simpler because it was designed for such purposes. It does not require hardware pass-through (e.g. VT-d) or VFs as a PCI device, either.
Quoted 8 lines
Regards,
Anthony Liguori
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected] More majordomo info at
http://vger.kernel.org/majordomo-info.html
.
Jun Nakajima | Intel Open Source Technology Center
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
In reply to Greg KHQuoted 29 lines · Greg KH
Quoted 11 lines · Yu Zhao
Documentation/kernel-parameters.txt | 10 ++++++++++ 1 files changed, 10 insertions(+), 0 deletions(-)Documentation/kernel-parameters.txt +0
-0@@ -1677,6 +1677,16 @@ and is between 256 and 4096 characters. It is defined in the file cbmemsize=nn[KMG] The fixed amount of bus space which is reserved for the CardBus bridge's memory window. The default value is 64 megabytes.Documentation/kernel-parameters.txt and is between 256 and 4096 characters. It is defined in the file · line 1689
Quoted 10 lines
Documentation/kernel-parameters.txt +10
-0+ assign-mmio=[dddd:]bb [X86] reassign memory resources of all + devices under bus [dddd:]bb (dddd is the domain + number and bb is the bus number). + assign-pio=[dddd:]bb [X86] reassign io port resources of all + devices under bus [dddd:]bb (dddd is the domain + number and bb is the bus number). + align-mmio=[dddd:]bb:dd.f [X86] relocate memory resources of a + device to minimum PAGE_SIZE alignment (dddd is + the domain number and bb, dd and f is the bus, + device and function number).This seems like a big problem. How are we going to know to add these command line options for devices we haven't even seen/known about yet?
How do we know the bus ids aren't going to change between boots (hint, they are, pci bus ids change all the time...)
Quoted 6 lines
We need to be able to do this kind of thing dynamically, not fixed at boot time, which seems way to early to even know about this, right?
thanks,
greg k-h
Yes, I totally agree. Doing things dynamically is better.
The purpose of these parameters is to rebalance and align resources for device that has BARs encapsulated in various new capabilities (SR-IOV, etc.), because most of existing BIOSes don't take care of those BARs.
If we do resource rebalance after system is up, do you think there is any side effect or impact to other subsystem other than PCI (e.g. MTRR)?
I haven't had much thinking on the dynamical resource rebalance. If you have any idea about this, can you please suggest?
Regards,
Yu
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
In reply to Greg KHQuoted 29 lines · Greg KH
Quoted 11 lines · Yu Zhao
Documentation/kernel-parameters.txt | 10 ++++++++++ 1 files changed, 10 insertions(+), 0 deletions(-)Documentation/kernel-parameters.txt +0
-0@@ -1677,6 +1677,16 @@ and is between 256 and 4096 characters. It is defined in the file cbmemsize=nn[KMG] The fixed amount of bus space which is reserved for the CardBus bridge's memory window. The default value is 64 megabytes.Documentation/kernel-parameters.txt and is between 256 and 4096 characters. It is defined in the file · line 1689
Quoted 10 lines
Documentation/kernel-parameters.txt +10
-0+ assign-mmio=[dddd:]bb [X86] reassign memory resources of all + devices under bus [dddd:]bb (dddd is the domain + number and bb is the bus number). + assign-pio=[dddd:]bb [X86] reassign io port resources of all + devices under bus [dddd:]bb (dddd is the domain + number and bb is the bus number). + align-mmio=[dddd:]bb:dd.f [X86] relocate memory resources of a + device to minimum PAGE_SIZE alignment (dddd is + the domain number and bb, dd and f is the bus, + device and function number).This seems like a big problem. How are we going to know to add these command line options for devices we haven't even seen/known about yet?
How do we know the bus ids aren't going to change between boots (hint, they are, pci bus ids change all the time...)
Quoted 6 lines
We need to be able to do this kind of thing dynamically, not fixed at boot time, which seems way to early to even know about this, right?
thanks,
greg k-h
Yes, I totally agree. Doing things dynamically is better.
The purpose of these parameters is to rebalance and align resources for device that has BARs encapsulated in various new capabilities (SR-IOV, etc.), because most of existing BIOSes don't take care of those BARs.
If we do resource rebalance after system is up, do you think there is any side effect or impact to other subsystem other than PCI (e.g. MTRR)?
I haven't had much thinking on the dynamical resource rebalance. If you have any idea about this, can you please suggest?
Regards,
Yu
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
In reply to Zhao, YuQuoted 34 lines · Yu
Quoted 24 lines · Greg KH
Quoted 13 lines · Yu Zhao
Documentation/kernel-parameters.txt | 10 ++++++++++ 1 files changed, 10 insertions(+), 0 deletions(-)Documentation/kernel-parameters.txt +0
-0@@ -1677,6 +1677,16 @@ and is between 256 and 4096 characters. It isdefined in the file
Documentation/kernel-parameters.txt +0
-0cbmemsize=nn[KMG] The fixed amount of bus space which is reserved for the CardBus bridge's memory window. The default value is 64 megabytes.Documentation/kernel-parameters.txt and is between 256 and 4096 characters. It is defined in the file · line 1689
Quoted 10 lines
Documentation/kernel-parameters.txt +10
-0+ assign-mmio=[dddd:]bb [X86] reassign memory resources of all + devices under bus [dddd:]bb (dddd is the domain + number and bb is the bus number). + assign-pio=[dddd:]bb [X86] reassign io port resources of all + devices under bus [dddd:]bb (dddd is the domain + number and bb is the bus number). + align-mmio=[dddd:]bb:dd.f [X86] relocate memory resources of a + device to minimum PAGE_SIZE alignment (dddd is + the domain number and bb, dd and f is the bus, + device and function number).Quoted 8 lines
This seems like a big problem. How are we going to know to add these command line options for devices we haven't even seen/known about yet? How do we know the bus ids aren't going to change between boots (hint, they are, pci bus ids change all the time...)
We need to be able to do this kind of thing dynamically, not fixed at boot time, which seems way to early to even know about this, right? thanks,
greg k-h
Quoted 5 lines
Yes, I totally agree. Doing things dynamically is better.
The purpose of these parameters is to rebalance and align resources for device that has BARs encapsulated in various new capabilities (SR-IOV, etc.), because most of existing BIOSes don't take care of those BARs.
But how are you going to know what the proper device ids are going to be before the machine boots? I don't see how these options are ever going to work properly for a "real" user.
Quoted 2 lines
If we do resource rebalance after system is up, do you think there is any side effect or impact to other subsystem other than PCI (e.g. MTRR)?
I don't think so.
Quoted 2 lines
I haven't had much thinking on the dynamical resource rebalance. If you have any idea about this, can you please suggest?
Yeah, it's going to be hard :)
We've thought about this in the past, and even Microsoft said it was going to happen for Vista, but they realized in the end, like we did a few years previously, that it would require full support of all PCI drivers as well (if you rebalance stuff that is already bound to a driver.) So they dropped it.
When would you want to do this kind of rebalancing? Before any PCI driver is bound to any devices? Or afterwards?
thanks,
greg k-h
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
In reply to Zhao, YuQuoted 34 lines · Yu
Quoted 24 lines · Greg KH
Quoted 13 lines · Yu Zhao
Documentation/kernel-parameters.txt | 10 ++++++++++ 1 files changed, 10 insertions(+), 0 deletions(-)Documentation/kernel-parameters.txt +0
-0@@ -1677,6 +1677,16 @@ and is between 256 and 4096 characters. It isdefined in the file
Documentation/kernel-parameters.txt +0
-0cbmemsize=nn[KMG] The fixed amount of bus space which is reserved for the CardBus bridge's memory window. The default value is 64 megabytes.Documentation/kernel-parameters.txt and is between 256 and 4096 characters. It is defined in the file · line 1689
Quoted 10 lines
Documentation/kernel-parameters.txt +10
-0+ assign-mmio=[dddd:]bb [X86] reassign memory resources of all + devices under bus [dddd:]bb (dddd is the domain + number and bb is the bus number). + assign-pio=[dddd:]bb [X86] reassign io port resources of all + devices under bus [dddd:]bb (dddd is the domain + number and bb is the bus number). + align-mmio=[dddd:]bb:dd.f [X86] relocate memory resources of a + device to minimum PAGE_SIZE alignment (dddd is + the domain number and bb, dd and f is the bus, + device and function number).Quoted 8 lines
This seems like a big problem. How are we going to know to add these command line options for devices we haven't even seen/known about yet? How do we know the bus ids aren't going to change between boots (hint, they are, pci bus ids change all the time...)
We need to be able to do this kind of thing dynamically, not fixed at boot time, which seems way to early to even know about this, right? thanks,
greg k-h
Quoted 5 lines
Yes, I totally agree. Doing things dynamically is better.
The purpose of these parameters is to rebalance and align resources for device that has BARs encapsulated in various new capabilities (SR-IOV, etc.), because most of existing BIOSes don't take care of those BARs.
But how are you going to know what the proper device ids are going to be before the machine boots? I don't see how these options are ever going to work properly for a "real" user.
Quoted 2 lines
If we do resource rebalance after system is up, do you think there is any side effect or impact to other subsystem other than PCI (e.g. MTRR)?
I don't think so.
Quoted 2 lines
I haven't had much thinking on the dynamical resource rebalance. If you have any idea about this, can you please suggest?
Yeah, it's going to be hard :)
We've thought about this in the past, and even Microsoft said it was going to happen for Vista, but they realized in the end, like we did a few years previously, that it would require full support of all PCI drivers as well (if you rebalance stuff that is already bound to a driver.) So they dropped it.
When would you want to do this kind of rebalancing? Before any PCI driver is bound to any devices? Or afterwards?
thanks,
greg k-h
Re: [PATCH 15/16 v6] PCI: document the SR-IOV sysfs entries
In reply to Greg KHQuoted 25 lines · Greg KH
Quoted 4 lines · Greg KH
Quoted 3 lines · Yu Zhao
Documentation/ABI/testing/sysfs-bus-pci | 33 +++++++++++++++++++++++++++++++ 1 files changed, 33 insertions(+), 0 deletions(-)Quoted 12 lines
Documentation/ABI/testing/sysfs-bus-pci Description: · line 13
Quoted 10 lines
Documentation/ABI/testing/sysfs-bus-pci +2
-0@@ -9,3 +9,36 @@ Description: that some devices may have malformatted data. If the underlying VPD has a writable section then the corresponding section of this file will be writable. + +What: /sys/bus/pci/devices/.../iov/enableAre you sure this is still the correct location with your change to struct device?
Nevermind, this is correct.
But the bigger problem is that userspace doesn't know when these attributes show up. So tools like udev and HAL and others can't look for them as they never get notified, and they don't even know if they should be looking for them or not.
Quoted 7 lines
Is there any way to tie these attributes to the "main" pci device so that they get created before the device is announced to the world? Doing that would solve this issue.
thanks,
greg k-h
Currently PCI subsystem has /sys/.../{vendor,device,...} bundled to the main PCI device (I suppose this means the entries are created by 'device_add')
And after the PCI device is announced, /sys/.../{config,resourceX,rom,vpd,iov,...} get created depending on if these features are supported.
Making dynamic entries tie to the main PCI device would require PCI subsystem to allocate different 'bus_type' for the devices, right?
Regards,
Yu
Re: [PATCH 15/16 v6] PCI: document the SR-IOV sysfs entries
In reply to Greg KHQuoted 25 lines · Greg KH
Quoted 4 lines · Greg KH
Quoted 3 lines · Yu Zhao
Documentation/ABI/testing/sysfs-bus-pci | 33 +++++++++++++++++++++++++++++++ 1 files changed, 33 insertions(+), 0 deletions(-)Quoted 12 lines
Documentation/ABI/testing/sysfs-bus-pci Description: · line 13
Quoted 10 lines
Documentation/ABI/testing/sysfs-bus-pci +2
-0@@ -9,3 +9,36 @@ Description: that some devices may have malformatted data. If the underlying VPD has a writable section then the corresponding section of this file will be writable. + +What: /sys/bus/pci/devices/.../iov/enableAre you sure this is still the correct location with your change to struct device?
Nevermind, this is correct.
But the bigger problem is that userspace doesn't know when these attributes show up. So tools like udev and HAL and others can't look for them as they never get notified, and they don't even know if they should be looking for them or not.
Quoted 7 lines
Is there any way to tie these attributes to the "main" pci device so that they get created before the device is announced to the world? Doing that would solve this issue.
thanks,
greg k-h
Currently PCI subsystem has /sys/.../{vendor,device,...} bundled to the main PCI device (I suppose this means the entries are created by 'device_add')
And after the PCI device is announced, /sys/.../{config,resourceX,rom,vpd,iov,...} get created depending on if these features are supported.
Making dynamic entries tie to the main PCI device would require PCI subsystem to allocate different 'bus_type' for the devices, right?
Regards,
Yu
Re: [PATCH 15/16 v6] PCI: document the SR-IOV sysfs entries
In reply to Zhao, YuQuoted 31 lines · Yu
Quoted 19 lines · Greg KH
Quoted 5 lines · Greg KH
Quoted 4 lines · Yu Zhao
Documentation/ABI/testing/sysfs-bus-pci | 33 +++++++++++++++++++++++++++++++ 1 files changed, 33 insertions(+), 0 deletions(-)Quoted 13 lines
Documentation/ABI/testing/sysfs-bus-pci Description: · line 13
Quoted 11 lines
Documentation/ABI/testing/sysfs-bus-pci +2
-0@@ -9,3 +9,36 @@ Description: that some devices may have malformatted data. If the underlying VPD has a writable section then the corresponding section of this file will be writable. + +What: /sys/bus/pci/devices/.../iov/enableAre you sure this is still the correct location with your change to struct device?
Quoted 10 lines
Nevermind, this is correct.
But the bigger problem is that userspace doesn't know when these attributes show up. So tools like udev and HAL and others can't look for them as they never get notified, and they don't even know if they should be looking for them or not.
Is there any way to tie these attributes to the "main" pci device so that they get created before the device is announced to the world? Doing that would solve this issue.
thanks,
greg k-h
Quoted 7 lines
Currently PCI subsystem has
/sys/.../{vendor,device,...}bundled to the main PCI device (I suppose this means the entries are created by 'device_add')And after the PCI device is announced,
/sys/.../{config,resourceX,rom,vpd,iov,...}get created depending on if these features are supported.
And that's a bug. Let's not continue to make the same bug here as well.
Quoted 2 lines
Making dynamic entries tie to the main PCI device would require PCI subsystem to allocate different '
bus_type' for the devices, right?
No, it would just mean they need to be all added before the device is fully registered with the driver core.
thanks,
greg k-h
Re: [PATCH 15/16 v6] PCI: document the SR-IOV sysfs entries
In reply to Zhao, YuQuoted 31 lines · Yu
Quoted 19 lines · Greg KH
Quoted 5 lines · Greg KH
Quoted 4 lines · Yu Zhao
Documentation/ABI/testing/sysfs-bus-pci | 33 +++++++++++++++++++++++++++++++ 1 files changed, 33 insertions(+), 0 deletions(-)Quoted 13 lines
Documentation/ABI/testing/sysfs-bus-pci Description: · line 13
Quoted 11 lines
Documentation/ABI/testing/sysfs-bus-pci +2
-0@@ -9,3 +9,36 @@ Description: that some devices may have malformatted data. If the underlying VPD has a writable section then the corresponding section of this file will be writable. + +What: /sys/bus/pci/devices/.../iov/enableAre you sure this is still the correct location with your change to struct device?
Quoted 10 lines
Nevermind, this is correct.
But the bigger problem is that userspace doesn't know when these attributes show up. So tools like udev and HAL and others can't look for them as they never get notified, and they don't even know if they should be looking for them or not.
Is there any way to tie these attributes to the "main" pci device so that they get created before the device is announced to the world? Doing that would solve this issue.
thanks,
greg k-h
Quoted 7 lines
Currently PCI subsystem has
/sys/.../{vendor,device,...}bundled to the main PCI device (I suppose this means the entries are created by 'device_add')And after the PCI device is announced,
/sys/.../{config,resourceX,rom,vpd,iov,...}get created depending on if these features are supported.
And that's a bug. Let's not continue to make the same bug here as well.
Quoted 2 lines
Making dynamic entries tie to the main PCI device would require PCI subsystem to allocate different '
bus_type' for the devices, right?
No, it would just mean they need to be all added before the device is fully registered with the driver core.
thanks,
greg k-h
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
In reply to Greg KHQuoted 49 lines · Greg KH
Quoted 32 lines · Yu
Quoted 23 lines · Greg KH
Quoted 12 lines · Yu Zhao
Documentation/kernel-parameters.txt | 10 ++++++++++ 1 files changed, 10 insertions(+), 0 deletions(-)Documentation/kernel-parameters.txt +0
-0@@ -1677,6 +1677,16 @@ and is between 256 and 4096 characters. It isdefined in the file
Documentation/kernel-parameters.txt +0
-0cbmemsize=nn[KMG] The fixed amount of bus space which is reserved for the CardBus bridge's memory window. The default value is 64 megabytes.Documentation/kernel-parameters.txt and is between 256 and 4096 characters. It is defined in the file · line 1689
Quoted 10 lines
Documentation/kernel-parameters.txt +10
-0+ assign-mmio=[dddd:]bb [X86] reassign memory resources of all + devices under bus [dddd:]bb (dddd is the domain + number and bb is the bus number). + assign-pio=[dddd:]bb [X86] reassign io port resources of all + devices under bus [dddd:]bb (dddd is the domain + number and bb is the bus number). + align-mmio=[dddd:]bb:dd.f [X86] relocate memory resources of a + device to minimum PAGE_SIZE alignment (dddd is + the domain number and bb, dd and f is the bus, + device and function number).Quoted 8 lines
This seems like a big problem. How are we going to know to add these command line options for devices we haven't even seen/known about yet? How do we know the bus ids aren't going to change between boots (hint, they are, pci bus ids change all the time...)
We need to be able to do this kind of thing dynamically, not fixed at boot time, which seems way to early to even know about this, right? thanks,
greg k-hQuoted 3 lines
Yes, I totally agree. Doing things dynamically is better.
The purpose of these parameters is to rebalance and align resources for device that has BARs encapsulated in various new capabilities (SR-IOV, etc.), because most of existing BIOSes don't take care of those BARs.
But how are you going to know what the proper device ids are going to be before the machine boots? I don't see how these options are ever going to work properly for a "real" user.
Quoted 1 line
If we do resource rebalance after system is up, do you think there is any side effect or impact to other subsystem other than PCI (e.g. MTRR)?
I don't think so.
Quoted 1 line
I haven't had much thinking on the dynamical resource rebalance. If you have any idea about this, can you please suggest?
Yeah, it's going to be hard :)
Quoted 8 lines
We've thought about this in the past, and even Microsoft said it was going to happen for Vista, but they realized in the end, like we did a few years previously, that it would require full support of all PCI drivers as well (if you rebalance stuff that is already bound to a driver.) So they dropped it.
When would you want to do this kind of rebalancing? Before any PCI driver is bound to any devices? Or afterwards?
I guess if we want the rebalance dynamic, then we should have it full -- the rebalance would be functional even after the driver is loaded.
But in most cases, there will be problem when we unload driver from a hard disk controller, etc. We can mount root on a ramdisk and do the rebalance there, but it's complicated for a real user.
So looks like doing rebalancing before any driver is bound to any device is also a nice idea, if user can get a shell to do rebalance before built-in PCI driver grabs device.
Regards,
Yu
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
In reply to Greg KHQuoted 54 lines · Greg KH
Quoted 33 lines · Yu
Quoted 24 lines · Greg KH
Quoted 13 lines · Yu Zhao
Documentation/kernel-parameters.txt | 10 ++++++++++ 1 files changed, 10 insertions(+), 0 deletions(-)Documentation/kernel-parameters.txt +0
-0@@ -1677,6 +1677,16 @@ and is between 256 and 4096 characters. It isdefined in the file
Documentation/kernel-parameters.txt +0
-0cbmemsize=nn[KMG] The fixed amount of bus space which is reserved for the CardBus bridge's memory window. The default value is 64 megabytes.Documentation/kernel-parameters.txt and is between 256 and 4096 characters. It is defined in the file · line 1689
Quoted 10 lines
Documentation/kernel-parameters.txt +10
-0+ assign-mmio=[dddd:]bb [X86] reassign memory resources of all + devices under bus [dddd:]bb (dddd is the domain + number and bb is the bus number). + assign-pio=[dddd:]bb [X86] reassign io port resources of all + devices under bus [dddd:]bb (dddd is the domain + number and bb is the bus number). + align-mmio=[dddd:]bb:dd.f [X86] relocate memory resources of a + device to minimum PAGE_SIZE alignment (dddd is + the domain number and bb, dd and f is the bus, + device and function number).Quoted 8 lines
This seems like a big problem. How are we going to know to add these command line options for devices we haven't even seen/known about yet? How do we know the bus ids aren't going to change between boots (hint, they are, pci bus ids change all the time...)
We need to be able to do this kind of thing dynamically, not fixed at boot time, which seems way to early to even know about this, right? thanks,
greg k-hQuoted 5 lines
Yes, I totally agree. Doing things dynamically is better.
The purpose of these parameters is to rebalance and align resources for device that has BARs encapsulated in various new capabilities (SR-IOV, etc.), because most of existing BIOSes don't take care of those BARs.
But how are you going to know what the proper device ids are going to be before the machine boots? I don't see how these options are ever going to work properly for a "real" user.
Quoted 2 lines
If we do resource rebalance after system is up, do you think there is any side effect or impact to other subsystem other than PCI (e.g. MTRR)?
I don't think so.
Quoted 2 lines
I haven't had much thinking on the dynamical resource rebalance. If you have any idea about this, can you please suggest?
Yeah, it's going to be hard :)
Quoted 8 lines
We've thought about this in the past, and even Microsoft said it was going to happen for Vista, but they realized in the end, like we did a few years previously, that it would require full support of all PCI drivers as well (if you rebalance stuff that is already bound to a driver.) So they dropped it.
When would you want to do this kind of rebalancing? Before any PCI driver is bound to any devices? Or afterwards?
I guess if we want the rebalance dynamic, then we should have it full -- the rebalance would be functional even after the driver is loaded.
But in most cases, there will be problem when we unload driver from a hard disk controller, etc. We can mount root on a ramdisk and do the rebalance there, but it's complicated for a real user.
So looks like doing rebalancing before any driver is bound to any device is also a nice idea, if user can get a shell to do rebalance before built-in PCI driver grabs device.
Regards,
Yu
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
In reply to Zhao, YuQuoted 13 lines · Yu
Quoted 8 lines · Greg KH
We've thought about this in the past, and even Microsoft said it was going to happen for Vista, but they realized in the end, like we did a few years previously, that it would require full support of all PCI drivers as well (if you rebalance stuff that is already bound to a driver.) So they dropped it.
When would you want to do this kind of rebalancing? Before any PCI driver is bound to any devices? Or afterwards?
I guess if we want the rebalance dynamic, then we should have it full -- the rebalance would be functional even after the driver is loaded.
Quoted 7 lines
But in most cases, there will be problem when we unload driver from a hard disk controller, etc. We can mount root on a ramdisk and do the rebalance there, but it's complicated for a real user.
So looks like doing rebalancing before any driver is bound to any device is also a nice idea, if user can get a shell to do rebalance before built-in PCI driver grabs device.
Can we use the suspend/resume code to do this? Some drivers (sym2 for one) would definitely need to rerun some of their init code to cope with a BAR address changing.
Signature · 4 lines
Matthew Wilcox Intel Open Source Technology Centre "Bill, look, we understand that you're interested in selling us this operating system, but compare it to ours. We can't possibly take such a retrograde step."
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
In reply to Zhao, YuQuoted 13 lines · Yu
Quoted 8 lines · Greg KH
We've thought about this in the past, and even Microsoft said it was going to happen for Vista, but they realized in the end, like we did a few years previously, that it would require full support of all PCI drivers as well (if you rebalance stuff that is already bound to a driver.) So they dropped it.
When would you want to do this kind of rebalancing? Before any PCI driver is bound to any devices? Or afterwards?
I guess if we want the rebalance dynamic, then we should have it full -- the rebalance would be functional even after the driver is loaded.
Quoted 7 lines
But in most cases, there will be problem when we unload driver from a hard disk controller, etc. We can mount root on a ramdisk and do the rebalance there, but it's complicated for a real user.
So looks like doing rebalancing before any driver is bound to any device is also a nice idea, if user can get a shell to do rebalance before built-in PCI driver grabs device.
Can we use the suspend/resume code to do this? Some drivers (sym2 for one) would definitely need to rerun some of their init code to cope with a BAR address changing.
Signature · 4 lines
Matthew Wilcox Intel Open Source Technology Centre "Bill, look, we understand that you're interested in selling us this operating system, but compare it to ours. We can't possibly take such a retrograde step."
Quoted 9 lines · Greg KH
Quoted 7 lines · Yu Zhao
Greetings,
Following patches are intended to support SR-IOV capability in the Linux kernel. With these patches, people can turn a PCI device with the capability into multiple ones from software perspective, which will benefit KVM and achieve other purposes such as QoS, security, and etc.
Quoted 3 lines
Is there any actual users of this API around yet? How was it tested as there is no hardware to test on? Which drivers are going to have to be rewritten to take advantage of this new interface?
Yes, the API is used by Intel, HP, NextIO and some other anonymous companies as they rise questions and send me feedback. I haven't seen their works but I guess some of drivers using SR-IOV API are going to be released soon.
My test was done with Intel 82576 Gigabit Ethernet Controller. The product brief is at
http://download.intel.com/design/network/ProdBrf/320025.pdf and the spec is available at
http://download.intel.com/design/network/datashts/82576_Datasheet_v2p1.pdf
Regards,
Yu
Quoted 9 lines · Greg KH
Quoted 7 lines · Yu Zhao
Greetings,
Following patches are intended to support SR-IOV capability in the Linux kernel. With these patches, people can turn a PCI device with the capability into multiple ones from software perspective, which will benefit KVM and achieve other purposes such as QoS, security, and etc.
Quoted 3 lines
Is there any actual users of this API around yet? How was it tested as there is no hardware to test on? Which drivers are going to have to be rewritten to take advantage of this new interface?
Yes, the API is used by Intel, HP, NextIO and some other anonymous companies as they rise questions and send me feedback. I haven't seen their works but I guess some of drivers using SR-IOV API are going to be released soon.
My test was done with Intel 82576 Gigabit Ethernet Controller. The product brief is at http://download.intel.com/design/network/ProdBrf/320025.pdf and the spec is available at http://download.intel.com/design/network/datashts/82576_Datasheet_v2p1.pdf
Regards,
Yu
Quoted 25 lines · Greg KH
Quoted 24 lines · H L
Quoted 22 lines · Greg KH
Quoted 1 line · H L
I have not modified any existing drivers, but instead
I threw together
Quoted 1 line
a bare-bones module enabling me to make a call to
pci_iov_register()Quoted 1 line
and then poke at an SR-IOV adapter's /sys entries
for which no driver
Quoted 3 lines
was loaded.
It appears from my perusal thus far that drivers using
these new
Quoted 1 line
SR-IOV patches will require modification; i.e. the
driver associated
Quoted 1 line
with the Physical Function (PF) will be required to
make the
Quoted 1 line
pci_iov_register()call along with the requisite
notify()function.Quoted 1 line
Essentially this suggests to me a model for the PF
driver to perform
Quoted 1 line
any "global actions" or setup on behalf of
VFs before enabling them
Quoted 1 line
after which VF drivers could be associated.
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
Where would the VF drivers have to be associated? On the "
pci_dev"
level or on a higher one?I have not yet fully grocked Yu Zhao's model to answer this. That said, I would *hope* to find it on the "
pci_dev" level.Me too.
VF is kind of lightweight PCI device, and it's represented by "struct pci_dev". VF driver bounds to the "pci_dev" and works in the same way as other drivers.
Quoted 16 lines
Quoted 4 lines
Quoted 4 lines
Will all drivers that want to bind to a "VF" device need to be
rewritten?Quoted 11 lines
Not necessarily, or perhaps minimally; depends on hardware/firmware and actions the driver wants to take. An example here might assist. Let's just say someone has created, oh, I don't know, maybe an SR-IOV NIC. Now, for 'general' I/O operations to pass network traffic back and forth there would ideally be no difference in the actions and therefore behavior of a PF driver and a VF driver. But, what do you do in the instance a VF wants to change link-speed? As that physical characteristic affects all VFs, how do you handle that? This is where the hardware/firmware implementation part comes to play. If a VF driver performs some actions to initiate the change in link speed, the logic in the adapter could be anything like:
Quoted 6 lines
<snip>
Yes, I agree that all of this needs to be done, somehow.
It's that "somehow" that I am interested in trying to see how it works out.
This is device specific part. VF driver is free to do what it wants to do with device specific registers and resources, and wouldn't concern us as far as it behaves as PCI device driver.
Quoted 27 lines
Quoted 16 lines
Quoted 7 lines
Quoted 2 lines
I have so far only seen Yu Zhao's
"7-patch" set. I've not yet looked
Quoted 1 line
at his subsequently tendered "15-patch" set
so I don't know what has
Quoted 1 line
changed. The hardware/firmware implementation for
any given SR-IOV
Quoted 8 lines
Quoted 1 line
compatible device, will determine the extent of
differences required
Quoted 1 line
between a PF driver and a VF driver.
Yeah, that's what I'm worried/curious about. Without seeing the code
for such a driver, how can we properly evaluate if this infrastructure
is the correct one and the proper way to do all of this?Quoted 10 lines
As the example above demonstrates, that's a tough question to answer. Ideally, in my view, there would only be one driver written per SR-IOV device and it would contain the logic to "do the right things" based on whether its running as a PF or VF with that determination easily accomplished by testing the existence of the SR-IOV extended
capability. Then, in an effort to minimize (if not eliminate) the
complexities of driver-to-driver actions for fielding "global events", contain as much of the logic as is possible within the adapter. Minimizing the efforts required for the device driver writers in my opinion paves the way to greater adoption of this technology.
Quoted 6 lines
Yes, making things easier is the key here.
Perhaps some of this could be hidden with a new bus type for these kinds of devices? Or a "virtual" bus of pci devices that the original SR-IOV device creates that corrispond to the individual virtual PCI devices? If that were the case, then it might be a lot easier in the end.
PCI SIG only defines SR-IOV at PCI level, we can't predict what the hardware vendors would implement at device specific logic level.
An example of SR-IOV NIC: PF may not have network functionality, it only controls VFs. Because people only want to use VFs in virtual machines, they don't need network functionality in the environment (e.g. hypervisor) where PF resides.
Thanks,
Yu
Quoted 25 lines · Greg KH
Quoted 24 lines · H L
Quoted 22 lines · Greg KH
Quoted 1 line · H L
I have not modified any existing drivers, but instead
I threw together
Quoted 1 line
a bare-bones module enabling me to make a call to
pci_iov_register()Quoted 1 line
and then poke at an SR-IOV adapter's /sys entries
for which no driver
Quoted 3 lines
was loaded.
It appears from my perusal thus far that drivers using
these new
Quoted 1 line
SR-IOV patches will require modification; i.e. the
driver associated
Quoted 1 line
with the Physical Function (PF) will be required to
make the
Quoted 1 line
pci_iov_register()call along with the requisite
notify()function.Quoted 1 line
Essentially this suggests to me a model for the PF
driver to perform
Quoted 1 line
any "global actions" or setup on behalf of
VFs before enabling them
Quoted 1 line
after which VF drivers could be associated.
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
Where would the VF drivers have to be associated? On the "
pci_dev"
level or on a higher one?I have not yet fully grocked Yu Zhao's model to answer this. That said, I would *hope* to find it on the "
pci_dev" level.Me too.
VF is kind of lightweight PCI device, and it's represented by "struct pci_dev". VF driver bounds to the "pci_dev" and works in the same way as other drivers.
Quoted 16 lines
Quoted 4 lines
Quoted 4 lines
Will all drivers that want to bind to a "VF"
device need to be
rewritten?Quoted 11 lines
Not necessarily, or perhaps minimally; depends on hardware/firmware and actions the driver wants to take. An example here might assist. Let's just say someone has created, oh, I don't know, maybe an SR-IOV NIC. Now, for 'general' I/O operations to pass network traffic back and forth there would ideally be no difference in the actions and therefore behavior of a PF driver and a VF driver. But, what do you do in the instance a VF wants to change link-speed? As that physical characteristic affects all VFs, how do you handle that? This is where the hardware/firmware implementation part comes to play. If a VF driver performs some actions to initiate the change in link speed, the logic in the adapter could be anything like:
Quoted 6 lines
<snip>
Yes, I agree that all of this needs to be done, somehow.
It's that "somehow" that I am interested in trying to see how it works out.
This is device specific part. VF driver is free to do what it wants to do with device specific registers and resources, and wouldn't concern us as far as it behaves as PCI device driver.
Quoted 26 lines
Quoted 15 lines
Quoted 6 lines
Quoted 2 lines
I have so far only seen Yu Zhao's
"7-patch" set. I've not yet looked
Quoted 1 line
at his subsequently tendered "15-patch" set
so I don't know what has
Quoted 1 line
changed. The hardware/firmware implementation for
Quoted 8 lines
any given SR-IOV
Quoted 1 line
compatible device, will determine the extent of
differences required
Quoted 1 line
between a PF driver and a VF driver.
Yeah, that's what I'm worried/curious about. Without seeing the code for such a driver, how can we properly evaluate if this infrastructure
is the correct one and the proper way to do all of this?Quoted 10 lines
As the example above demonstrates, that's a tough question to answer. Ideally, in my view, there would only be one driver written per SR-IOV device and it would contain the logic to "do the right things" based on whether its running as a PF or VF with that determination easily accomplished by testing the existence of the SR-IOV extended
capability. Then, in an effort to minimize (if not eliminate) the
complexities of driver-to-driver actions for fielding "global events", contain as much of the logic as is possible within the adapter. Minimizing the efforts required for the device driver writers in my opinion paves the way to greater adoption of this technology.
Quoted 6 lines
Yes, making things easier is the key here.
Perhaps some of this could be hidden with a new bus type for these kinds of devices? Or a "virtual" bus of pci devices that the original SR-IOV device creates that corrispond to the individual virtual PCI devices? If that were the case, then it might be a lot easier in the end.
PCI SIG only defines SR-IOV at PCI level, we can't predict what the hardware vendors would implement at device specific logic level.
An example of SR-IOV NIC: PF may not have network functionality, it only controls VFs. Because people only want to use VFs in virtual machines, they don't need network functionality in the environment (e.g. hypervisor) where PF resides.
Thanks,
Yu
Quoted 9 lines · Yu
Quoted 8 lines · Greg KH
Quoted 7 lines · Yu Zhao
Greetings,
Following patches are intended to support SR-IOV capability in the Linux kernel. With these patches, people can turn a PCI device with the capability into multiple ones from software perspective, which will benefit KVM and achieve other purposes such as QoS, security, and etc.
Quoted 8 lines
Quoted 3 lines
Is there any actual users of this API around yet? How was it tested as there is no hardware to test on? Which drivers are going to have to be rewritten to take advantage of this new interface?
Yes, the API is used by Intel, HP, NextIO and some other anonymous companies as they rise questions and send me feedback. I haven't seen their works but I guess some of drivers using SR-IOV API are going to be released soon.
Well, we can't merge infrastructure without seeing the users of that infrastructure, right?
Quoted 4 lines
My test was done with Intel 82576 Gigabit Ethernet Controller. The product brief is at http://download.intel.com/design/network/ProdBrf/320025.pdf and the spec is available at
http://download.intel.com/design/network/datashts/82576_Datasheet_v2p1.pdf
Cool, do you have that driver we can see?
How does it interact and handle the kvm and xen issues that have been posted?
thanks,
greg k-h
Quoted 9 lines · Yu
Quoted 8 lines · Greg KH
Quoted 7 lines · Yu Zhao
Greetings,
Following patches are intended to support SR-IOV capability in the Linux kernel. With these patches, people can turn a PCI device with the capability into multiple ones from software perspective, which will benefit KVM and achieve other purposes such as QoS, security, and etc.
Quoted 8 lines
Quoted 3 lines
Is there any actual users of this API around yet? How was it tested as there is no hardware to test on? Which drivers are going to have to be rewritten to take advantage of this new interface?
Yes, the API is used by Intel, HP, NextIO and some other anonymous companies as they rise questions and send me feedback. I haven't seen their works but I guess some of drivers using SR-IOV API are going to be released soon.
Well, we can't merge infrastructure without seeing the users of that infrastructure, right?
Quoted 4 lines
My test was done with Intel 82576 Gigabit Ethernet Controller. The product brief is at http://download.intel.com/design/network/ProdBrf/320025.pdf and the spec is available at
http://download.intel.com/design/network/datashts/82576_Datasheet_v2p1.pdf
Cool, do you have that driver we can see?
How does it interact and handle the kvm and xen issues that have been posted?
thanks,
greg k-h
Quoted 53 lines · Chris Wright
Quoted 27 lines · * Greg KH ([email protected]) wrote:
Quoted 18 lines · Matthew Wilcox
Quoted 17 lines · Greg KH
Quoted 5 lines · H L
I have not modified any existing drivers, but instead I threw together a bare-bones module enabling me to make a call to
pci_iov_register()and then poke at an SR-IOV adapter's /sys entries for which no driver was loaded.Quoted 7 lines
It appears from my perusal thus far that drivers using these new SR-IOV patches will require modification; i.e. the driver associated with the Physical Function (PF) will be required to make the
pci_iov_register()call along with the requisitenotify()function. Essentially this suggests to me a model for the PF driver to perform any "global actions" or setup on behalf of VFs before enabling them after which VF drivers could be associated.Where would the VF drivers have to be associated? On the "
pci_dev" level or on a higher one?Quoted 7 lines
Quoted 2 lines
Will all drivers that want to bind to a "VF" device need to be rewritten?
The current model being implemented by my colleagues has separate drivers for the PF (aka native) and VF devices. I don't personally believe this is the correct path, but I'm reserving judgement until I see some code.
Quoted 8 lines
Hm, I would like to see that code before we can properly evaluate this interface. Especially as they are all tightly tied together.
Quoted 3 lines
I don't think we really know what the One True Usage model is for VF devices. Chris Wright has some ideas, I have some ideas and Yu Zhao has some ideas. I bet there's other people who have other ideas too.
I'd love to hear those ideas.
First there's the question of how to represent the VF on the host. Ideally (IMO) this would show up as a normal interface so that normal tools can configure the interface. This is not exactly how the first round of patches were designed.
Second there's the question of reserving the BDF on the host such that we don't have two drivers (one in the host and one in a guest) trying to drive the same device (an issue that shows up for device assignment as well as VF assignment).
Third there's the question of whether the VF can be used in the host at all.
Fourth there's the question of whether the VF and PF drivers are the same or separate.
Quoted 8 lines
The typical usecase is assigning the VF to the guest directly, so there's only enough functionality in the host side to allocate a VF, configure it, and assign it (and propagate AER). This is with separate PF and VF driver.
As Anthony mentioned, we are interested in allowing the host to use the VF. This could be useful for containers as well as dedicating a VF (a set of device resources) to a guest w/out passing it through.
All of this looks great. So, with all of these questions, how does the current code pertain to these issues? It seems like we have a long way to go...
thanks,
greg k-h
Quoted 53 lines · Chris Wright
Quoted 27 lines · * Greg KH ([email protected]) wrote:
Quoted 18 lines · Matthew Wilcox
Quoted 17 lines · Greg KH
Quoted 5 lines · H L
I have not modified any existing drivers, but instead I threw together a bare-bones module enabling me to make a call to
pci_iov_register()and then poke at an SR-IOV adapter's /sys entries for which no driver was loaded.Quoted 7 lines
It appears from my perusal thus far that drivers using these new SR-IOV patches will require modification; i.e. the driver associated with the Physical Function (PF) will be required to make the
pci_iov_register()call along with the requisitenotify()function. Essentially this suggests to me a model for the PF driver to perform any "global actions" or setup on behalf of VFs before enabling them after which VF drivers could be associated.Where would the VF drivers have to be associated? On the "
pci_dev" level or on a higher one?Quoted 7 lines
Quoted 2 lines
Will all drivers that want to bind to a "VF" device need to be rewritten?
The current model being implemented by my colleagues has separate drivers for the PF (aka native) and VF devices. I don't personally believe this is the correct path, but I'm reserving judgement until I see some code.
Quoted 8 lines
Hm, I would like to see that code before we can properly evaluate this interface. Especially as they are all tightly tied together.
Quoted 3 lines
I don't think we really know what the One True Usage model is for VF devices. Chris Wright has some ideas, I have some ideas and Yu Zhao has some ideas. I bet there's other people who have other ideas too.
I'd love to hear those ideas.
First there's the question of how to represent the VF on the host. Ideally (IMO) this would show up as a normal interface so that normal tools can configure the interface. This is not exactly how the first round of patches were designed.
Second there's the question of reserving the BDF on the host such that we don't have two drivers (one in the host and one in a guest) trying to drive the same device (an issue that shows up for device assignment as well as VF assignment).
Third there's the question of whether the VF can be used in the host at all.
Fourth there's the question of whether the VF and PF drivers are the same or separate.
Quoted 8 lines
The typical usecase is assigning the VF to the guest directly, so there's only enough functionality in the host side to allocate a VF, configure it, and assign it (and propagate AER). This is with separate PF and VF driver.
As Anthony mentioned, we are interested in allowing the host to use the VF. This could be useful for containers as well as dedicating a VF (a set of device resources) to a guest w/out passing it through.
All of this looks great. So, with all of these questions, how does the current code pertain to these issues? It seems like we have a long way to go...
thanks,
greg k-h
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
In reply to Zhao, YuQuoted 60 lines · Yu
Quoted 47 lines · Greg KH
Quoted 33 lines · Yu
Quoted 24 lines · Greg KH
Quoted 13 lines · Yu Zhao
Documentation/kernel-parameters.txt | 10 ++++++++++ 1 files changed, 10 insertions(+), 0 deletions(-)Documentation/kernel-parameters.txt +0
-0@@ -1677,6 +1677,16 @@ and is between 256 and 4096 characters. It isdefined in the file
Documentation/kernel-parameters.txt +0
-0cbmemsize=nn[KMG] The fixed amount of bus space which is reserved for the CardBus bridge's memory window. The default value is 64 megabytes.Documentation/kernel-parameters.txt and is between 256 and 4096 characters. It is defined in the file · line 1689
Quoted 10 lines
Documentation/kernel-parameters.txt +10
-0+ assign-mmio=[dddd:]bb [X86] reassign memory resources of all + devices under bus [dddd:]bb (dddd is the domain + number and bb is the bus number). + assign-pio=[dddd:]bb [X86] reassign io port resources of all + devices under bus [dddd:]bb (dddd is the domain + number and bb is the bus number). + align-mmio=[dddd:]bb:dd.f [X86] relocate memory resources of a + device to minimum PAGE_SIZE alignment (dddd is + the domain number and bb, dd and f is the bus, + device and function number).Quoted 8 lines
This seems like a big problem. How are we going to know to add these command line options for devices we haven't even seen/known about yet? How do we know the bus ids aren't going to change between boots (hint, they are, pci bus ids change all the time...)
We need to be able to do this kind of thing dynamically, not fixed at boot time, which seems way to early to even know about this, right? thanks,
greg k-hQuoted 5 lines
Yes, I totally agree. Doing things dynamically is better.
The purpose of these parameters is to rebalance and align resources for device that has BARs encapsulated in various new capabilities (SR-IOV, etc.), because most of existing BIOSes don't take care of those BARs.
But how are you going to know what the proper device ids are going to be before the machine boots? I don't see how these options are ever going to work properly for a "real" user.
Quoted 2 lines
If we do resource rebalance after system is up, do you think there is any side effect or impact to other subsystem other than PCI (e.g. MTRR)?
I don't think so.
Quoted 2 lines
I haven't had much thinking on the dynamical resource rebalance. If you have any idea about this, can you please suggest?
Quoted 8 lines
Yeah, it's going to be hard :)
We've thought about this in the past, and even Microsoft said it was going to happen for Vista, but they realized in the end, like we did a few years previously, that it would require full support of all PCI drivers as well (if you rebalance stuff that is already bound to a driver.) So they dropped it.
When would you want to do this kind of rebalancing? Before any PCI driver is bound to any devices? Or afterwards?I guess if we want the rebalance dynamic, then we should have it full -- the rebalance would be functional even after the driver is loaded.
Quoted 7 lines
But in most cases, there will be problem when we unload driver from a hard disk controller, etc. We can mount root on a ramdisk and do the rebalance there, but it's complicated for a real user.
So looks like doing rebalancing before any driver is bound to any device is also a nice idea, if user can get a shell to do rebalance before built-in PCI driver grabs device.
That's not going to work, it needs to happen before any PCI device is bound, which is before init runs.
thanks,
greg k-h
Quoted 8 lines · Anthony Liguori
Quoted 7 lines · Greg KH
Quoted 4 lines · Matthew Wilcox
I don't think we really know what the One True Usage model is for VF devices. Chris Wright has some ideas, I have some ideas and Yu Zhao has some ideas. I bet there's other people who have other ideas too.
Quoted 7 lines
Quoted 2 lines
I'd love to hear those ideas.
We've been talking about avoiding hardware passthrough entirely and just backing a virtio-net backend driver by a dedicated VF in the host. That avoids a huge amount of guest-facing complexity, let's migration Just Work, and should give the same level of performance.
Does that involve this patch set? Or a different type of interface.
thanks,
greg k-h
Quoted 10 lines · Matthew Wilcox
Quoted 7 lines
What we would rather do in KVM, is have the VFs appear in the host as standard network devices. We would then like to back our existing PV driver to this VF directly bypassing the host networking stack. A key feature here is being able to fill the VF's receive queue with guest memory instead of host kernel memory so that you can get zero-copy receive traffic. This will perform just as well as doing passthrough (at least) and avoid all that ugliness of dealing with SR-IOV in the guest.
This argues for ignoring the SR-IOV mess completely. Just have the host driver expose multiple 'ethN' devices.
That would work, but do we want to do that for every different type of driver?
thanks,
greg k-h
Quoted 10 lines · Matthew Wilcox
Quoted 7 lines
What we would rather do in KVM, is have the VFs appear in the host as standard network devices. We would then like to back our existing PV driver to this VF directly bypassing the host networking stack. A key feature here is being able to fill the VF's receive queue with guest memory instead of host kernel memory so that you can get zero-copy receive traffic. This will perform just as well as doing passthrough (at least) and avoid all that ugliness of dealing with SR-IOV in the guest.
This argues for ignoring the SR-IOV mess completely. Just have the host driver expose multiple 'ethN' devices.
That would work, but do we want to do that for every different type of driver?
thanks,
greg k-h
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
In reply to Zhao, YuQuoted 60 lines · Yu
Quoted 47 lines · Greg KH
Quoted 33 lines · Yu
Quoted 24 lines · Greg KH
Quoted 13 lines · Yu Zhao
Documentation/kernel-parameters.txt | 10 ++++++++++ 1 files changed, 10 insertions(+), 0 deletions(-)Documentation/kernel-parameters.txt +0
-0@@ -1677,6 +1677,16 @@ and is between 256 and 4096 characters. It isdefined in the file
Documentation/kernel-parameters.txt +0
-0cbmemsize=nn[KMG] The fixed amount of bus space which is reserved for the CardBus bridge's memory window. The default value is 64 megabytes.Documentation/kernel-parameters.txt and is between 256 and 4096 characters. It is defined in the file · line 1689
Quoted 10 lines
Documentation/kernel-parameters.txt +10
-0+ assign-mmio=[dddd:]bb [X86] reassign memory resources of all + devices under bus [dddd:]bb (dddd is the domain + number and bb is the bus number). + assign-pio=[dddd:]bb [X86] reassign io port resources of all + devices under bus [dddd:]bb (dddd is the domain + number and bb is the bus number). + align-mmio=[dddd:]bb:dd.f [X86] relocate memory resources of a + device to minimum PAGE_SIZE alignment (dddd is + the domain number and bb, dd and f is the bus, + device and function number).Quoted 8 lines
This seems like a big problem. How are we going to know to add these command line options for devices we haven't even seen/known about yet? How do we know the bus ids aren't going to change between boots (hint, they are, pci bus ids change all the time...)
We need to be able to do this kind of thing dynamically, not fixed at boot time, which seems way to early to even know about this, right? thanks,
greg k-hQuoted 5 lines
Yes, I totally agree. Doing things dynamically is better.
The purpose of these parameters is to rebalance and align resources for device that has BARs encapsulated in various new capabilities (SR-IOV, etc.), because most of existing BIOSes don't take care of those BARs.
But how are you going to know what the proper device ids are going to be before the machine boots? I don't see how these options are ever going to work properly for a "real" user.
Quoted 2 lines
If we do resource rebalance after system is up, do you think there is any side effect or impact to other subsystem other than PCI (e.g. MTRR)?
I don't think so.
Quoted 2 lines
I haven't had much thinking on the dynamical resource rebalance. If you have any idea about this, can you please suggest?
Quoted 8 lines
Yeah, it's going to be hard :)
We've thought about this in the past, and even Microsoft said it was going to happen for Vista, but they realized in the end, like we did a few years previously, that it would require full support of all PCI drivers as well (if you rebalance stuff that is already bound to a driver.) So they dropped it.
When would you want to do this kind of rebalancing? Before any PCI driver is bound to any devices? Or afterwards?I guess if we want the rebalance dynamic, then we should have it full -- the rebalance would be functional even after the driver is loaded.
Quoted 7 lines
But in most cases, there will be problem when we unload driver from a hard disk controller, etc. We can mount root on a ramdisk and do the rebalance there, but it's complicated for a real user.
So looks like doing rebalancing before any driver is bound to any device is also a nice idea, if user can get a shell to do rebalance before built-in PCI driver grabs device.
That's not going to work, it needs to happen before any PCI device is bound, which is before init runs.
thanks,
greg k-h
Quoted 8 lines · Anthony Liguori
Quoted 7 lines · Greg KH
Quoted 4 lines · Matthew Wilcox
I don't think we really know what the One True Usage model is for VF devices. Chris Wright has some ideas, I have some ideas and Yu Zhao has some ideas. I bet there's other people who have other ideas too.
Quoted 7 lines
Quoted 2 lines
I'd love to hear those ideas.
We've been talking about avoiding hardware passthrough entirely and just backing a virtio-net backend driver by a dedicated VF in the host. That avoids a huge amount of guest-facing complexity, let's migration Just Work, and should give the same level of performance.
Does that involve this patch set? Or a different type of interface.
thanks,
greg k-h
Quoted 6 lines · Anna
Quoted 5 lines
Perhaps some of this could be hidden with a new bus type for these kinds
of devices? Or a "virtual" bus of pci devices that the original SR-IOV device creates that corrispond to the individual virtual PCI devices? If that were the case, then it might be a lot easier in the end.
Quoted 8 lines
I think a standard communication channel in Linux for SR-IOV devices would be a good start, and help to adopt the technology. Something like the virtual bus you are describing. It means that vendors do not need to write their own communication channel in the drivers. It would need to have well defined APIs though, as I guess that devices will have very different capabilities and hardware implementations for PFs and VFs, and so they might have very different events and information to propagate.
That would be good to standardize on. Have patches?
thanks,
greg k-h
Quoted 31 lines · Chris Wright
Quoted 30 lines · * Greg KH ([email protected]) wrote:
Quoted 17 lines · Matthew Wilcox
Quoted 16 lines · Greg KH
Quoted 5 lines · H L
I have not modified any existing drivers, but instead I threw together a bare-bones module enabling me to make a call to
pci_iov_register()and then poke at an SR-IOV adapter's /sys entries for which no driver was loaded.Quoted 7 lines
It appears from my perusal thus far that drivers using these new SR-IOV patches will require modification; i.e. the driver associated with the Physical Function (PF) will be required to make the
pci_iov_register()call along with the requisitenotify()function. Essentially this suggests to me a model for the PF driver to perform any "global actions" or setup on behalf of VFs before enabling them after which VF drivers could be associated.Where would the VF drivers have to be associated? On the "
pci_dev" level or on a higher one?Quoted 6 lines
Quoted 2 lines
Will all drivers that want to bind to a "VF" device need to be rewritten?
The current model being implemented by my colleagues has separate drivers for the PF (aka native) and VF devices. I don't personally believe this is the correct path, but I'm reserving judgement until I see some code.
Hm, I would like to see that code before we can properly evaluate this interface. Especially as they are all tightly tied together.
Quoted 3 lines
I don't think we really know what the One True Usage model is for VF devices. Chris Wright has some ideas, I have some ideas and Yu Zhao has some ideas. I bet there's other people who have other ideas too.
Quoted 6 lines
Quoted 1 line
I'd love to hear those ideas.
First there's the question of how to represent the VF on the host. Ideally (IMO) this would show up as a normal interface so that normal tools can configure the interface. This is not exactly how the first round of patches were designed.
Whether the VF can show up as a normal interface is decided by VF driver. VF is represented by 'pci_dev' at PCI level, so VF driver can be loaded as normal PCI device driver.
What the software representation (eth, framebuffer, etc.) created by VF driver is not controlled by SR-IOV framework.
So you definitely can use normal tool to configure the VF if its driver supports that :-)
Quoted 5 lines
Second there's the question of reserving the BDF on the host such that we don't have two drivers (one in the host and one in a guest) trying to drive the same device (an issue that shows up for device assignment as well as VF assignment).
If we don't reserve BDF for the device, they can't work neither in the host nor the guest.
Without BDF, we can't access the config space of the device, the device also can't do DMA.
Did I miss your point?
Quoted 3 lines
Third there's the question of whether the VF can be used in the host at all.
Why can't? My VFs work well in the host as normal PCI devices :-)
Quoted 3 lines
Fourth there's the question of whether the VF and PF drivers are the same or separate.
As I mentioned in another email of this thread. We can't predict how hardware vendor creates their SR-IOV device. PCI SIG doesn't define device specific logics.
So I think the answer of this question is up to the device driver developers. If PF and VF in a SR-IOV device have similar logics, then they can combine the driver. Otherwise, e.g., if PF doesn't have real functionality at all -- it only has registers to control internal resource allocation for VFs, then the drivers should be separate, right?
Quoted 9 lines
The typical usecase is assigning the VF to the guest directly, so there's only enough functionality in the host side to allocate a VF, configure it, and assign it (and propagate AER). This is with separate PF and VF driver.
As Anthony mentioned, we are interested in allowing the host to use the VF. This could be useful for containers as well as dedicating a VF (a set of device resources) to a guest w/out passing it through.
I've considered the container cases, we don't have problem with running VF driver in the host.
Thanks,
Yu
Quoted 31 lines · Chris Wright
Quoted 30 lines · * Greg KH ([email protected]) wrote:
Quoted 17 lines · Matthew Wilcox
Quoted 16 lines · Greg KH
Quoted 5 lines · H L
I have not modified any existing drivers, but instead I threw together a bare-bones module enabling me to make a call to
pci_iov_register()and then poke at an SR-IOV adapter's /sys entries for which no driver was loaded.Quoted 7 lines
It appears from my perusal thus far that drivers using these new SR-IOV patches will require modification; i.e. the driver associated with the Physical Function (PF) will be required to make the
pci_iov_register()call along with the requisitenotify()function. Essentially this suggests to me a model for the PF driver to perform any "global actions" or setup on behalf of VFs before enabling them after which VF drivers could be associated.Where would the VF drivers have to be associated? On the "
pci_dev" level or on a higher one?Quoted 6 lines
Quoted 2 lines
Will all drivers that want to bind to a "VF" device need to be rewritten?
The current model being implemented by my colleagues has separate drivers for the PF (aka native) and VF devices. I don't personally believe this is the correct path, but I'm reserving judgement until I see some code.
Hm, I would like to see that code before we can properly evaluate this interface. Especially as they are all tightly tied together.
Quoted 3 lines
I don't think we really know what the One True Usage model is for VF devices. Chris Wright has some ideas, I have some ideas and Yu Zhao has some ideas. I bet there's other people who have other ideas too.
Quoted 6 lines
Quoted 1 line
I'd love to hear those ideas.
First there's the question of how to represent the VF on the host. Ideally (IMO) this would show up as a normal interface so that normal tools can configure the interface. This is not exactly how the first round of patches were designed.
Whether the VF can show up as a normal interface is decided by VF driver. VF is represented by 'pci_dev' at PCI level, so VF driver can be loaded as normal PCI device driver.
What the software representation (eth, framebuffer, etc.) created by VF driver is not controlled by SR-IOV framework.
So you definitely can use normal tool to configure the VF if its driver supports that :-)
Quoted 5 lines
Second there's the question of reserving the BDF on the host such that we don't have two drivers (one in the host and one in a guest) trying to drive the same device (an issue that shows up for device assignment as well as VF assignment).
If we don't reserve BDF for the device, they can't work neither in the host nor the guest.
Without BDF, we can't access the config space of the device, the device also can't do DMA.
Did I miss your point?
Quoted 3 lines
Third there's the question of whether the VF can be used in the host at all.
Why can't? My VFs work well in the host as normal PCI devices :-)
Quoted 3 lines
Fourth there's the question of whether the VF and PF drivers are the same or separate.
As I mentioned in another email of this thread. We can't predict how hardware vendor creates their SR-IOV device. PCI SIG doesn't define device specific logics.
So I think the answer of this question is up to the device driver developers. If PF and VF in a SR-IOV device have similar logics, then they can combine the driver. Otherwise, e.g., if PF doesn't have real functionality at all -- it only has registers to control internal resource allocation for VFs, then the drivers should be separate, right?
Quoted 9 lines
The typical usecase is assigning the VF to the guest directly, so there's only enough functionality in the host side to allocate a VF, configure it, and assign it (and propagate AER). This is with separate PF and VF driver.
As Anthony mentioned, we are interested in allowing the host to use the VF. This could be useful for containers as well as dedicating a VF (a set of device resources) to a guest w/out passing it through.
I've considered the container cases, we don't have problem with running VF driver in the host.
Thanks,
Yu
Quoted 5 lines
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf
Of
Quoted 4 lines
Zhao, Yu
Sent: Thursday, November 06, 2008 11:06 PM
To: Chris Wright
Cc: [email protected]; [email protected];
Quoted 1 line
Matthew Wilcox; Greg KH; [email protected];
Quoted 4 lines
[email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 0/16 v6] PCI: Linux kernel SR-IOV support
Quoted 8 lines
Quoted 5 lines · Chris Wright
Quoted 4 lines · * Greg KH ([email protected]) wrote:
Quoted 3 lines · Matthew Wilcox
Quoted 2 lines · Greg KH
Quoted 1 line · H L
I have not modified any existing drivers, but instead I threw
together
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
a bare-bones module enabling me to make a call to
pci_iov_register()
Quoted 6 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
and then poke at an SR-IOV adapter's /sys entries for which no
driver
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
was loaded.
It appears from my perusal thus far that drivers using these new SR-IOV patches will require modification; i.e. the driver
associated
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
with the Physical Function (PF) will be required to make the
pci_iov_register()call along with the requisitenotify()
function.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Essentially this suggests to me a model for the PF driver to
perform
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
any "global actions" or setup on behalf of VFs before enabling
them
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 1 line
after which VF drivers could be associated.
Where would the VF drivers have to be associated? On the
"pci_dev"
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 4 lines
level or on a higher one?
Will all drivers that want to bind to a "VF" device need to be rewritten?
The current model being implemented by my colleagues has separate drivers for the PF (aka native) and VF devices. I don't
personally
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
believe this is the correct path, but I'm reserving judgement
until I
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 1 line
see some code.
Hm, I would like to see that code before we can properly evaluate
this
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
interface. Especially as they are all tightly tied together.
Quoted 1 line
I don't think we really know what the One True Usage model is for
VF
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
devices. Chris Wright has some ideas, I have some ideas and Yu
Zhao
Quoted 6 lines
has
Quoted 5 lines
Quoted 2 lines
Quoted 1 line
some ideas. I bet there's other people who have other ideas too.
I'd love to hear those ideas.
First there's the question of how to represent the VF on the host. Ideally (IMO) this would show up as a normal interface so that
normal
Quoted 2 lines
tools
Quoted 1 line
can configure the interface. This is not exactly how the first
round of
Quoted 4 lines
Quoted 1 line
patches were designed.
Whether the VF can show up as a normal interface is decided by VF driver. VF is represented by '
pci_dev' at PCI level, so VF driver can
be
Quoted 3 lines
loaded as normal PCI device driver.
What the software representation (eth, framebuffer, etc.) created by
VF
Quoted 3 lines
driver is not controlled by SR-IOV framework.
So you definitely can use normal tool to configure the VF if its
driver
Quoted 4 lines
supports that :-)
Quoted 2 lines
Second there's the question of reserving the BDF on the host such
that
Quoted 1 line
Quoted 1 line
we don't have two drivers (one in the host and one in a guest)
trying to
Quoted 1 line
Quoted 1 line
drive the same device (an issue that shows up for device assignment
as
Quoted 6 lines
Quoted 1 line
well as VF assignment).
If we don't reserve BDF for the device, they can't work neither in the host nor the guest.
Without BDF, we can't access the config space of the device, the
device
Quoted 6 lines
also can't do DMA.
Did I miss your point?
Quoted 2 lines
Third there's the question of whether the VF can be used in the host
at
Quoted 12 lines
Quoted 1 line
all.
Why can't? My VFs work well in the host as normal PCI devices :-)
Quoted 3 lines
Fourth there's the question of whether the VF and PF drivers are the same or separate.
As I mentioned in another email of this thread. We can't predict how hardware vendor creates their SR-IOV device. PCI SIG doesn't define device specific logics.
Quoted 5 lines
So I think the answer of this question is up to the device driver developers. If PF and VF in a SR-IOV device have similar logics, then they can combine the driver. Otherwise, e.g., if PF doesn't have real functionality at all -- it only has registers to control internal resource allocation for VFs, then the drivers should be separate,
right?
Right, this really depends upon the functionality behind a VF. If VF is done as a subset of netdev interface (for example, a queue pair), then a split VF/PF driver model and a proprietary communication channel is in order.
If each VF is done as a complete netdev interface (like in our 10GbE IOV controllers), then PF and VF drivers could be the same. Each VF can be independently driven by such "native" netdev driver; this includes the ability to run a native driver in a guest in passthru mode. A PF driver in a privileged domain doesn't even have to be present.
Quoted 5 lines
Quoted 5 lines
The typical usecase is assigning the VF to the guest directly, so there's only enough functionality in the host side to allocate a VF, configure it, and assign it (and propagate AER). This is with
separate
Quoted 3 lines
Quoted 3 lines
PF and VF driver.
As Anthony mentioned, we are interested in allowing the host to use
the
Quoted 1 line
Quoted 1 line
VF. This could be useful for containers as well as dedicating a VF
(a
Quoted 3 lines
Quoted 1 line
set of device resources) to a guest w/out passing it through.
I've considered the container cases, we don't have problem with
running
Quoted 8 lines
VF driver in the host.
Thanks,
Yu
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linux-foundation.org/mailman/listinfo/virtualization
Quoted 5 lines
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf
Of
Quoted 4 lines
Zhao, Yu
Sent: Thursday, November 06, 2008 11:06 PM
To: Chris Wright
Cc: [email protected]; [email protected];
Quoted 1 line
Matthew Wilcox; Greg KH; [email protected];
Quoted 4 lines
[email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 0/16 v6] PCI: Linux kernel SR-IOV support
Quoted 8 lines
Quoted 5 lines · Chris Wright
Quoted 4 lines · * Greg KH ([email protected]) wrote:
Quoted 3 lines · Matthew Wilcox
Quoted 2 lines · Greg KH
Quoted 1 line · H L
I have not modified any existing drivers, but instead I threw
together
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
a bare-bones module enabling me to make a call to
pci_iov_register()
Quoted 6 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
and then poke at an SR-IOV adapter's /sys entries for which no
driver
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
was loaded.
It appears from my perusal thus far that drivers using these new SR-IOV patches will require modification; i.e. the driver
associated
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
with the Physical Function (PF) will be required to make the
pci_iov_register()call along with the requisitenotify()
function.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Essentially this suggests to me a model for the PF driver to
perform
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
any "global actions" or setup on behalf of VFs before enabling
them
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 1 line
after which VF drivers could be associated.
Where would the VF drivers have to be associated? On the
"pci_dev"
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 4 lines
level or on a higher one?
Will all drivers that want to bind to a "VF" device need to be rewritten?
The current model being implemented by my colleagues has separate drivers for the PF (aka native) and VF devices. I don't
personally
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
believe this is the correct path, but I'm reserving judgement
until I
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 1 line
see some code.
Hm, I would like to see that code before we can properly evaluate
this
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
interface. Especially as they are all tightly tied together.
Quoted 1 line
I don't think we really know what the One True Usage model is for
VF
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
devices. Chris Wright has some ideas, I have some ideas and Yu
Zhao
Quoted 6 lines
has
Quoted 5 lines
Quoted 2 lines
Quoted 1 line
some ideas. I bet there's other people who have other ideas too.
I'd love to hear those ideas.
First there's the question of how to represent the VF on the host. Ideally (IMO) this would show up as a normal interface so that
normal
Quoted 2 lines
tools
Quoted 1 line
can configure the interface. This is not exactly how the first
round of
Quoted 4 lines
Quoted 1 line
patches were designed.
Whether the VF can show up as a normal interface is decided by VF driver. VF is represented by '
pci_dev' at PCI level, so VF driver can
be
Quoted 3 lines
loaded as normal PCI device driver.
What the software representation (eth, framebuffer, etc.) created by
VF
Quoted 3 lines
driver is not controlled by SR-IOV framework.
So you definitely can use normal tool to configure the VF if its
driver
Quoted 4 lines
supports that :-)
Quoted 2 lines
Second there's the question of reserving the BDF on the host such
that
Quoted 1 line
Quoted 1 line
we don't have two drivers (one in the host and one in a guest)
trying to
Quoted 1 line
Quoted 1 line
drive the same device (an issue that shows up for device assignment
as
Quoted 6 lines
Quoted 1 line
well as VF assignment).
If we don't reserve BDF for the device, they can't work neither in the host nor the guest.
Without BDF, we can't access the config space of the device, the
device
Quoted 6 lines
also can't do DMA.
Did I miss your point?
Quoted 2 lines
Third there's the question of whether the VF can be used in the host
at
Quoted 12 lines
Quoted 1 line
all.
Why can't? My VFs work well in the host as normal PCI devices :-)
Quoted 3 lines
Fourth there's the question of whether the VF and PF drivers are the same or separate.
As I mentioned in another email of this thread. We can't predict how hardware vendor creates their SR-IOV device. PCI SIG doesn't define device specific logics.
Quoted 5 lines
So I think the answer of this question is up to the device driver developers. If PF and VF in a SR-IOV device have similar logics, then they can combine the driver. Otherwise, e.g., if PF doesn't have real functionality at all -- it only has registers to control internal resource allocation for VFs, then the drivers should be separate,
right?
Right, this really depends upon the functionality behind a VF. If VF is done as a subset of netdev interface (for example, a queue pair), then a split VF/PF driver model and a proprietary communication channel is in order.
If each VF is done as a complete netdev interface (like in our 10GbE IOV controllers), then PF and VF drivers could be the same. Each VF can be independently driven by such "native" netdev driver; this includes the ability to run a native driver in a guest in passthru mode. A PF driver in a privileged domain doesn't even have to be present.
Quoted 5 lines
Quoted 5 lines
The typical usecase is assigning the VF to the guest directly, so there's only enough functionality in the host side to allocate a VF, configure it, and assign it (and propagate AER). This is with
separate
Quoted 3 lines
Quoted 3 lines
PF and VF driver.
As Anthony mentioned, we are interested in allowing the host to use
the
Quoted 1 line
Quoted 1 line
VF. This could be useful for containers as well as dedicating a VF
(a
Quoted 3 lines
Quoted 1 line
set of device resources) to a guest w/out passing it through.
I've considered the container cases, we don't have problem with
running
Quoted 8 lines
VF driver in the host.
Thanks,
Yu
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linux-foundation.org/mailman/listinfo/virtualization
Quoted 8 lines · Greg KH
Quoted 7 lines · Anthony Liguori
Quoted 6 lines · Greg KH
Quoted 4 lines · Matthew Wilcox
I don't think we really know what the One True Usage model is for VF devices. Chris Wright has some ideas, I have some ideas and Yu Zhao has some ideas. I bet there's other people who have other ideas too.
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
I'd love to hear those ideas.
We've been talking about avoiding hardware passthrough entirely and just backing a virtio-net backend driver by a dedicated VF in the host. That avoids a huge amount of guest-facing complexity, let's migration Just Work, and should give the same level of performance.
This can be commonly used not only with VF -- devices that have multiple DMA queues (e.g., Intel VMDq, Neterion Xframe) and even traditional devices can also take the advantage of this.
CC Rusty Russel in case he has more comments.
Quoted 2 lines
Does that involve this patch set? Or a different type of interface.
I think that is a different type of interface. We need to hook the DMA interface in the device driver to virtio-net backend so the hardware (normal device, VF, VMDq, etc.) can DMA data to/from the virtio-net backend.
Regards,
Yu
Quoted 8 lines · Greg KH
Quoted 7 lines · Anthony Liguori
Quoted 6 lines · Greg KH
Quoted 4 lines · Matthew Wilcox
I don't think we really know what the One True Usage model is for VF devices. Chris Wright has some ideas, I have some ideas and Yu Zhao has some ideas. I bet there's other people who have other ideas too.
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
I'd love to hear those ideas.
We've been talking about avoiding hardware passthrough entirely and just backing a virtio-net backend driver by a dedicated VF in the host. That avoids a huge amount of guest-facing complexity, let's migration Just Work, and should give the same level of performance.
This can be commonly used not only with VF -- devices that have multiple DMA queues (e.g., Intel VMDq, Neterion Xframe) and even traditional devices can also take the advantage of this.
CC Rusty Russel in case he has more comments.
Quoted 2 lines
Does that involve this patch set? Or a different type of interface.
I think that is a different type of interface. We need to hook the DMA interface in the device driver to virtio-net backend so the hardware (normal device, VF, VMDq, etc.) can DMA data to/from the virtio-net backend.
Regards,
Yu
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
In reply to Greg KHQuoted 52 lines · Greg KH
Quoted 51 lines · Yu
Quoted 50 lines · Greg KH
Quoted 32 lines · Yu
Quoted 23 lines · Greg KH
Quoted 12 lines · Yu Zhao
Documentation/kernel-parameters.txt | 10 ++++++++++ 1 files changed, 10 insertions(+), 0 deletions(-)Documentation/kernel-parameters.txt +0
-0@@ -1677,6 +1677,16 @@ and is between 256 and 4096 characters. It isdefined in the file
Documentation/kernel-parameters.txt +0
-0cbmemsize=nn[KMG] The fixed amount of bus space which is reserved for the CardBus bridge's memory window. The default value is 64 megabytes.Documentation/kernel-parameters.txt and is between 256 and 4096 characters. It is defined in the file · line 1689
Quoted 10 lines
Documentation/kernel-parameters.txt +10
-0+ assign-mmio=[dddd:]bb [X86] reassign memory resources of all + devices under bus [dddd:]bb (dddd is the domain + number and bb is the bus number). + assign-pio=[dddd:]bb [X86] reassign io port resources of all + devices under bus [dddd:]bb (dddd is the domain + number and bb is the bus number). + align-mmio=[dddd:]bb:dd.f [X86] relocate memory resources of a + device to minimum PAGE_SIZE alignment (dddd is + the domain number and bb, dd and f is the bus, + device and function number).Quoted 8 lines
This seems like a big problem. How are we going to know to add these command line options for devices we haven't even seen/known about yet? How do we know the bus ids aren't going to change between boots (hint, they are, pci bus ids change all the time...)
We need to be able to do this kind of thing dynamically, not fixed at boot time, which seems way to early to even know about this, right? thanks,
greg k-hQuoted 3 lines
Yes, I totally agree. Doing things dynamically is better.
The purpose of these parameters is to rebalance and align resources for device that has BARs encapsulated in various new capabilities (SR-IOV, etc.), because most of existing BIOSes don't take care of those BARs.
But how are you going to know what the proper device ids are going to be before the machine boots? I don't see how these options are ever going to work properly for a "real" user.
Quoted 1 line
If we do resource rebalance after system is up, do you think there is any side effect or impact to other subsystem other than PCI (e.g. MTRR)?
I don't think so.
Quoted 1 line
I haven't had much thinking on the dynamical resource rebalance. If you have any idea about this, can you please suggest?
Yeah, it's going to be hard :)
We've thought about this in the past, and even Microsoft said it was going to happen for Vista, but they realized in the end, like we did a few years previously, that it would require full support of all PCI drivers as well (if you rebalance stuff that is already bound to a driver.) So they dropped it.
When would you want to do this kind of rebalancing? Before any PCI driver is bound to any devices? Or afterwards?
Quoted 8 lines
Quoted 5 lines
I guess if we want the rebalance dynamic, then we should have it full -- the rebalance would be functional even after the driver is loaded.
But in most cases, there will be problem when we unload driver from a hard disk controller, etc. We can mount root on a ramdisk and do the rebalance there, but it's complicated for a real user.
So looks like doing rebalancing before any driver is bound to any device is also a nice idea, if user can get a shell to do rebalance before built-in PCI driver grabs device.
That's not going to work, it needs to happen before any PCI device is bound, which is before init runs.
I don't think it can work either. Then we have to do rebalance after the driver bounding. But what should we do if we can't unload the driver (hard disk controller, etc.)?
Thanks,
Yu
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
In reply to Greg KHQuoted 64 lines · Greg KH
Quoted 63 lines · Yu
Quoted 47 lines · Greg KH
Quoted 33 lines · Yu
Quoted 24 lines · Greg KH
Quoted 13 lines · Yu Zhao
Documentation/kernel-parameters.txt | 10 ++++++++++ 1 files changed, 10 insertions(+), 0 deletions(-)Documentation/kernel-parameters.txt +0
-0@@ -1677,6 +1677,16 @@ and is between 256 and 4096 characters. It isdefined in the file
Documentation/kernel-parameters.txt +0
-0cbmemsize=nn[KMG] The fixed amount of bus space which is reserved for the CardBus bridge's memory window. The default value is 64 megabytes.Documentation/kernel-parameters.txt and is between 256 and 4096 characters. It is defined in the file · line 1689
Quoted 10 lines
Documentation/kernel-parameters.txt +10
-0+ assign-mmio=[dddd:]bb [X86] reassign memory resources of all + devices under bus [dddd:]bb (dddd is the domain + number and bb is the bus number). + assign-pio=[dddd:]bb [X86] reassign io port resources of all + devices under bus [dddd:]bb (dddd is the domain + number and bb is the bus number). + align-mmio=[dddd:]bb:dd.f [X86] relocate memory resources of a + device to minimum PAGE_SIZE alignment (dddd is + the domain number and bb, dd and f is the bus, + device and function number).Quoted 8 lines
This seems like a big problem. How are we going to know to add these command line options for devices we haven't even seen/known about yet? How do we know the bus ids aren't going to change between boots (hint, they are, pci bus ids change all the time...)
We need to be able to do this kind of thing dynamically, not fixed at boot time, which seems way to early to even know about this, right? thanks,
greg k-hQuoted 5 lines
Yes, I totally agree. Doing things dynamically is better.
The purpose of these parameters is to rebalance and align resources for device that has BARs encapsulated in various new capabilities (SR-IOV, etc.), because most of existing BIOSes don't take care of those BARs.
But how are you going to know what the proper device ids are going to be before the machine boots? I don't see how these options are ever going to work properly for a "real" user.
Quoted 2 lines
If we do resource rebalance after system is up, do you think there is any side effect or impact to other subsystem other than PCI (e.g. MTRR)?
I don't think so.
Quoted 2 lines
I haven't had much thinking on the dynamical resource rebalance. If you have any idea about this, can you please suggest?
Quoted 8 lines
Yeah, it's going to be hard :)
We've thought about this in the past, and even Microsoft said it was going to happen for Vista, but they realized in the end, like we did a few years previously, that it would require full support of all PCI drivers as well (if you rebalance stuff that is already bound to a driver.) So they dropped it.
When would you want to do this kind of rebalancing? Before any PCI driver is bound to any devices? Or afterwards?I guess if we want the rebalance dynamic, then we should have it full -- the rebalance would be functional even after the driver is loaded.
But in most cases, there will be problem when we unload driver from a hard disk controller, etc. We can mount root on a ramdisk and do the rebalance there, but it's complicated for a real user.
Quoted 6 lines
Quoted 3 lines
So looks like doing rebalancing before any driver is bound to any device is also a nice idea, if user can get a shell to do rebalance before built-in PCI driver grabs device.
That's not going to work, it needs to happen before any PCI device is bound, which is before init runs.
I don't think it can work either. Then we have to do rebalance after the driver bounding. But what should we do if we can't unload the driver (hard disk controller, etc.)?
Thanks,
Yu
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
In reply to Zhao, YuQuoted 69 lines · Yu
Quoted 68 lines · Greg KH
Quoted 67 lines · Yu
Quoted 48 lines · Greg KH
Quoted 33 lines · Yu
Quoted 24 lines · Greg KH
Quoted 13 lines · Yu Zhao
Documentation/kernel-parameters.txt | 10 ++++++++++ 1 files changed, 10 insertions(+), 0 deletions(-)Documentation/kernel-parameters.txt +0
-0@@ -1677,6 +1677,16 @@ and is between 256 and 4096 characters. It isdefined in the file
Documentation/kernel-parameters.txt +0
-0cbmemsize=nn[KMG] The fixed amount of bus space which is reserved for the CardBus bridge's memory window. The default value is 64 megabytes.Documentation/kernel-parameters.txt
Quoted 10 lines
Documentation/kernel-parameters.txt +10
-0+ assign-mmio=[dddd:]bb [X86] reassign memory resources of all + devices under bus [dddd:]bb (dddd is the domain + number and bb is the bus number). + assign-pio=[dddd:]bb [X86] reassign io port resources of all + devices under bus [dddd:]bb (dddd is the domain + number and bb is the bus number). + align-mmio=[dddd:]bb:dd.f [X86] relocate memory resources of a + device to minimum PAGE_SIZE alignment (dddd is + the domain number and bb, dd and f is the bus, + device and function number).Quoted 8 lines
This seems like a big problem. How are we going to know to add these command line options for devices we haven't even seen/known about yet? How do we know the bus ids aren't going to change between boots (hint, they are, pci bus ids change all the time...)
We need to be able to do this kind of thing dynamically, not fixed at boot time, which seems way to early to even know about this, right? thanks,
greg k-hQuoted 5 lines
Yes, I totally agree. Doing things dynamically is better.
The purpose of these parameters is to rebalance and align resources for device that has BARs encapsulated in various new capabilities (SR-IOV, etc.), because most of existing BIOSes don't take care of those BARs.
But how are you going to know what the proper device ids are going to be before the machine boots? I don't see how these options are ever going to work properly for a "real" user.
Quoted 3 lines
If we do resource rebalance after system is up, do you think there is any side effect or impact to other subsystem other than PCI (e.g. MTRR)?
I don't think so.
Quoted 2 lines
I haven't had much thinking on the dynamical resource rebalance. If you have any idea about this, can you please suggest?
Quoted 8 lines
Yeah, it's going to be hard :)
We've thought about this in the past, and even Microsoft said it was going to happen for Vista, but they realized in the end, like we did a few years previously, that it would require full support of all PCI drivers as well (if you rebalance stuff that is already bound to a driver.) So they dropped it.
When would you want to do this kind of rebalancing? Before any PCI driver is bound to any devices? Or afterwards?I guess if we want the rebalance dynamic, then we should have it full -- the rebalance would be functional even after the driver is loaded.
But in most cases, there will be problem when we unload driver from a hard disk controller, etc. We can mount root on a ramdisk and do the rebalance there, but it's complicated for a real user.
So looks like doing rebalancing before any driver is bound to any device is also a nice idea, if user can get a shell to do rebalance before built-in PCI driver grabs device.
Quoted 6 lines
Quoted 2 lines
That's not going to work, it needs to happen before any PCI device is bound, which is before init runs.
I don't think it can work either. Then we have to do rebalance after the driver bounding. But what should we do if we can't unload the driver (hard disk controller, etc.)?
Well, to do it "correctly" you are going to have to tell the driver to shut itself down, and reinitialize itself.
Turns out, that doesn't really work for disk and network devices without dropping the connection (well, network devices should be fine probably).
So you just can't do this, sorry. That's why the BIOS handles all of these issues in a PCI hotplug system.
How does the hardware people think we are going to handle this in the OS? It's not something that any operating system can do, is it part of the IOV PCI spec somewhere?
thanks,
greg k-h
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
In reply to Zhao, YuQuoted 69 lines · Yu
Quoted 68 lines · Greg KH
Quoted 67 lines · Yu
Quoted 48 lines · Greg KH
Quoted 33 lines · Yu
Quoted 24 lines · Greg KH
Quoted 13 lines · Yu Zhao
Documentation/kernel-parameters.txt | 10 ++++++++++ 1 files changed, 10 insertions(+), 0 deletions(-)Documentation/kernel-parameters.txt +0
-0@@ -1677,6 +1677,16 @@ and is between 256 and 4096 characters. It isdefined in the file
Documentation/kernel-parameters.txt +0
-0cbmemsize=nn[KMG] The fixed amount of bus space which is reserved for the CardBus bridge's memory window. The default value is 64 megabytes.Documentation/kernel-parameters.txt
Quoted 10 lines
Documentation/kernel-parameters.txt +10
-0+ assign-mmio=[dddd:]bb [X86] reassign memory resources of all + devices under bus [dddd:]bb (dddd is the domain + number and bb is the bus number). + assign-pio=[dddd:]bb [X86] reassign io port resources of all + devices under bus [dddd:]bb (dddd is the domain + number and bb is the bus number). + align-mmio=[dddd:]bb:dd.f [X86] relocate memory resources of a + device to minimum PAGE_SIZE alignment (dddd is + the domain number and bb, dd and f is the bus, + device and function number).Quoted 8 lines
This seems like a big problem. How are we going to know to add these command line options for devices we haven't even seen/known about yet? How do we know the bus ids aren't going to change between boots (hint, they are, pci bus ids change all the time...)
We need to be able to do this kind of thing dynamically, not fixed at boot time, which seems way to early to even know about this, right? thanks,
greg k-hQuoted 5 lines
Yes, I totally agree. Doing things dynamically is better.
The purpose of these parameters is to rebalance and align resources for device that has BARs encapsulated in various new capabilities (SR-IOV, etc.), because most of existing BIOSes don't take care of those BARs.
But how are you going to know what the proper device ids are going to be before the machine boots? I don't see how these options are ever going to work properly for a "real" user.
Quoted 3 lines
If we do resource rebalance after system is up, do you think there is any side effect or impact to other subsystem other than PCI (e.g. MTRR)?
I don't think so.
Quoted 2 lines
I haven't had much thinking on the dynamical resource rebalance. If you have any idea about this, can you please suggest?
Quoted 8 lines
Yeah, it's going to be hard :)
We've thought about this in the past, and even Microsoft said it was going to happen for Vista, but they realized in the end, like we did a few years previously, that it would require full support of all PCI drivers as well (if you rebalance stuff that is already bound to a driver.) So they dropped it.
When would you want to do this kind of rebalancing? Before any PCI driver is bound to any devices? Or afterwards?I guess if we want the rebalance dynamic, then we should have it full -- the rebalance would be functional even after the driver is loaded.
But in most cases, there will be problem when we unload driver from a hard disk controller, etc. We can mount root on a ramdisk and do the rebalance there, but it's complicated for a real user.
So looks like doing rebalancing before any driver is bound to any device is also a nice idea, if user can get a shell to do rebalance before built-in PCI driver grabs device.
Quoted 6 lines
Quoted 2 lines
That's not going to work, it needs to happen before any PCI device is bound, which is before init runs.
I don't think it can work either. Then we have to do rebalance after the driver bounding. But what should we do if we can't unload the driver (hard disk controller, etc.)?
Well, to do it "correctly" you are going to have to tell the driver to shut itself down, and reinitialize itself.
Turns out, that doesn't really work for disk and network devices without dropping the connection (well, network devices should be fine probably).
So you just can't do this, sorry. That's why the BIOS handles all of these issues in a PCI hotplug system.
How does the hardware people think we are going to handle this in the OS? It's not something that any operating system can do, is it part of the IOV PCI spec somewhere?
thanks,
greg k-h
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
Quoted 46 lines · Greg KH
Quoted 30 lines · Yu
Quoted 29 lines · Greg KH
Quoted 28 lines · Yu
Quoted 27 lines · Greg KH
Quoted 9 lines · Yu
Quoted 8 lines · Greg KH
This seems like a big problem. How are we going to know to add these command line options for devices we haven't even seen/known about yet? How do we know the bus ids aren't going to change between boots (hint, they are, pci bus ids change all the time...)
We need to be able to do this kind of thing dynamically, not fixed at boot time, which seems way to early to even know about this, right? thanks,
greg k-hQuoted 3 lines
Yes, I totally agree. Doing things dynamically is better.
The purpose of these parameters is to rebalance and align resources for device that has BARs encapsulated in various new capabilities (SR-IOV, etc.), because most of existing BIOSes don't take care of those BARs.
But how are you going to know what the proper device ids are going to be before the machine boots? I don't see how these options are ever going to work properly for a "real" user.
Quoted 1 line
If we do resource rebalance after system is up, do you think there is any side effect or impact to other subsystem other than PCI (e.g. MTRR)?
I don't think so.
Quoted 1 line
I haven't had much thinking on the dynamical resource rebalance. If you have any idea about this, can you please suggest?
Yeah, it's going to be hard :)
We've thought about this in the past, and even Microsoft said it was going to happen for Vista, but they realized in the end, like we did a few years previously, that it would require full support of all PCI drivers as well (if you rebalance stuff that is already bound to a driver.) So they dropped it.
When would you want to do this kind of rebalancing? Before any PCI driver is bound to any devices? Or afterwards?Quoted 8 lines
Quoted 7 lines
Quoted 5 lines
I guess if we want the rebalance dynamic, then we should have it full -- the rebalance would be functional even after the driver is loaded.
But in most cases, there will be problem when we unload driver from a hard disk controller, etc. We can mount root on a ramdisk and do the rebalance there, but it's complicated for a real user.
So looks like doing rebalancing before any driver is bound to any device is also a nice idea, if user can get a shell to do rebalance before built-in PCI driver grabs device.
That's not going to work, it needs to happen before any PCI device is bound, which is before init runs.
I don't think it can work either. Then we have to do rebalance after the driver bounding. But what should we do if we can't unload the driver (hard disk controller, etc.)?
Well, to do it "correctly" you are going to have to tell the driver to shut itself down, and reinitialize itself.
Turns out, that doesn't really work for disk and network devices without dropping the connection (well, network devices should be fine probably).
Quoted 6 lines
So you just can't do this, sorry. That's why the BIOS handles all of these issues in a PCI hotplug system.
How does the hardware people think we are going to handle this in the OS? It's not something that any operating system can do, is it part of the IOV PCI spec somewhere?
No, it's not part of the PCI IOV spec.
I just want the IOV (and whole PCI subsystem) have more flexibility on various BIOSes. So can we reconsider about resource rebalance as boot option, or should we forget about this idea?
Regards,
Yu
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
In reply to Greg KHQuoted 58 lines · Greg KH
Quoted 42 lines · Yu
Quoted 41 lines · Greg KH
Quoted 40 lines · Yu
Quoted 24 lines · Greg KH
Quoted 9 lines · Yu
Quoted 8 lines · Greg KH
This seems like a big problem. How are we going to know to add these command line options for devices we haven't even seen/known about yet? How do we know the bus ids aren't going to change between boots (hint, they are, pci bus ids change all the time...)
We need to be able to do this kind of thing dynamically, not fixed at boot time, which seems way to early to even know about this, right? thanks,
greg k-hQuoted 5 lines
Yes, I totally agree. Doing things dynamically is better.
The purpose of these parameters is to rebalance and align resources for device that has BARs encapsulated in various new capabilities (SR-IOV, etc.), because most of existing BIOSes don't take care of those BARs.
But how are you going to know what the proper device ids are going to be before the machine boots? I don't see how these options are ever going to work properly for a "real" user.
Quoted 3 lines
If we do resource rebalance after system is up, do you think there is any side effect or impact to other subsystem other than PCI (e.g. MTRR)?
I don't think so.
Quoted 2 lines
I haven't had much thinking on the dynamical resource rebalance. If you have any idea about this, can you please suggest?
Quoted 8 lines
Yeah, it's going to be hard :)
We've thought about this in the past, and even Microsoft said it was going to happen for Vista, but they realized in the end, like we did a few years previously, that it would require full support of all PCI drivers as well (if you rebalance stuff that is already bound to a driver.) So they dropped it.
When would you want to do this kind of rebalancing? Before any PCI driver is bound to any devices? Or afterwards?I guess if we want the rebalance dynamic, then we should have it full -- the rebalance would be functional even after the driver is loaded.
But in most cases, there will be problem when we unload driver from a hard disk controller, etc. We can mount root on a ramdisk and do the rebalance there, but it's complicated for a real user.
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
So looks like doing rebalancing before any driver is bound to any device is also a nice idea, if user can get a shell to do rebalance before built-in PCI driver grabs device.
That's not going to work, it needs to happen before any PCI device is bound, which is before init runs.
I don't think it can work either. Then we have to do rebalance after the driver bounding. But what should we do if we can't unload the driver (hard disk controller, etc.)?
Well, to do it "correctly" you are going to have to tell the driver to shut itself down, and reinitialize itself.
Turns out, that doesn't really work for disk and network devices without dropping the connection (well, network devices should be fine probably).
Quoted 6 lines
So you just can't do this, sorry. That's why the BIOS handles all of these issues in a PCI hotplug system.
How does the hardware people think we are going to handle this in the OS? It's not something that any operating system can do, is it part of the IOV PCI spec somewhere?
No, it's not part of the PCI IOV spec.
I just want the IOV (and whole PCI subsystem) have more flexibility on various BIOSes. So can we reconsider about resource rebalance as boot option, or should we forget about this idea?
Regards,
Yu
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
In reply to Zhao, YuQuoted 10 lines · Yu
Quoted 9 lines
Well, to do it "correctly" you are going to have to tell the driver to shut itself down, and reinitialize itself.
Turns out, that doesn't really work for disk and network devices without dropping the connection (well, network devices should be fine probably). So you just can't do this, sorry. That's why the BIOS handles all of these issues in a PCI hotplug system.
How does the hardware people think we are going to handle this in the OS? It's not something that any operating system can do, is it part of the IOV PCI spec somewhere?
Quoted 5 lines
No, it's not part of the PCI IOV spec.
I just want the IOV (and whole PCI subsystem) have more flexibility on various BIOSes. So can we reconsider about resource rebalance as boot option, or should we forget about this idea?
As you have proposed it, the boot option will not work at all, so I think we need to forget about it. Especially if it is not really needed.
thanks,
greg k-h
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
In reply to Zhao, YuQuoted 10 lines · Yu
Quoted 9 lines
Well, to do it "correctly" you are going to have to tell the driver to shut itself down, and reinitialize itself.
Turns out, that doesn't really work for disk and network devices without dropping the connection (well, network devices should be fine probably). So you just can't do this, sorry. That's why the BIOS handles all of these issues in a PCI hotplug system.
How does the hardware people think we are going to handle this in the OS? It's not something that any operating system can do, is it part of the IOV PCI spec somewhere?
Quoted 5 lines
No, it's not part of the PCI IOV spec.
I just want the IOV (and whole PCI subsystem) have more flexibility on various BIOSes. So can we reconsider about resource rebalance as boot option, or should we forget about this idea?
As you have proposed it, the boot option will not work at all, so I think we need to forget about it. Especially if it is not really needed.
thanks,
greg k-h
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
Quoted 10 lines · Greg KH
Quoted 9 lines · Yu
Quoted 9 lines
Well, to do it "correctly" you are going to have to tell the driver to shut itself down, and reinitialize itself.
Turns out, that doesn't really work for disk and network devices without dropping the connection (well, network devices should be fine probably). So you just can't do this, sorry. That's why the BIOS handles all of these issues in a PCI hotplug system.
How does the hardware people think we are going to handle this in the OS? It's not something that any operating system can do, is it part of the IOV PCI spec somewhere?
Quoted 7 lines
Quoted 3 lines
No, it's not part of the PCI IOV spec.
I just want the IOV (and whole PCI subsystem) have more flexibility on various BIOSes. So can we reconsider about resource rebalance as boot option, or should we forget about this idea?
As you have proposed it, the boot option will not work at all, so I think we need to forget about it. Especially if it is not really needed.
I guess at least one thing would work if people don't want to boot twice: give the bus number 0 as rebalance starting point, then all system resources would be reshuffled :-)
Thanks,
Yu
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
In reply to Greg KHQuoted 12 lines · Greg KH
Quoted 11 lines · Yu
Quoted 9 lines
Well, to do it "correctly" you are going to have to tell the driver to shut itself down, and reinitialize itself.
Turns out, that doesn't really work for disk and network devices without dropping the connection (well, network devices should be fine probably). So you just can't do this, sorry. That's why the BIOS handles all of these issues in a PCI hotplug system.
How does the hardware people think we are going to handle this in the OS? It's not something that any operating system can do, is it part of the IOV PCI spec somewhere?No, it's not part of the PCI IOV spec.
Quoted 7 lines
Quoted 3 lines
I just want the IOV (and whole PCI subsystem) have more flexibility on various BIOSes. So can we reconsider about resource rebalance as boot option, or should we forget about this idea?
As you have proposed it, the boot option will not work at all, so I think we need to forget about it. Especially if it is not really needed.
I guess at least one thing would work if people don't want to boot
twice: give the bus number 0 as rebalance starting point, then all system resources would be reshuffled :-)
Thanks,
Yu
Re: git repository for SR-IOV development?
In reply to Greg KHHello Lance,
Thanks for your interest in SR-IOV. As Greg said we can't have a git tree for the change, but you are welcome to ask any question here and I also will keep you informed if there is any update on the SR-IOV patches.
Thanks,
Yu
Quoted 34 lines · Greg KH
Quoted 12 lines · H L
Quoted 9 lines · Greg KH
Quoted 1 line · H L
Has anyone initiated or given consideration to the
creation of a git
Quoted 1 line
repository (say, on kernel.org) for SR-IOV
development?
Why? It's only a few patches, right? Why would it need a whole new git
tree?Quoted 8 lines
So as to minimize the time and effort patching a kernel, especially if the tree (and/or hash level) against which the patches were created fails to be specified on a mailing-list. Plus, there appears to be questions raised on how, precisely, the implementation should ultimately be modeled and especially given that, who knows at this point what number of patches will ultimately be submitted? I know I've built the "7-patch" one (painfully, by the way), and I'm aware there's another 15-patch set out there which I've not yet examined.
It's a mere 7 or 15 patches, you don't need a whole git tree for something small like that.
Especially as there only seems to be one developer doing real work...
thanks,
greg k-h
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: git repository for SR-IOV development?
In reply to Greg KHHello Lance,
Thanks for your interest in SR-IOV. As Greg said we can't have a git tree for the change, but you are welcome to ask any question here and I also will keep you informed if there is any update on the SR-IOV patches.
Thanks,
Yu
Quoted 34 lines · Greg KH
Quoted 12 lines · H L
Quoted 9 lines · Greg KH
Quoted 1 line · H L
Has anyone initiated or given consideration to the
creation of a git
Quoted 1 line
repository (say, on kernel.org) for SR-IOV
development?
Why? It's only a few patches, right? Why would it need a whole new git
tree?Quoted 8 lines
So as to minimize the time and effort patching a kernel, especially if the tree (and/or hash level) against which the patches were created fails to be specified on a mailing-list. Plus, there appears to be questions raised on how, precisely, the implementation should ultimately be modeled and especially given that, who knows at this point what number of patches will ultimately be submitted? I know I've built the "7-patch" one (painfully, by the way), and I'm aware there's another 15-patch set out there which I've not yet examined.
It's a mere 7 or 15 patches, you don't need a whole git tree for something small like that.
Especially as there only seems to be one developer doing real work...
thanks,
greg k-h
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 9 lines · Anthony Liguori
What we would rather do in KVM, is have the VFs appear in the host as standard network devices. We would then like to back our existing PV driver to this VF directly bypassing the host networking stack. A key feature here is being able to fill the VF's receive queue with guest memory instead of host kernel memory so that you can get zero-copy receive traffic. This will perform just as well as doing passthrough (at least) and avoid all that ugliness of dealing with SR-IOV in the guest.
But you shift a lot of ugliness into the host network stack again. Not sure that is a good trade off.
Also it would always require context switches and I believe one of the reasons for the PV/VF model is very low latency IO and having heavyweight switches to the host and back would be against that.
-Andi
Signature · 1 line
[email protected]
While we are arguing what the software model the SR-IOV should be, let me ask two simple questions first:
1, What does the SR-IOV looks like?
2, Why do we need to support it?
I'm sure people have different understandings from their own view points. No one is wrong, but, please don't make thing complicated and don't ignore user requirements.
PCI SIG and hardware vendors create such thing intending to make hardware resource in one PCI device be shared from different software instances -- I guess all of us agree with this. No doubt PF is real function in the PCI device, but VF is different? No, it also has its own Bus, Device and Function numbers, and PCI configuration space and Memory Space (MMIO). To be more detailed, it can response to and initiate PCI Transaction Layer Protocol packets, which means it can do everything a PF can in PCI level. From these obvious behaviors, we can conclude PCI SIG model VF as a normal PCI device function, even it's not standalone.
As you know the Linux kernel is the base of various virtual machine monitors such as KVM, Xen, OpenVZ and VServer. We need SR-IOV support in the kernel because mostly it helps high-end users (IT departments, HPC, etc.) to share limited hardware resources among hundreds or even thousands virtual machines and hence reduce the cost. How can we make these virtual machine monitors utilize the advantage of SR-IOV without spending too much effort meanwhile remaining architectural correctness? I believe making VF represent as much closer as a normal PCI device (struct pci_dev) is the best way in current situation, because this is not only what the hardware designers expect us to do but also the usage model that KVM, Xen and other VMMs have already supported.
I agree that API in the SR-IOV pacth is arguable and the concerns such as lack of PF driver, etc. are also valid. But I personally think these stuff are not essential problems to me and other SR-IOV driver developers. People can refine things but don't want to recreate things in another totally different way especially that way doesn't bring them obvious benefits.
As I can see that we are now reaching a point that a decision must be made, I know this is such difficult thing in an open and free community but fortunately we have a lot of talented and experienced people here. So let's make it happen, and keep our loyal users happy!
Thanks,
Yu
While we are arguing what the software model the SR-IOV should be, let me ask two simple questions first:
1, What does the SR-IOV looks like?
2, Why do we need to support it?
I'm sure people have different understandings from their own view points. No one is wrong, but, please don't make thing complicated and don't ignore user requirements.
PCI SIG and hardware vendors create such thing intending to make hardware resource in one PCI device be shared from different software instances -- I guess all of us agree with this. No doubt PF is real function in the PCI device, but VF is different? No, it also has its own Bus, Device and Function numbers, and PCI configuration space and Memory Space (MMIO). To be more detailed, it can response to and initiate PCI Transaction Layer Protocol packets, which means it can do everything a PF can in PCI level. From these obvious behaviors, we can conclude PCI SIG model VF as a normal PCI device function, even it's not standalone.
As you know the Linux kernel is the base of various virtual machine monitors such as KVM, Xen, OpenVZ and VServer. We need SR-IOV support in the kernel because mostly it helps high-end users (IT departments, HPC, etc.) to share limited hardware resources among hundreds or even thousands virtual machines and hence reduce the cost. How can we make these virtual machine monitors utilize the advantage of SR-IOV without spending too much effort meanwhile remaining architectural correctness? I believe making VF represent as much closer as a normal PCI device (struct pci_dev) is the best way in current situation, because this is not only what the hardware designers expect us to do but also the usage model that KVM, Xen and other VMMs have already supported.
I agree that API in the SR-IOV pacth is arguable and the concerns such as lack of PF driver, etc. are also valid. But I personally think these stuff are not essential problems to me and other SR-IOV driver developers. People can refine things but don't want to recreate things in another totally different way especially that way doesn't bring them obvious benefits.
As I can see that we are now reaching a point that a decision must be made, I know this is such difficult thing in an open and free community but fortunately we have a lot of talented and experienced people here. So let's make it happen, and keep our loyal users happy!
Thanks,
Yu
Quoted 9 lines · Anthony Liguori
What we would rather do in KVM, is have the VFs appear in the host as standard network devices. We would then like to back our existing PV driver to this VF directly bypassing the host networking stack. A key feature here is being able to fill the VF's receive queue with guest memory instead of host kernel memory so that you can get zero-copy receive traffic. This will perform just as well as doing passthrough (at least) and avoid all that ugliness of dealing with SR-IOV in the guest.
But you shift a lot of ugliness into the host network stack again. Not sure that is a good trade off.
Also it would always require context switches and I believe one of the reasons for the PV/VF model is very low latency IO and having heavyweight switches to the host and back would be against that.
-Andi
Signature · 1 line
[email protected]
Quoted 16 lines · Anthony Liguori
Quoted 15 lines · Matthew Wilcox
[Anna, can you fix your word-wrapping please? Your lines appear to be infinitely long which is most unpleasant to reply to]
Quoted 9 lines · Anna
Quoted 3 lines
Where would the VF drivers have to be associated? On the "
pci_dev" level or on a higher one?A VF appears to the Linux OS as a standard (full, additional) PCI device. The driver is associated in the same way as for a normal PCI device. Ideally, you would use SR-IOV devices on a virtualized system, for example, using Xen. A VF can then be assigned to a guest domain as a full PCI device.
Quoted 7 lines
Quoted 5 lines
It's not clear thats the right solution. If the VF devices are _only_ going to be used by the guest, then arguably, we don't want to create
pci_devsfor them in the host. (I think it _is_ the right answer, but I want to make it clear there's multiple opinions on this).The VFs shouldn't be limited to being used by the guest.
Yes, VF driver running in the host is supported :-)
Quoted 2 lines
SR-IOV is actually an incredibly painful thing. You need to have a VF driver in the guest, do hardware pass through, have a PV driver stub in the guest that's hypervisor specific (a VF is not usable on it's own), have a device specific backend in the VMM, and if you want to do live migration, have another PV driver in the guest that you can do teaming with. Just a mess.
Actually not so mess. VF driver can be a plain PCI device driver and doesn't require any backend in the VMM, or hypervisor specific knowledge, if the hardware is properly designed. In this case PF driver controls hardware resource allocation for VFs and VF driver can work without any communication to PF driver or VMM.
Quoted 2 lines
What we would rather do in KVM, is have the VFs appear in the host as standard network devices. We would then like to back our existing PV driver to this VF directly bypassing the host networking stack. A key feature here is being able to fill the VF's receive queue with guest memory instead of host kernel memory so that you can get zero-copy receive traffic. This will perform just as well as doing passthrough (at least) and avoid all that ugliness of dealing with SR-IOV in the guest.
If the hardware supports both SR-IOV and IOMMU, I wouldn't suggest people to do so, because they will get better performance by directly assigning VF to the guest.
However, lots of low-end machines don't have SR-IOV and IOMMU support. They may have multi queue NIC, which uses built-in L2 switch to dispense packets to different DMA queue according to MAC address. They definitely can benefit a lot if there is software support for the DMA queue hooking virtio-net backend as you suggested.
Quoted 4 lines
This eliminates all of the mess of various drivers in the guest and all the associated baggage of doing hardware passthrough.
So IMHO, having VFs be usable in the host is absolutely critical because I think it's the only reasonable usage model.
Please don't worry, we have take this usage model as well as container model into account when designing SR-IOV framework for the kernel.
Quoted 8 lines
Regards,
Anthony Liguori
Signature · 3 lines
To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 16 lines · Anthony Liguori
Quoted 15 lines · Matthew Wilcox
[Anna, can you fix your word-wrapping please? Your lines appear to be infinitely long which is most unpleasant to reply to]
Quoted 9 lines · Anna
Quoted 3 lines
Where would the VF drivers have to be associated? On the "
pci_dev" level or on a higher one?A VF appears to the Linux OS as a standard (full, additional) PCI device. The driver is associated in the same way as for a normal PCI device. Ideally, you would use SR-IOV devices on a virtualized system, for example, using Xen. A VF can then be assigned to a guest domain as a full PCI device.
Quoted 7 lines
Quoted 5 lines
It's not clear thats the right solution. If the VF devices are _only_ going to be used by the guest, then arguably, we don't want to create
pci_devsfor them in the host. (I think it _is_ the right answer, but I want to make it clear there's multiple opinions on this).The VFs shouldn't be limited to being used by the guest.
Yes, VF driver running in the host is supported :-)
Quoted 7 lines
SR-IOV is actually an incredibly painful thing. You need to have a VF driver in the guest, do hardware pass through, have a PV driver stub in the guest that's hypervisor specific (a VF is not usable on it's own), have a device specific backend in the VMM, and if you want to do live migration, have another PV driver in the guest that you can do teaming with. Just a mess.
Actually not so mess. VF driver can be a plain PCI device driver and doesn't require any backend in the VMM, or hypervisor specific knowledge, if the hardware is properly designed. In this case PF driver controls hardware resource allocation for VFs and VF driver can work without any communication to PF driver or VMM.
Quoted 8 lines
What we would rather do in KVM, is have the VFs appear in the host as standard network devices. We would then like to back our existing PV driver to this VF directly bypassing the host networking stack. A key feature here is being able to fill the VF's receive queue with guest memory instead of host kernel memory so that you can get zero-copy receive traffic. This will perform just as well as doing passthrough (at least) and avoid all that ugliness of dealing with SR-IOV in the guest.
If the hardware supports both SR-IOV and IOMMU, I wouldn't suggest people to do so, because they will get better performance by directly assigning VF to the guest.
However, lots of low-end machines don't have SR-IOV and IOMMU support. They may have multi queue NIC, which uses built-in L2 switch to dispense packets to different DMA queue according to MAC address. They definitely can benefit a lot if there is software support for the DMA queue hooking virtio-net backend as you suggested.
Quoted 6 lines
This eliminates all of the mess of various drivers in the guest and all the associated baggage of doing hardware passthrough.
So IMHO, having VFs be usable in the host is absolutely critical because I think it's the only reasonable usage model.
Please don't worry, we have take this usage model as well as container model into account when designing SR-IOV framework for the kernel.
Quoted 8 lines
Regards,
Anthony Liguori
Signature · 3 lines
To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 5 lines · Yu Zhao
While we are arguing what the software model the SR-IOV should be, let me ask two simple questions first:
1, What does the SR-IOV looks like?
2, Why do we need to support it?
I don't think we need to worry about those questions, as we can see what the SR-IOV interface looks like by looking at the PCI spec, and we know Linux needs to support it, as Linux needs to support everything :)
(note, community members that can not see the PCI specs at this point in time, please know that we are working on resolving these issues, hopefully we will have some good news within a month or so.)
Quoted 11 lines
As you know the Linux kernel is the base of various virtual machine monitors such as KVM, Xen, OpenVZ and VServer. We need SR-IOV support in the kernel because mostly it helps high-end users (IT departments, HPC, etc.) to share limited hardware resources among hundreds or even thousands virtual machines and hence reduce the cost. How can we make these virtual machine monitors utilize the advantage of SR-IOV without spending too much effort meanwhile remaining architectural correctness? I believe making VF represent as much closer as a normal PCI device (struct
pci_dev) is the best way in current situation, because this is not only what the hardware designers expect us to do but also the usage model that KVM, Xen and other VMMs have already supported.
But would such an api really take advantage of the new IOV interfaces that are exposed by the new device type?
Quoted 3 lines
I agree that API in the SR-IOV pacth is arguable and the concerns such as lack of PF driver, etc. are also valid. But I personally think these stuff are not essential problems to me and other SR-IOV driver developers.
How can the lack of a PF driver not be a valid concern at this point in time? Without such a driver written, how can we know that the SR-IOV interface as created is sufficient, or that it even works properly?
Here's what I see we need to have before we can evaluate if the IOV core PCI patches are acceptable:
- a driver that uses this interface
- a PF driver that uses this interface.
Without those, we can't determine if the infrastructure provided by the IOV core even is sufficient, right?
Rumor has it that there is both of the above things floating around, can someone please post them to the linux-pci list so that we can see how this all works together?
thanks,
greg k-h
Quoted 5 lines · Yu Zhao
While we are arguing what the software model the SR-IOV should be, let me ask two simple questions first:
1, What does the SR-IOV looks like?
2, Why do we need to support it?
I don't think we need to worry about those questions, as we can see what the SR-IOV interface looks like by looking at the PCI spec, and we know Linux needs to support it, as Linux needs to support everything :)
(note, community members that can not see the PCI specs at this point in time, please know that we are working on resolving these issues, hopefully we will have some good news within a month or so.)
Quoted 11 lines
As you know the Linux kernel is the base of various virtual machine monitors such as KVM, Xen, OpenVZ and VServer. We need SR-IOV support in the kernel because mostly it helps high-end users (IT departments, HPC, etc.) to share limited hardware resources among hundreds or even thousands virtual machines and hence reduce the cost. How can we make these virtual machine monitors utilize the advantage of SR-IOV without spending too much effort meanwhile remaining architectural correctness? I believe making VF represent as much closer as a normal PCI device (struct
pci_dev) is the best way in current situation, because this is not only what the hardware designers expect us to do but also the usage model that KVM, Xen and other VMMs have already supported.
But would such an api really take advantage of the new IOV interfaces that are exposed by the new device type?
Quoted 3 lines
I agree that API in the SR-IOV pacth is arguable and the concerns such as lack of PF driver, etc. are also valid. But I personally think these stuff are not essential problems to me and other SR-IOV driver developers.
How can the lack of a PF driver not be a valid concern at this point in time? Without such a driver written, how can we know that the SR-IOV interface as created is sufficient, or that it even works properly?
Here's what I see we need to have before we can evaluate if the IOV core PCI patches are acceptable:
- a driver that uses this interface
- a PF driver that uses this interface.
Without those, we can't determine if the infrastructure provided by the IOV core even is sufficient, right?
Rumor has it that there is both of the above things floating around, can someone please post them to the linux-pci list so that we can see how this all works together?
thanks,
greg k-h
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
In reply to Zhao, YuQuoted 16 lines · Yu
Quoted 15 lines · Greg KH
Quoted 14 lines · Yu
Quoted 9 lines
Well, to do it "correctly" you are going to have to tell the driver to shut itself down, and reinitialize itself.
Turns out, that doesn't really work for disk and network devices without dropping the connection (well, network devices should be fine probably). So you just can't do this, sorry. That's why the BIOS handles all of these issues in a PCI hotplug system.
How does the hardware people think we are going to handle this in the OS? It's not something that any operating system can do, is it part of the IOV PCI spec somewhere?No, it's not part of the PCI IOV spec.
I just want the IOV (and whole PCI subsystem) have more flexibility on various BIOSes. So can we reconsider about resource rebalance as boot option, or should we forget about this idea?
Quoted 7 lines
Quoted 3 lines
As you have proposed it, the boot option will not work at all, so I think we need to forget about it. Especially if it is not really needed.
I guess at least one thing would work if people don't want to boot twice:
give the bus number 0 as rebalance starting point, then all system resources would be reshuffled :-)
Hm, but don't we do that today with our basic resource reservation logic at boot time? What would be different about this kind of proposal?
thanks,
greg k-h
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
In reply to Zhao, YuQuoted 16 lines · Yu
Quoted 15 lines · Greg KH
Quoted 14 lines · Yu
Quoted 9 lines
Well, to do it "correctly" you are going to have to tell the driver to shut itself down, and reinitialize itself.
Turns out, that doesn't really work for disk and network devices without dropping the connection (well, network devices should be fine probably). So you just can't do this, sorry. That's why the BIOS handles all of these issues in a PCI hotplug system.
How does the hardware people think we are going to handle this in the OS? It's not something that any operating system can do, is it part of the IOV PCI spec somewhere?No, it's not part of the PCI IOV spec.
I just want the IOV (and whole PCI subsystem) have more flexibility on various BIOSes. So can we reconsider about resource rebalance as boot option, or should we forget about this idea?
Quoted 7 lines
Quoted 3 lines
As you have proposed it, the boot option will not work at all, so I think we need to forget about it. Especially if it is not really needed.
I guess at least one thing would work if people don't want to boot twice:
give the bus number 0 as rebalance starting point, then all system resources would be reshuffled :-)
Hm, but don't we do that today with our basic resource reservation logic at boot time? What would be different about this kind of proposal?
thanks,
greg k-h
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
In reply to Greg KHQuoted 20 lines · Greg KH
Quoted 19 lines · Yu
Quoted 18 lines · Greg KH
Quoted 9 lines · Yu
Quoted 9 lines
Well, to do it "correctly" you are going to have to tell the driver to shut itself down, and reinitialize itself.
Turns out, that doesn't really work for disk and network devices without dropping the connection (well, network devices should be fine probably). So you just can't do this, sorry. That's why the BIOS handles all of these issues in a PCI hotplug system.
How does the hardware people think we are going to handle this in the OS? It's not something that any operating system can do, is it part of the IOV PCI spec somewhere?Quoted 5 lines
No, it's not part of the PCI IOV spec.
I just want the IOV (and whole PCI subsystem) have more flexibility on various BIOSes. So can we reconsider about resource rebalance as boot option, or should we forget about this idea?
As you have proposed it, the boot option will not work at all, so I think we need to forget about it. Especially if it is not really needed.
Quoted 6 lines
Quoted 3 lines
I guess at least one thing would work if people don't want to boot twice:
give the bus number 0 as rebalance starting point, then all system resources would be reshuffled :-)Hm, but don't we do that today with our basic resource reservation logic at boot time? What would be different about this kind of proposal?
The generic PCI core can do this but this feature is kind of disabled by low level PCI code in x86. The low level code tries to reserve resource according to configuration from BIOS. If the BIOS is wrong, the allocation would fail and the generic PCI core couldn't repair it because the bridge resources may have been allocated by the PCI low level and the PCI core can't expand them to find enough resource for the subordinates.
The proposal is to disable x86 PCI low level to allocation resources according to BIOS so PCI core can fully control the resource allocation. The PCI core takes all resources from BARs it knows into account and configure the resource windows on the bridges according to its own calculation.
Regards,
Yu
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
In reply to Greg KHQuoted 14 lines · Greg KH
Quoted 13 lines · Yu
Quoted 12 lines · Greg KH
Quoted 11 lines · Yu
Quoted 9 lines
Well, to do it "correctly" you are going to have to tell the driver to shut itself down, and reinitialize itself.
Turns out, that doesn't really work for disk and network devices without dropping the connection (well, network devices should be fine probably). So you just can't do this, sorry. That's why the BIOS handles all of these issues in a PCI hotplug system.
How does the hardware people think we are going to handle this in the OS? It's not something that any operating system can do, is it part of the IOV PCI spec somewhere?No, it's not part of the PCI IOV spec.
Quoted 8 lines
Quoted 5 lines
Quoted 4 lines
Quoted 1 line
I just want the IOV (and whole PCI subsystem) have more flexibility on various BIOSes. So can we reconsider about resource rebalance as boot option, or should we forget about this idea?
As you have proposed it, the boot option will not work at all, so I think we need to forget about it. Especially if it is not really needed.
I guess at least one thing would work if people don't want to boot twice: give the bus number 0 as rebalance starting point, then all system resources would be reshuffled :-)
Hm, but don't we do that today with our basic resource reservation logic at boot time? What would be different about this kind of proposal?
The generic PCI core can do this but this feature is kind of disabled by low level PCI code in x86. The low level code tries to reserve resource according to configuration from BIOS. If the BIOS is wrong, the allocation would fail and the generic PCI core couldn't repair it because the bridge resources may have been allocated by the PCI low level and the PCI core can't expand them to find enough resource for the subordinates.
The proposal is to disable x86 PCI low level to allocation resources according to BIOS so PCI core can fully control the resource allocation. The PCI core takes all resources from BARs it knows into account and configure the resource windows on the bridges according to its own calculation.
Regards,
Yu
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
Quoted 30 lines · Yu Zhao
Quoted 20 lines · Greg KH
Quoted 19 lines · Yu
Quoted 18 lines · Greg KH
Quoted 17 lines · Yu
Quoted 4 lines
Well, to do it "correctly" you are going to have to tell the driver to shut itself down, and reinitialize itself.
Turns out, that doesn't really work for disk and network devices withoutQuoted 8 lines
dropping the connection (well, network devices should be fine probably).
So you just can't do this, sorry. That's why the BIOS handles all of these issues in a PCI hotplug system.
How does the hardware people think we are going to handle this in the OS? It's not something that any operating system can do, is it part of
the IOV PCI spec somewhere?No, it's not part of the PCI IOV spec.
I just want the IOV (and whole PCI subsystem) have more flexibility on various BIOSes. So can we reconsider about resource rebalance as boot option, or should we forget about this idea?
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
As you have proposed it, the boot option will not work at all, so I think we need to forget about it. Especially if it is not really needed.
I guess at least one thing would work if people don't want to boot twice:
give the bus number 0 as rebalance starting point, then all system resources would be reshuffled :-)Hm, but don't we do that today with our basic resource reservation logic at boot time? What would be different about this kind of proposal?
Quoted 6 lines
The generic PCI core can do this but this feature is kind of disabled by low level PCI code in x86. The low level code tries to reserve resource according to configuration from BIOS. If the BIOS is wrong, the allocation would fail and the generic PCI core couldn't repair it because the bridge resources may have been allocated by the PCI low level and the PCI core can't expand them to find enough resource for the subordinates.
Yes, we do this on purpose.
Quoted 5 lines
The proposal is to disable x86 PCI low level to allocation resources according to BIOS so PCI core can fully control the resource allocation. The PCI core takes all resources from BARs it knows into account and configure the resource windows on the bridges according to its own calculation.
Ah, so you mean we should revert back to the way we use to do x86 PCI resource allocation from about a year and a half ago to about 8 years ago?
Hint, there was a reason why we switched over to using the BIOS instead of doing it ourselves. Turns out we have to trust the BIOS here, as that is exactly what other operating systems do. Trying to do it on our own was too fragile and resulted in too many problems over time.
Go look at the archives for when this all was switched, you'll see the reasons why.
So no, we will not be going back to the way we used to do things, we changed for a reason :)
thanks,
greg k-h
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
In reply to Yu ZhaoQuoted 30 lines · Yu Zhao
Quoted 20 lines · Greg KH
Quoted 19 lines · Yu
Quoted 18 lines · Greg KH
Quoted 17 lines · Yu
Quoted 4 lines
Well, to do it "correctly" you are going to have to tell the driver to shut itself down, and reinitialize itself.
Turns out, that doesn't really work for disk and network devices withoutQuoted 8 lines
dropping the connection (well, network devices should be fine probably).
So you just can't do this, sorry. That's why the BIOS handles all of these issues in a PCI hotplug system.
How does the hardware people think we are going to handle this in the OS? It's not something that any operating system can do, is it part of
the IOV PCI spec somewhere?No, it's not part of the PCI IOV spec.
I just want the IOV (and whole PCI subsystem) have more flexibility on various BIOSes. So can we reconsider about resource rebalance as boot option, or should we forget about this idea?
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
As you have proposed it, the boot option will not work at all, so I think we need to forget about it. Especially if it is not really needed.
I guess at least one thing would work if people don't want to boot twice:
give the bus number 0 as rebalance starting point, then all system resources would be reshuffled :-)Hm, but don't we do that today with our basic resource reservation logic at boot time? What would be different about this kind of proposal?
Quoted 6 lines
The generic PCI core can do this but this feature is kind of disabled by low level PCI code in x86. The low level code tries to reserve resource according to configuration from BIOS. If the BIOS is wrong, the allocation would fail and the generic PCI core couldn't repair it because the bridge resources may have been allocated by the PCI low level and the PCI core can't expand them to find enough resource for the subordinates.
Yes, we do this on purpose.
Quoted 5 lines
The proposal is to disable x86 PCI low level to allocation resources according to BIOS so PCI core can fully control the resource allocation. The PCI core takes all resources from BARs it knows into account and configure the resource windows on the bridges according to its own calculation.
Ah, so you mean we should revert back to the way we use to do x86 PCI resource allocation from about a year and a half ago to about 8 years ago?
Hint, there was a reason why we switched over to using the BIOS instead of doing it ourselves. Turns out we have to trust the BIOS here, as that is exactly what other operating systems do. Trying to do it on our own was too fragile and resulted in too many problems over time.
Go look at the archives for when this all was switched, you'll see the reasons why.
So no, we will not be going back to the way we used to do things, we changed for a reason :)
thanks,
greg k-h
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
In reply to Yu ZhaoDear all,
I'm glad to hear this. In fact, I'm developing for BIOS area. This feature is very useful when your system have one or more PCI/PCIe hotplug slot. Generally, BIOS reserve amount of resource for empty hotplug slot by default, but it is not always enough for all device. We have many kind of Express Modules which consume different amount of resouce, generally we reserve a small number of resouce for this, so, sometime, some Express Modules hotplug without card installed at boot time, it will not useable.
Then, Microsoft say they implement PCI Multi-level Resource Rebanlence in Vista and Server 2008, you can refer
http://www.microsoft.com/whdc/archive/multilevel-rebal.mspx
http://www.microsoft.com/whdc/connect/pci/PCI-rsc.mspx
They use a method of ACPI to tell OS that you can ignore the resource allocation of PCI devices below the bridge. I think this is more useful than specify the BUS number to ignore resouce allocation, because the BUS number often change due to some need by BIOS or new PCI/PCIe device added in system. Users generally do not know the system architecture and can not specify the BUS number of the root bridge, while if you specify the _DSM method like MS to the root bridge of hotplug slot, it is a much easier way to archive for BIOS writers.
PS:
Since my mail address was blocked by the maillist, this mail may not reach people who only in linux kernel maillist.
Thanks.
Jensen
Quoted 52 lines · Yu Zhao
Quoted 29 lines · Greg KH
Quoted 27 lines · Yu
Quoted 25 lines · Greg KH
Quoted 13 lines · Yu
Well, to do it "correctly" you are going to have to tell the driver to
Quoted 12 lines
shut itself down, and reinitialize itself.
Turns out, that doesn't really work for disk and network devices without
dropping the connection (well, network devices should be fine probably).
So you just can't do this, sorry. That's why the BIOS handles all of these issues in a PCI hotplug system.
How does the hardware people think we are going to handle this in the OS? It's not something that any operating system can do, is it part of
the IOV PCI spec somewhere?Quoted 6 lines
No, it's not part of the PCI IOV spec.
I just want the IOV (and whole PCI subsystem) have more flexibility on various BIOSes. So can we reconsider about resource rebalance as boot option, or should we forget about this idea?
As you have proposed it, the boot option will not work at all, so I think we need to forget about it. Especially if it is not really needed.
Quoted 8 lines
Quoted 4 lines
I guess at least one thing would work if people don't want to boot twice:
give the bus number 0 as rebalance starting point, then all system resources would be reshuffled :-)Hm, but don't we do that today with our basic resource reservation logic at boot time? What would be different about this kind of proposal?
The generic PCI core can do this but this feature is kind of disabled by low level PCI code in x86. The low level code tries to reserve resource according to configuration from BIOS. If the BIOS is wrong, the allocation would fail and the generic PCI core couldn't repair it because the bridge resources may have been allocated by the PCI low level and the PCI core can't expand them to find enough resource for the subordinates.
The proposal is to disable x86 PCI low level to allocation resources according to BIOS so PCI core can fully control the resource allocation. The PCI core takes all resources from BARs it knows into account and configure the resource windows on the bridges according to its own calculation.
Quoted 9 lines
Regards,
Yu--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linux-foundation.org/mailman/listinfo/virtualization
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
Quoted 9 lines · freevanx
Dear all,
I'm glad to hear this. In fact, I'm developing for BIOS area. This feature is very useful when your system have one or more PCI/PCIe hotplug slot. Generally, BIOS reserve amount of resource for empty hotplug slot by default, but it is not always enough for all device. We have many kind of Express Modules which consume different amount of resouce, generally we reserve a small number of resouce for this, so, sometime, some Express Modules hotplug without card installed at boot time, it will not useable.
Then fix the BIOS :)
Seriously, that is what the PCI hotplug spec says to do, right?
Quoted 4 lines
Then, Microsoft say they implement PCI Multi-level Resource Rebanlence in Vista and Server 2008, you can refer
http://www.microsoft.com/whdc/archive/multilevel-rebal.mspx
http://www.microsoft.com/whdc/connect/pci/PCI-rsc.mspx
But they did not implement this for Vista, and pulled it before it shipped, right? That is what the driver development documentation for Vista said that I read.
Do you know if they are going to add it back for Windows 7? If so, then we should probably look into this, otherwise, no need to, as the BIOSes will be fixed properly.
Quoted 8 lines
They use a method of ACPI to tell OS that you can ignore the resource allocation of PCI devices below the bridge. I think this is more useful than specify the BUS number to ignore resouce allocation, because the BUS number often change due to some need by BIOS or new PCI/PCIe device added in system. Users generally do not know the system architecture and can not specify the BUS number of the root bridge, while if you specify the
_DSMmethod like MS to the root bridge of hotplug slot, it is a much easier way to archive for BIOS writers.
Yes, push the burden of getting this right onto the OS developers, instead of doing it properly in the BIOS, how fun :(
Seriously, it isn't that hard to reserve enough space on most machines in the BIOS to get this correct. It only gets messy when you have hundreds of hotplug PCI slots and bridges. Even then, the BIOS writers have been able to resolve this for a while due to this kind of hardware shipping successfully with Linux for many years now.
Quoted 3 lines
PS:
Since my mail address was blocked by the maillist, this mail may not reach people who only in linux kernel maillist.
It is being blocked because you are sending out html email.
Please reconfigure your gmail client to not do that, and your mail will go through just fine.
thanks,
greg k-h
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
In reply to freevanxQuoted 9 lines · freevanx
Dear all,
I'm glad to hear this. In fact, I'm developing for BIOS area. This feature is very useful when your system have one or more PCI/PCIe hotplug slot. Generally, BIOS reserve amount of resource for empty hotplug slot by default, but it is not always enough for all device. We have many kind of Express Modules which consume different amount of resouce, generally we reserve a small number of resouce for this, so, sometime, some Express Modules hotplug without card installed at boot time, it will not useable.
Then fix the BIOS :)
Seriously, that is what the PCI hotplug spec says to do, right?
Quoted 4 lines
Then, Microsoft say they implement PCI Multi-level Resource Rebanlence in Vista and Server 2008, you can refer
http://www.microsoft.com/whdc/archive/multilevel-rebal.mspx
http://www.microsoft.com/whdc/connect/pci/PCI-rsc.mspx
But they did not implement this for Vista, and pulled it before it shipped, right? That is what the driver development documentation for Vista said that I read.
Do you know if they are going to add it back for Windows 7? If so, then we should probably look into this, otherwise, no need to, as the BIOSes will be fixed properly.
Quoted 8 lines
They use a method of ACPI to tell OS that you can ignore the resource allocation of PCI devices below the bridge. I think this is more useful than specify the BUS number to ignore resouce allocation, because the BUS number often change due to some need by BIOS or new PCI/PCIe device added in system. Users generally do not know the system architecture and can not specify the BUS number of the root bridge, while if you specify the
_DSMmethod like MS to the root bridge of hotplug slot, it is a much easier way to archive for BIOS writers.
Yes, push the burden of getting this right onto the OS developers, instead of doing it properly in the BIOS, how fun :(
Seriously, it isn't that hard to reserve enough space on most machines in the BIOS to get this correct. It only gets messy when you have hundreds of hotplug PCI slots and bridges. Even then, the BIOS writers have been able to resolve this for a while due to this kind of hardware shipping successfully with Linux for many years now.
Quoted 3 lines
PS:
Since my mail address was blocked by the maillist, this mail may not reach people who only in linux kernel maillist.
It is being blocked because you are sending out html email.
Please reconfigure your gmail client to not do that, and your mail will go through just fine.
thanks,
greg k-h
Re: [PATCH 16/16 v6] PCI: document the new PCI boot parameters
In reply to Greg KHQuoted 54 lines · Greg KH
Quoted 27 lines · Yu Zhao
Quoted 26 lines · Greg KH
Quoted 25 lines · Yu
Quoted 13 lines · Greg KH
Quoted 12 lines · Yu
Quoted 12 lines
Well, to do it "correctly" you are going to have to tell the driver to shut itself down, and reinitialize itself.
Turns out, that doesn't really work for disk and network devices without
dropping the connection (well, network devices should be fine probably).
So you just can't do this, sorry. That's why the BIOS handles all of these issues in a PCI hotplug system.
How does the hardware people think we are going to handle this in the OS? It's not something that any operating system can do, is it part of
the IOV PCI spec somewhere?Quoted 8 lines
Quoted 5 lines
No, it's not part of the PCI IOV spec.
I just want the IOV (and whole PCI subsystem) have more flexibility on various BIOSes. So can we reconsider about resource rebalance as boot option, or should we forget about this idea?
As you have proposed it, the boot option will not work at all, so I think we need to forget about it. Especially if it is not really needed.
I guess at least one thing would work if people don't want to boot twice:
give the bus number 0 as rebalance starting point, then all system resources would be reshuffled :-)Quoted 8 lines
Quoted 2 lines
Hm, but don't we do that today with our basic resource reservation logic at boot time? What would be different about this kind of proposal?
The generic PCI core can do this but this feature is kind of disabled by low level PCI code in x86. The low level code tries to reserve resource according to configuration from BIOS. If the BIOS is wrong, the allocation would fail and the generic PCI core couldn't repair it because the bridge resources may have been allocated by the PCI low level and the PCI core can't expand them to find enough resource for the subordinates.
Yes, we do this on purpose.
Quoted 5 lines
The proposal is to disable x86 PCI low level to allocation resources according to BIOS so PCI core can fully control the resource allocation. The PCI core takes all resources from BARs it knows into account and configure the resource windows on the bridges according to its own calculation.
Ah, so you mean we should revert back to the way we use to do x86 PCI resource allocation from about a year and a half ago to about 8 years ago?
Hint, there was a reason why we switched over to using the BIOS instead of doing it ourselves. Turns out we have to trust the BIOS here, as that is exactly what other operating systems do. Trying to do it on our own was too fragile and resulted in too many problems over time.
Quoted 5 lines
Go look at the archives for when this all was switched, you'll see the reasons why.
So no, we will not be going back to the way we used to do things, we changed for a reason :)
So it's really a long story, and I'm glad to see the reason.
Actually there was no such thing in early SR-IOV patches, but months ago I heard some complaints that pushed me to do this kind of reverse. Looks like I have to let these complaints turn to BIOS people from now on :-)
Regards,
Yu