SR-IOV problem with Intel 82599EB (not enough MMIO resources for SR-IOV)

Hello,

I installed kvm and tried to use SR-IOV virtualizaton for 82599EB(Intel XT-520 T2) dual port card with latest ixgbe driver(version:3.11.33) , kernel2.6.32-279.14.1(OS:Centos6.3) ,after configuration and reboot
It seems that only first port of the card's VFs works,second port of the card's VFs didn't work
I found some errors in /var/log messages:

Nov  8 14:56:54 12 kernel: ixgbe 0000:07:00.1: not enough MMIO resources for SR-IOV
Nov  8 14:56:54 12 kernel: ixgbe 0000:07:00.1: (unregistered net_device): Failed to enable PCI sriov: -12

How to fix it,any help would be greatly appreciated.

below is the related information:
Server: R710
OS: centos6.3
NIC: X520-T2(dual port)
kernel: 2.6.32-279.14.1.el6.x86_64
BIOSVersion: 6.3.0(latest)
BIOS:Inter VT/VT-d or SR-IOV(enabled)
ixgbe:3.11.33(latest)
ixgbevf:2.7.12(latest)
grub config:intel_iommu=on appended

/var/log/messages:

Nov  8 14:56:54 12 kernel: Intel(R) 10 Gigabit PCI Express Network Driver - version 3.11.33
Nov  8 14:56:54 12 kernel: Copyright (c) 1999-2012 Intel Corporation.
Nov  8 14:56:54 12 kernel: ixgbe 0000:07:00.0: PCI INT B -> GSI 50 (level, low) -> IRQ 50
Nov  8 14:56:54 12 kernel: ixgbe: I/O Virtualization (IOV) set to 2
Nov  8 14:56:54 12 kernel: ixgbe: 0000:07:00.0: ixgbe_check_options: FCoE Offload feature enabled
Nov  8 14:56:54 12 kernel: ixgbe 0000:07:00.0: (unregistered net_device): SR-IOV enabled with 2 VFs
Nov  8 14:56:54 12 kernel: ixgbe 0000:07:00.0: FCoE offload feature is not available. Disabling FCoE offload feature
Nov  8 14:56:54 12 kernel: ixgbe 0000:07:00.0: (PCI Express:5.0GT/s:Width x8) 68:05:ca:0c:7a:e2
Nov  8 14:56:54 12 kernel: ixgbe 0000:07:00.0: eth4: MAC: 2, PHY: 2, PBA No: G21371-003
Nov  8 14:56:54 12 kernel: ixgbe 0000:07:00.0: eth4: Enabled Features: RxQ: 1 TxQ: 1 LRO
Nov  8 14:56:54 12 kernel: ixgbe 0000:07:00.0: eth4: IOV: VF 0 is enabled mac 0E:15:B9:26:B3:7D
Nov  8 14:56:54 12 kernel: ixgbe 0000:07:00.0: eth4: IOV: VF 1 is enabled mac 32:69:2D:16:B9:40
Nov  8 14:56:54 12 kernel: ixgbe 0000:07:00.0: eth4: Intel(R) 10 Gigabit Network Connection
Nov  8 14:56:54 12 kernel: ixgbe 0000:07:00.1: PCI INT A -> GSI 40 (level, low) -> IRQ 40
Nov  8 14:56:54 12 kernel: ixgbe: I/O Virtualization (IOV) set to 2
Nov  8 14:56:54 12 kernel: ixgbe: 0000:07:00.1: ixgbe_check_options: FCoE Offload feature enabled
Nov  8 14:56:54 12 kernel: ixgbe 0000:07:00.1: not enough MMIO resources for SR-IOV
Nov  8 14:56:54 12 kernel: ixgbe 0000:07:00.1: (unregistered net_device): Failed to enable PCI sriov: -12
Nov  8 14:56:54 12 kernel: ixgbe 0000:07:00.1: FCoE offload feature is not available. Disabling FCoE offload feature
Nov  8 14:56:54 12 kernel: ixgbe 0000:07:00.1: (PCI Express:5.0GT/s:Width x8) 68:05:ca:0c:7a:e3
Nov  8 14:56:54 12 kernel: ixgbe 0000:07:00.1: eth5: MAC: 2, PHY: 2, PBA No: G21371-003
Nov  8 14:56:54 12 kernel: ixgbe 0000:07:00.1: eth5: Enabled Features:
RxQ: 16 TxQ: 16 FdirHash RSC
Nov  8 14:56:54 12 kernel: ixgbe 0000:07:00.1: eth5: Intel(R) 10

Gigabit Network Connection

# dmesg |grep -E 'DMA|IOMMU'
ACPI: DMAR 00000000bf3b3668 001C0 (v01 DELL   PE_SC3   00000001 DELL 00000001)
  DMA      0x00000001 -> 0x00001000
  DMA32    0x00001000 -> 0x00100000
  DMA zone: 56 pages used for memmap
  DMA zone: 102 pages reserved
  DMA zone: 3839 pages, LIFO batch:0
  DMA32 zone: 14280 pages used for memmap
  DMA32 zone: 764849 pages, LIFO batch:31
Intel-IOMMU: enabled
DMAR: Host address width 40
DMAR: DRHD base: 0x000000fed90000 flags: 0x1
IOMMU fed90000: ver 1:0 cap c90780106f0462 ecap f020fe
DMAR: RMRR base: 0x000000bf4c8000 end: 0x000000bf4dffff
DMAR: RMRR base: 0x000000bf4b1000 end: 0x000000bf4bffff
DMAR: RMRR base: 0x000000bf4a1000 end: 0x000000bf4a1fff
DMAR: RMRR base: 0x000000bf4a3000 end: 0x000000bf4a3fff
DMAR: RMRR base: 0x000000bf4a5000 end: 0x000000bf4a5fff
DMAR: RMRR base: 0x000000bf4a7000 end: 0x000000bf4a7fff
DMAR: RMRR base: 0x000000bf4c0000 end: 0x000000bf4c0fff
DMAR: RMRR base: 0x000000bf4c2000 end: 0x000000bf4c2fff
DMAR: ATSR flags: 0x0
DMAR: Device scope device [0000:00:1a.02] not found
DMAR: Device scope device [0000:00:1a.02] not found
DMAR: Device scope device [0000:00:1d.02] not found
DMAR: Device scope device [0000:00:1d.02] not found
IOMMU 0xfed90000: using Queued invalidation
IOMMU: Setting RMRR:
IOMMU: Setting identity map for device 0000:00:1d.7 [0xbf4c2000 - 0xbf4c3000]
IOMMU: Setting identity map for device 0000:00:1a.7 [0xbf4c0000 - 0xbf4c1000]
IOMMU: Setting identity map for device 0000:00:1d.1 [0xbf4a7000 - 0xbf4a8000]
IOMMU: Setting identity map for device 0000:00:1d.0 [0xbf4a5000 - 0xbf4a6000]
IOMMU: Setting identity map for device 0000:00:1a.1 [0xbf4a3000 - 0xbf4a4000]
IOMMU: Setting identity map for device 0000:00:1a.0 [0xbf4a1000 - 0xbf4a2000]
IOMMU: Setting identity map for device 0000:00:1a.0 [0xbf4b1000 - 0xbf4c0000]
IOMMU: Setting identity map for device 0000:00:1a.1 [0xbf4b1000 - 0xbf4c0000]
IOMMU: Setting identity map for device 0000:00:1d.0 [0xbf4b1000 - 0xbf4c0000]
IOMMU: Setting identity map for device 0000:00:1d.1 [0xbf4b1000 - 0xbf4c0000]
IOMMU: Setting identity map for device 0000:00:1a.7 [0xbf4c8000 - 0xbf4e0000]
IOMMU: Setting identity map for device 0000:00:1d.7 [0xbf4c8000 - 0xbf4e0000]
IOMMU: Prepare 0-16MiB unity mapping for LPC
IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0x1000000]
PCI-DMA: Intel(R) Virtualization Technology for Directed I/O
ata1: SATA max UDMA/133 cmd 0xdc10 ctl 0xdc08 bmdma 0xdc20 irq 23
ata2: SATA max UDMA/133 cmd 0xdc18 ctl 0xdc0c bmdma 0xdc28 irq 23
ata1.00: ATAPI: TSSTcorp DVD-ROM SN-108BB, D150, max UDMA/100
ata1.00: configured for UDMA/100

# lspci|grep Virtual
07:10.0 Ethernet controller: Intel Corporation 82599 Virtual Function (rev 01) 07:10.2 Ethernet controller: Intel Corporation 82599 Virtual Function (rev 01)

#lspci|grep '82599EB'
07:00.0 Ethernet controller: Intel Corporation 82599EB 10 Gigabit TN Network Connection (rev 01)
07:00.1 Ethernet controller: Intel Corporation 82599EB 10 Gigabit TN Network Connection (rev 01)

# lspci -n|grep 07:00
07:00.0 0200: 8086:151c (rev 01)
07:00.1 0200: 8086:151c (rev 01)

# lspci -n -d 8086:151c -vvv|grep -i width
LnkCap: Port #2, Speed 5GT/s, Width x8, ASPM L0s, Latency L0 <1us, L1 <8us
LnkSta: Speed 5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
LnkCap: Port #2, Speed 5GT/s, Width x8, ASPM L0s, Latency L0 <1us, L1 <8us
LnkSta: Speed 5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-

#lspci -vvv -s 07:00
07:00.0 Ethernet controller: Intel Corporation 82599EB 10 Gigabit TN Network Connection (rev 01)
Subsystem: Intel Corporation Ethernet Server Adapter X520-T2
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin B routed to IRQ 50
Region 0: Memory at de400000 (64-bit, non-prefetchable) [size=2M]
Region 2: I/O ports at ecc0 [size=32]
Region 4: Memory at de3f8000 (64-bit, non-prefetchable) [size=16K]
Expansion ROM at de200000 [disabled] [size=1M]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
Address: 0000000000000000 Data: 0000
Masking: 00000000 Pending: 00000000
Capabilities: [70] MSI-X: Enable+ Count=64 Masked-
Vector table: BAR=4 offset=00000000
PBA: BAR=4 offset=00002000
Capabilities: [a0] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported+
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 256 bytes, MaxReadReq 512 bytes
DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+ TransPend-
LnkCap: Port #2, Speed 5GT/s, Width x8, ASPM L0s, Latency L0 <1us, L1 <8us
ClockPM- Surprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+
DevCtl2: Completion Timeout: 65ms to 210ms, TimeoutDis-
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB
Capabilities: [e0] Vital Product Data
Unknown small resource type 06, will not decode more.
Capabilities: [100] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt+ UnxCmplt+ RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES- TLP+ FCP+ CmpltTO+ CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC+ UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
CEMsk: RxErr+ BadTLP+ BadDLLP+ Rollover+ Timeout+ NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [140] Device Serial Number 68-05-ca-ff-ff-0c-7a-e2
Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 1
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
IOVCap: Migration-, Interrupt Message Number: 000
IOVCtl: Enable+ Migration- Interrupt- MSE+ ARIHierarchy+
IOVSta: Migration-
Initial VFs: 64, Total VFs: 64, Number of VFs: 2, Function Dependency Link: 00
VF offset: 128, stride: 2, Device ID: 10ed
Supported Page Size: 00000553, System Page Size: 00000001
Region 0: Memory at 00000000d5900000 (64-bit, prefetchable)
Region 3: Memory at 00000000d5a00000 (64-bit, prefetchable)
VF Migration: offset: 00000000, BIR: 0
Kernel driver in use: ixgbe
Kernel modules: ixgbe
07:00.1 Ethernet controller: Intel Corporation 82599EB 10 Gigabit TN Network Connection (rev 01)
Subsystem: Intel Corporation Ethernet Server Adapter X520-T2
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 40
Region 0: Memory at de600000 (64-bit, non-prefetchable) [size=2M]
Region 2: I/O ports at ece0 [size=32]
Region 4: Memory at de3fc000 (64-bit, non-prefetchable) [size=16K]
Expansion ROM at d5800000 [disabled] [size=1M]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
Address: 0000000000000000 Data: 0000
Masking: 00000000 Pending: 00000000
Capabilities: [70] MSI-X: Enable+ Count=64 Masked-
Vector table: BAR=4 offset=00000000
PBA: BAR=4 offset=00002000
Capabilities: [a0] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported+
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 256 bytes, MaxReadReq 512 bytes
DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+ TransPend-
LnkCap: Port #2, Speed 5GT/s, Width x8, ASPM L0s, Latency L0 <1us, L1 <8us
ClockPM- Surprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+
DevCtl2: Completion Timeout: 65ms to 210ms, TimeoutDis-
LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB
Capabilities: [e0] Vital Product Data
Unknown small resource type 06, will not decode more.
Capabilities: [100] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt+ UnxCmplt+ RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES- TLP+ FCP+ CmpltTO+ CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC+ UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
CEMsk: RxErr+ BadTLP+ BadDLLP+ Rollover+ Timeout+ NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [140] Device Serial Number 68-05-ca-ff-ff-0c-7a-e2
Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 0
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
IOVCap: Migration-, Interrupt Message Number: 000
IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy-
IOVSta: Migration-
Initial VFs: 64, Total VFs: 64, Number of VFs: 2, Function Dependency Link: 01
VF offset: 128, stride: 2, Device ID: 10ed
Supported Page Size: 00000553, System Page Size: 00000001
Region 0: Memory at 00000000d5b00000 (64-bit, prefetchable)
Region 3: Memory at 00000000d5b00000 (64-bit, prefetchable)
VF Migration: offset: 00000000, BIR: 0
Kernel driver in use: ixgbe
Kernel modules: ixgbe

#modinfo ixgbe

filename:       /lib/modules/2.6.32-279.14.1.el6.x86_64/kernel/drivers/net/ixgbe/ixgbe.ko
version:        3.11.33
license:        GPL
description:    Intel(R) 10 Gigabit PCI Express Network Driver
author:         Intel Corporation, <[email protected]>
srcversion:     ADDCBFA6FAFE8A0299074F8
alias:          pci:v00008086d0000154Asv*sd*bc*sc*i*
alias:          pci:v00008086d00001557sv*sd*bc*sc*i*
alias:          pci:v00008086d0000154Fsv*sd*bc*sc*i*
alias:          pci:v00008086d0000154Dsv*sd*bc*sc*i*
alias:          pci:v00008086d00001528sv*sd*bc*sc*i*
alias:          pci:v00008086d000010F8sv*sd*bc*sc*i*
alias:          pci:v00008086d0000151Csv*sd*bc*sc*i*
alias:          pci:v00008086d00001529sv*sd*bc*sc*i*
alias:          pci:v00008086d0000152Asv*sd*bc*sc*i*
alias:          pci:v00008086d000010F9sv*sd*bc*sc*i*
alias:          pci:v00008086d00001514sv*sd*bc*sc*i*
alias:          pci:v00008086d00001507sv*sd*bc*sc*i*
alias:          pci:v00008086d000010FBsv*sd*bc*sc*i*
alias:          pci:v00008086d00001517sv*sd*bc*sc*i*
alias:          pci:v00008086d000010FCsv*sd*bc*sc*i*
alias:          pci:v00008086d000010F7sv*sd*bc*sc*i*
alias:          pci:v00008086d00001508sv*sd*bc*sc*i*
alias:          pci:v00008086d000010DBsv*sd*bc*sc*i*
alias:          pci:v00008086d000010F4sv*sd*bc*sc*i*
alias:          pci:v00008086d000010E1sv*sd*bc*sc*i*
alias:          pci:v00008086d000010F1sv*sd*bc*sc*i*
alias:          pci:v00008086d000010ECsv*sd*bc*sc*i*
alias:          pci:v00008086d000010DDsv*sd*bc*sc*i*
alias:          pci:v00008086d0000150Bsv*sd*bc*sc*i*
alias:          pci:v00008086d000010C8sv*sd*bc*sc*i*
alias:          pci:v00008086d000010C7sv*sd*bc*sc*i*
alias:          pci:v00008086d000010C6sv*sd*bc*sc*i*
alias:          pci:v00008086d000010B6sv*sd*bc*sc*i*
depends:        dca
vermagic:       2.6.32-279.14.1.el6.x86_64 SMP mod_unload modversions 
parm:           InterruptType:Change Interrupt Mode (0=Legacy, 1=MSI, 2=MSI-X), default IntMode (deprecated) (array of int)
parm:           IntMode:Change Interrupt Mode (0=Legacy, 1=MSI, 2=MSI-X), default 2 (array of int)
parm:           MQ:Disable or enable Multiple Queues, default 1 (array of int)
parm:           DCA:Disable or enable Direct Cache Access, 0=disabled, 1=descriptor only, 2=descriptor and data (array of int)
parm:           RSS:Number of Receive-Side Scaling Descriptor Queues, default 0=number of cpus (array of int)
parm:           VMDQ:Number of Virtual Machine Device Queues: 0/1 = disable, 2-16 enable (default=8) (array of int)
parm:           max_vfs:Number of Virtual Functions: 0 = disable (default), 1-63 = enable this many VFs (array of int)
parm:           L2LBen:L2 Loopback Enable: 0 = disable, 1 = enable (default) (array of int)
parm:           InterruptThrottleRate:Maximum interrupts per second, per vector, (0,1,956-488281), default 1 (array of int)
parm:           LLIPort:Low Latency Interrupt TCP Port (0-65535) (array of int)
parm:           LLIPush:Low Latency Interrupt on TCP Push flag (0,1) (array of int)
parm:           LLISize:Low Latency Interrupt on Packet Size (0-1500) (array of int)
parm:           LLIEType:Low Latency Interrupt Ethernet Protocol Type (array of int)
parm:           LLIVLANP:Low Latency Interrupt on VLAN priority threshold (array of int)
parm:           FdirPballoc:Flow Director packet buffer allocation level:
1 = 8k hash filters or 2k perfect filters
2 = 16k hash filters or 4k perfect filters
3 = 32k hash filters or 8k perfect filters (array of int)
parm:           AtrSampleRate:Software ATR Tx packet sample rate (array of int)
parm:           FCoE:Disable or enable FCoE Offload, default 1 (array of int)
parm:           LRO:Large Receive Offload (0,1), default 1 = on (array of int)
parm:           allow_unsupported_sfp:Allow unsupported and untested SFP+ modules on 82599 based adapters, default 0 = Disable (array of int)

# modinfo ixgbevf
filename:       /lib/modules/2.6.32-279.14.1.el6.x86_64/kernel/drivers/net/ixgbevf/ixgbevf.ko
version:        2.7.12
license:        GPL
description:    Intel(R) 10 Gigabit PCI Express Virtual Function Network Driver
author:         Intel Corporation, <[email protected]>
srcversion:     4D00B4A5B811D6D07D8BD15
alias:          pci:v00008086d00001515sv*sd*bc*sc*i*
alias:          pci:v00008086d000010EDsv*sd*bc*sc*i*
depends:        
vermagic:       2.6.32-279.14.1.el6.x86_64 SMP mod_unload modversions 
parm:           InterruptThrottleRate:Maximum interrupts per second, per vector, (956-488281), default 8000, use 1 for dynamic (array of int)

# dmidecode -t bios
# dmidecode 2.11
SMBIOS 2.6 present.
Handle 0x0000, DMI type 0, 24 bytes
BIOS Information
        Vendor: Dell Inc.
        Version: 6.3.0
        Release Date: 07/24/2012
        Address: 0xF0000
        Runtime Size: 64 kB
        ROM Size: 4096 kB
        Characteristics:
                ISA is supported
                PCI is supported
                PNP is supported
                BIOS is upgradeable
                BIOS shadowing is allowed
                Boot from CD is supported
                Selectable boot is supported
                EDD is supported
                Japanese floppy for Toshiba 1.2 MB is supported (int 13h)
                5.25"/360 kB floppy services are supported (int 13h)
                5.25"/1.2 MB floppy services are supported (int 13h)
                3.5"/720 kB floppy services are supported (int 13h)
                8042 keyboard services are supported (int 9h)
                Serial services are supported (int 14h)
                CGA/mono video services are supported (int 10h)
                ACPI is supported
                USB legacy is supported
                BIOS boot specification is supported
                Function key-initiated network boot is supported
                Targeted content distribution is supported
        BIOS Revision: 6.3
Handle 0x0D00, DMI type 13, 22 bytes
BIOS Language Information
        Language Description Format: Long
        Installable Languages: 1
                en|US|iso8859-1
        Currently Installed Language: en|US|iso8859-1

Jason Gao
--

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

-----Original Message-----
From: Jeff Kirsher [mailto:[email protected]]
Sent: Thursday, November 08, 2012 3:23 AM
To: pkill.2012
Cc: linux-kernel; netdev; kvm; [email protected]; Rose, Gregory V
Subject: Re: SR-IOV problem with Intel 82599EB (not enough MMIO resources for SR-IOV)

Quoted 14 lines · On 11/08/2012 03:15 AM, pkill.2012 wrote:

Hello,

I installed kvm and tried to use SR-IOV virtualizaton for 82599EB(Intel XT-520 T2) dual port card with latest ixgbe driver(version:3.11.33) , kernel2.6.32-279.14.1(OS:Centos6.3) ,after configuration and reboot It seems that only first port of the card's VFs works,second port of the card's VFs didn't work I found some errors in /var/log messages:

Nov  8 14:56:54 12 kernel: ixgbe 0000:07:00.1: not enough MMIO

resources for SR-IOV Nov 8 14:56:54 12 kernel: ixgbe 0000:07:00.1:
(unregistered net_device): Failed to enable PCI sriov: -12

How to fix it,any help would be greatly appreciated.

Quoted 12 lines

below is the related information:
Server: R710
OS: centos6.3
NIC: X520-T2(dual port)
kernel: 2.6.32-279.14.1.el6.x86_64
BIOSVersion: 6.3.0(latest)
BIOS:Inter VT/VT-d or SR-IOV(enabled)
ixgbe:3.11.33(latest)
ixgbevf:2.7.12(latest)
grub config:intel_iommu=on appended

Quoted 3 lines

/var/log/messages:

Nov  8 14:56:54 12 kernel: Intel(R) 10 Gigabit PCI Express Network

Driver - version 3.11.33 Nov 8 14:56:54 12 kernel: Copyright (c) 1999-

2012 Intel Corporation.

Quoted 19 lines

Nov  8 14:56:54 12 kernel: ixgbe 0000:07:00.0: PCI INT B -> GSI 50

(level, low) -> IRQ 50 Nov 8 14:56:54 12 kernel: ixgbe: I/O Virtualization (IOV) set to 2 Nov 8 14:56:54 12 kernel: ixgbe:
0000:07:00.0: ixgbe_check_options: FCoE Offload feature enabled Nov 8 14:56:54 12 kernel: ixgbe 0000:07:00.0: (unregistered net_device):
SR-IOV enabled with 2 VFs Nov 8 14:56:54 12 kernel: ixgbe 0000:07:00.0: FCoE offload feature is not available. Disabling FCoE offload feature Nov 8 14:56:54 12 kernel: ixgbe 0000:07:00.0: (PCI Express:5.0GT/s:Width x8) 68:05:ca:0c:7a:e2 Nov 8 14:56:54 12 kernel:
ixgbe 0000:07:00.0: eth4: MAC: 2, PHY: 2, PBA No: G21371-003 Nov 8 14:56:54 12 kernel: ixgbe 0000:07:00.0: eth4: Enabled Features: RxQ: 1
TxQ: 1 LRO Nov 8 14:56:54 12 kernel: ixgbe 0000:07:00.0: eth4: IOV:
VF 0 is enabled mac 0E:15:B9:26:B3:7D Nov 8 14:56:54 12 kernel: ixgbe 0000:07:00.0: eth4: IOV: VF 1 is enabled mac 32:69:2D:16:B9:40 Nov 8 14:56:54 12 kernel: ixgbe 0000:07:00.0: eth4: Intel(R) 10 Gigabit Network Connection Nov 8 14:56:54 12 kernel: ixgbe 0000:07:00.1: PCI INT A -> GSI 40 (level, low) -> IRQ 40 Nov 8 14:56:54 12 kernel:
ixgbe: I/O Virtualization (IOV) set to 2 Nov 8 14:56:54 12 kernel:
ixgbe: 0000:07:00.1: ixgbe_check_options: FCoE Offload feature enabled

Quoted 1 line

Nov  8 14:56:54 12 kernel: ixgbe 0000:07:00.1: not enough MMIO resources

for SR-IOV Nov 8 14:56:54 12 kernel: ixgbe 0000:07:00.1: (unregistered net_device): Failed to enable PCI sriov: -12 Nov 8 14:56:54 12 kernel:
ixgbe 0000:07:00.1: FCoE offload feature is not available. Disabling FCoE offload feature Nov 8 14:56:54 12 kernel: ixgbe 0000:07:00.1: (PCI Express:5.0GT/s:Width x8) 68:05:ca:0c:7a:e3 Nov 8 14:56:54 12 kernel:
ixgbe 0000:07:00.1: eth5: MAC: 2, PHY: 2, PBA No: G21371-003 Nov 8 14:56:54 12 kernel: ixgbe 0000:07:00.1: eth5: Enabled Features:

Quoted 6 lines

RxQ: 16 TxQ: 16 FdirHash RSC

Nov  8 14:56:54 12 kernel: ixgbe 0000:07:00.1: eth5: Intel(R) 10

Gigabit Network Connection

# dmesg |grep -E 'DMA|IOMMU'
ACPI: DMAR 00000000bf3b3668 001C0 (v01 DELL   PE_SC3   00000001 DELL

00000001)

Quoted 226 lines

DMA 0x00000001 -> 0x00001000
DMA32 0x00001000 -> 0x00100000
DMA zone: 56 pages used for memmap
DMA zone: 102 pages reserved
DMA zone: 3839 pages, LIFO batch:0
DMA32 zone: 14280 pages used for memmap
DMA32 zone: 764849 pages, LIFO batch:31
Intel-IOMMU: enabled
DMAR: Host address width 40
DMAR: DRHD base: 0x000000fed90000 flags: 0x1 IOMMU fed90000: ver 1:0 cap c90780106f0462 ecap f020fe
DMAR: RMRR base: 0x000000bf4c8000 end: 0x000000bf4dffff
DMAR: RMRR base: 0x000000bf4b1000 end: 0x000000bf4bffff
DMAR: RMRR base: 0x000000bf4a1000 end: 0x000000bf4a1fff
DMAR: RMRR base: 0x000000bf4a3000 end: 0x000000bf4a3fff
DMAR: RMRR base: 0x000000bf4a5000 end: 0x000000bf4a5fff
DMAR: RMRR base: 0x000000bf4a7000 end: 0x000000bf4a7fff
DMAR: RMRR base: 0x000000bf4c0000 end: 0x000000bf4c0fff
DMAR: RMRR base: 0x000000bf4c2000 end: 0x000000bf4c2fff
DMAR: ATSR flags: 0x0
DMAR: Device scope device [0000:00:1a.02] not found
DMAR: Device scope device [0000:00:1a.02] not found
DMAR: Device scope device [0000:00:1d.02] not found
DMAR: Device scope device [0000:00:1d.02] not found IOMMU 0xfed90000:
using Queued invalidation
IOMMU: Setting RMRR:
IOMMU: Setting identity map for device 0000:00:1d.7 [0xbf4c2000 - 0xbf4c3000]
IOMMU: Setting identity map for device 0000:00:1a.7 [0xbf4c0000 - 0xbf4c1000]
IOMMU: Setting identity map for device 0000:00:1d.1 [0xbf4a7000 - 0xbf4a8000]
IOMMU: Setting identity map for device 0000:00:1d.0 [0xbf4a5000 - 0xbf4a6000]
IOMMU: Setting identity map for device 0000:00:1a.1 [0xbf4a3000 - 0xbf4a4000]
IOMMU: Setting identity map for device 0000:00:1a.0 [0xbf4a1000 - 0xbf4a2000]
IOMMU: Setting identity map for device 0000:00:1a.0 [0xbf4b1000 - 0xbf4c0000]
IOMMU: Setting identity map for device 0000:00:1a.1 [0xbf4b1000 - 0xbf4c0000]
IOMMU: Setting identity map for device 0000:00:1d.0 [0xbf4b1000 - 0xbf4c0000]
IOMMU: Setting identity map for device 0000:00:1d.1 [0xbf4b1000 - 0xbf4c0000]
IOMMU: Setting identity map for device 0000:00:1a.7 [0xbf4c8000 - 0xbf4e0000]
IOMMU: Setting identity map for device 0000:00:1d.7 [0xbf4c8000 - 0xbf4e0000]
IOMMU: Prepare 0-16MiB unity mapping for LPC
IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0x1000000]
PCI-DMA: Intel(R) Virtualization Technology for Directed I/O
ata1: SATA max UDMA/133 cmd 0xdc10 ctl 0xdc08 bmdma 0xdc20 irq 23
ata2: SATA max UDMA/133 cmd 0xdc18 ctl 0xdc0c bmdma 0xdc28 irq 23 ata1.00: ATAPI: TSSTcorp DVD-ROM SN-108BB, D150, max UDMA/100 ata1.00: configured for UDMA/100

# lspci|grep Virtual
07:10.0 Ethernet controller: Intel Corporation 82599 Virtual Function (rev 01)
07:10.2 Ethernet controller: Intel Corporation 82599 Virtual Function (rev 01)

#lspci|grep '82599EB'
07:00.0 Ethernet controller: Intel Corporation 82599EB 10 Gigabit TN Network Connection (rev 01)
07:00.1 Ethernet controller: Intel Corporation 82599EB 10 Gigabit TN Network Connection (rev 01)

# lspci -n|grep 07:00
07:00.0 0200: 8086:151c (rev 01)
07:00.1 0200: 8086:151c (rev 01)

# lspci -n -d 8086:151c -vvv|grep -i width
LnkCap: Port #2, Speed 5GT/s, Width x8, ASPM L0s, Latency L0 <1us, L1
<8us
LnkSta: Speed 5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive-
BWMgmt- ABWMgmt-
LnkCap: Port #2, Speed 5GT/s, Width x8, ASPM L0s, Latency L0 <1us, L1
<8us
LnkSta: Speed 5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive-
BWMgmt- ABWMgmt-

#lspci -vvv -s 07:00
07:00.0 Ethernet controller: Intel Corporation 82599EB 10 Gigabit TN Network Connection (rev 01)
Subsystem: Intel Corporation Ethernet Server Adapter X520-T2
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin B routed to IRQ 50
Region 0: Memory at de400000 (64-bit, non-prefetchable) [size=2M] Region 2: I/O ports at ecc0 [size=32] Region 4: Memory at de3f8000 (64-bit, non-prefetchable) [size=16K] Expansion ROM at de200000 [disabled] [size=1M]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
Address: 0000000000000000 Data: 0000
Masking: 00000000 Pending: 00000000
Capabilities: [70] MSI-X: Enable+ Count=64 Masked- Vector table: BAR=4 offset=00000000
PBA: BAR=4 offset=00002000
Capabilities: [a0] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported+ RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset- MaxPayload 256 bytes, MaxReadReq 512 bytes
DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+ TransPend-
LnkCap: Port #2, Speed 5GT/s, Width x8, ASPM L0s, Latency L0 <1us, L1 <8us
ClockPM- Surprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+
DevCtl2: Completion Timeout: 65ms to 210ms, TimeoutDis-
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB
Capabilities: [e0] Vital Product Data
Unknown small resource type 06, will not decode more.
Capabilities: [100] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt+ UnxCmplt+ RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES- TLP+ FCP+ CmpltTO+ CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC+ UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
CEMsk: RxErr+ BadTLP+ BadDLLP+ Rollover+ Timeout+ NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [140] Device Serial Number 68-05-ca-ff-ff-0c-7a-e2
Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 1
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
IOVCap: Migration-, Interrupt Message Number: 000
IOVCtl: Enable+ Migration- Interrupt- MSE+ ARIHierarchy+
IOVSta: Migration-
Initial VFs: 64, Total VFs: 64, Number of VFs: 2, Function Dependency
Link: 00 VF offset: 128, stride: 2, Device ID: 10ed Supported Page
Size: 00000553, System Page Size: 00000001 Region 0: Memory at 00000000d5900000 (64-bit, prefetchable) Region 3: Memory at 00000000d5a00000 (64-bit, prefetchable) VF Migration: offset:
00000000, BIR: 0 Kernel driver in use: ixgbe Kernel modules: ixgbe 07:00.1 Ethernet controller: Intel Corporation 82599EB 10 Gigabit TN Network Connection (rev 01)
Subsystem: Intel Corporation Ethernet Server Adapter X520-T2
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 40
Region 0: Memory at de600000 (64-bit, non-prefetchable) [size=2M] Region 2: I/O ports at ece0 [size=32] Region 4: Memory at de3fc000 (64-bit, non-prefetchable) [size=16K] Expansion ROM at d5800000 [disabled] [size=1M]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
Address: 0000000000000000 Data: 0000
Masking: 00000000 Pending: 00000000
Capabilities: [70] MSI-X: Enable+ Count=64 Masked- Vector table: BAR=4 offset=00000000
PBA: BAR=4 offset=00002000
Capabilities: [a0] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported+ RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset- MaxPayload 256 bytes, MaxReadReq 512 bytes
DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+ TransPend-
LnkCap: Port #2, Speed 5GT/s, Width x8, ASPM L0s, Latency L0 <1us, L1 <8us
ClockPM- Surprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+
DevCtl2: Completion Timeout: 65ms to 210ms, TimeoutDis-
LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB
Capabilities: [e0] Vital Product Data
Unknown small resource type 06, will not decode more.
Capabilities: [100] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt+ UnxCmplt+ RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES- TLP+ FCP+ CmpltTO+ CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC+ UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
CEMsk: RxErr+ BadTLP+ BadDLLP+ Rollover+ Timeout+ NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [140] Device Serial Number 68-05-ca-ff-ff-0c-7a-e2
Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 0
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
IOVCap: Migration-, Interrupt Message Number: 000
IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy-
IOVSta: Migration-
Initial VFs: 64, Total VFs: 64, Number of VFs: 2, Function Dependency
Link: 01 VF offset: 128, stride: 2, Device ID: 10ed Supported Page
Size: 00000553, System Page Size: 00000001 Region 0: Memory at 00000000d5b00000 (64-bit, prefetchable) Region 3: Memory at 00000000d5b00000 (64-bit, prefetchable) VF Migration: offset:
00000000, BIR: 0 Kernel driver in use: ixgbe Kernel modules: ixgbe

Quoted 2 lines

#modinfo ixgbe
filename: /lib/modules/2.6.32-

279.14.1.el6.x86_64/kernel/drivers/net/ixgbe/ixgbe.ko

Quoted 28 lines

version:        3.11.33
license:        GPL
description:    Intel(R) 10 Gigabit PCI Express Network Driver
author:         Intel Corporation, <[email protected]>
srcversion:     ADDCBFA6FAFE8A0299074F8
alias:          pci:v00008086d0000154Asv*sd*bc*sc*i*
alias:          pci:v00008086d00001557sv*sd*bc*sc*i*
alias:          pci:v00008086d0000154Fsv*sd*bc*sc*i*
alias:          pci:v00008086d0000154Dsv*sd*bc*sc*i*
alias:          pci:v00008086d00001528sv*sd*bc*sc*i*
alias:          pci:v00008086d000010F8sv*sd*bc*sc*i*
alias:          pci:v00008086d0000151Csv*sd*bc*sc*i*
alias:          pci:v00008086d00001529sv*sd*bc*sc*i*
alias:          pci:v00008086d0000152Asv*sd*bc*sc*i*
alias:          pci:v00008086d000010F9sv*sd*bc*sc*i*
alias:          pci:v00008086d00001514sv*sd*bc*sc*i*
alias:          pci:v00008086d00001507sv*sd*bc*sc*i*
alias:          pci:v00008086d000010FBsv*sd*bc*sc*i*
alias:          pci:v00008086d00001517sv*sd*bc*sc*i*
alias:          pci:v00008086d000010FCsv*sd*bc*sc*i*
alias:          pci:v00008086d000010F7sv*sd*bc*sc*i*
alias:          pci:v00008086d00001508sv*sd*bc*sc*i*
alias:          pci:v00008086d000010DBsv*sd*bc*sc*i*
alias:          pci:v00008086d000010F4sv*sd*bc*sc*i*
alias:          pci:v00008086d000010E1sv*sd*bc*sc*i*
alias:          pci:v00008086d000010F1sv*sd*bc*sc*i*
alias:          pci:v00008086d000010ECsv*sd*bc*sc*i*
alias:          pci:v00008086d000010DDsv*sd*bc*sc*i*

Quoted 8 lines

alias:          pci:v00008086d0000150Bsv*sd*bc*sc*i*
alias:          pci:v00008086d000010C8sv*sd*bc*sc*i*
alias:          pci:v00008086d000010C7sv*sd*bc*sc*i*
alias:          pci:v00008086d000010C6sv*sd*bc*sc*i*
alias:          pci:v00008086d000010B6sv*sd*bc*sc*i*
depends:        dca
vermagic:       2.6.32-279.14.1.el6.x86_64 SMP mod_unload modversions
parm:           InterruptType:Change Interrupt Mode (0=Legacy, 1=MSI,

2=MSI-X), default IntMode (deprecated) (array of int)

Quoted 1 line

parm: IntMode:Change Interrupt Mode (0=Legacy, 1=MSI, 2=MSI-

X), default 2 (array of int)

Quoted 1 line

parm: MQ:Disable or enable Multiple Queues, default 1 (array

of int)

Quoted 1 line

parm: DCA:Disable or enable Direct Cache Access, 0=disabled,

1=descriptor only, 2=descriptor and data (array of int)

Quoted 1 line

parm: RSS:Number of Receive-Side Scaling Descriptor Queues,

default 0=number of cpus (array of int)

Quoted 1 line

parm: VMDQ:Number of Virtual Machine Device Queues: 0/1 =

disable, 2-16 enable (default=8) (array of int)

Quoted 1 line

parm: max_vfs:Number of Virtual Functions: 0 = disable

(default), 1-63 = enable this many VFs (array of int)

Quoted 1 line

parm: L2LBen:L2 Loopback Enable: 0 = disable, 1 = enable

(default) (array of int)

Quoted 1 line

parm: InterruptThrottleRate:Maximum interrupts per second, per

vector, (0,1,956-488281), default 1 (array of int)

Quoted 1 line

parm: LLIPort:Low Latency Interrupt TCP Port (0-65535) (array

of int)

Quoted 1 line

parm: LLIPush:Low Latency Interrupt on TCP Push flag (0,1)

(array of int)

Quoted 1 line

parm: LLISize:Low Latency Interrupt on Packet Size (0-1500)

(array of int)

Quoted 1 line

parm: LLIEType:Low Latency Interrupt Ethernet Protocol Type

(array of int)

Quoted 1 line

parm: LLIVLANP:Low Latency Interrupt on VLAN priority

threshold (array of int)

Quoted 1 line

parm: FdirPballoc:Flow Director packet buffer allocation

level:

Quoted 4 lines

1 = 8k hash filters or 2k perfect filters
2 = 16k hash filters or 4k perfect filters
3 = 32k hash filters or 8k perfect filters (array of int)
parm: AtrSampleRate:Software ATR Tx packet sample rate (array

of int)

Quoted 1 line

parm: FCoE:Disable or enable FCoE Offload, default 1 (array of

int)

Quoted 1 line

parm: LRO:Large Receive Offload (0,1), default 1 = on (array

of int)

Quoted 1 line

parm: allow_unsupported_sfp:Allow unsupported and untested

SFP+ modules on 82599 based adapters, default 0 = Disable (array of int)

Quoted 3 lines

# modinfo ixgbevf
filename:       /lib/modules/2.6.32-

279.14.1.el6.x86_64/kernel/drivers/net/ixgbevf/ixgbevf.ko

Quoted 3 lines

version:        2.7.12
license:        GPL
description:    Intel(R) 10 Gigabit PCI Express Virtual Function Network

Driver

Quoted 7 lines

author:         Intel Corporation, <[email protected]>
srcversion:     4D00B4A5B811D6D07D8BD15
alias:          pci:v00008086d00001515sv*sd*bc*sc*i*
alias:          pci:v00008086d000010EDsv*sd*bc*sc*i*
depends:
vermagic:       2.6.32-279.14.1.el6.x86_64 SMP mod_unload modversions
parm:           InterruptThrottleRate:Maximum interrupts per second, per

vector, (956-488281), default 8000, use 1 for dynamic (array of int)

Quoted 15 lines

# dmidecode -t bios
# dmidecode 2.11
SMBIOS 2.6 present.
Handle 0x0000, DMI type 0, 24 bytes
BIOS Information
        Vendor: Dell Inc.
        Version: 6.3.0
        Release Date: 07/24/2012
        Address: 0xF0000
        Runtime Size: 64 kB
        ROM Size: 4096 kB
        Characteristics:
                ISA is supported

Quoted 8 lines

                PCI is supported
                PNP is supported
                BIOS is upgradeable
                BIOS shadowing is allowed
                Boot from CD is supported
                Selectable boot is supported
                EDD is supported
                Japanese floppy for Toshiba 1.2 MB is supported (int

13h)

Quoted 22 lines

5.25"/360 kB floppy services are supported (int 13h) 5.25"/1.2 MB floppy services are supported (int 13h) 3.5"/720 kB floppy services are supported (int 13h) 8042 keyboard services are supported (int 9h) Serial services are supported (int 14h) CGA/mono video services are supported (int 10h) ACPI is supported
USB legacy is supported
BIOS boot specification is supported Function key-initiated network boot is supported Targeted content distribution is supported
BIOS Revision: 6.3
Handle 0x0D00, DMI type 13, 22 bytes
BIOS Language Information
Language Description Format: Long
Installable Languages: 1
en|US|iso8859-1
Currently Installed Language: en|US|iso8859-1

Jason Gao
--

Quoted 5 lines

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Adding e100-devel mailing list

The BIOS in your machine doesn't support SR-IOV. You'll need to ask the manufacturer for a BIOS upgrade, if in fact one is available. Sometimes they're not.

Quoted 1 line

The BIOS in your machine doesn't support SR-IOV. You'll need to ask the manufacturer for a BIOS upgrade, if in fact one is available. Sometimes they're not.

very thanks Greg,my server Dell R710 with latest BIOS version and option for SR-IOV(SR-IOV Global Enable->Enabled) opened,I'm confused that Does R710 provide full support for SR-IOV, kernel or ixgbe driver's bug? but I'm not sure where the problem lies,anyone has any
experience about this? .

[+ linux-pci, Yinghai]

Quoted 7 lines · Jason Gao

Quoted 1 line

The BIOS in your machine doesn't support SR-IOV. You'll need to ask the manufacturer for a BIOS upgrade, if in fact one is available. Sometimes they're not.

very thanks Greg,my server Dell R710 with latest BIOS version and option for SR-IOV(SR-IOV Global Enable->Enabled) opened,I'm confused that Does R710 provide full support for SR-IOV, kernel or ixgbe driver's bug? but I'm not sure where the problem lies,anyone has any
experience about this? .

Linux normally uses the resource assignments done by the BIOS, but it is possible for the kernel to reassign those. We don't have good automatic support for that yet, but on a recent upstream kernel, you can try "pci=realloc". I doubt this option is in CentOS 6.3, though.

If an upstream kernel with "pci=realloc" still doesn't work, please post the entire dmesg log.

From afb9975bddd92b17c2f9859676ce521e2e014997 Mon Sep 17 00:00:00 2001

[+ linux-pci, Yinghai]

Quoted 7 lines · Jason Gao

Quoted 1 line

The BIOS in your machine doesn't support SR-IOV. You'll need to ask the manufacturer for a BIOS upgrade, if in fact one is available. Sometimes they're not.

very thanks Greg,my server Dell R710 with latest BIOS version and option for SR-IOV(SR-IOV Global Enable->Enabled) opened,I'm confused that Does R710 provide full support for SR-IOV, kernel or ixgbe driver's bug? but I'm not sure where the problem lies,anyone has any
experience about this? .

Linux normally uses the resource assignments done by the BIOS, but it is possible for the kernel to reassign those. We don't have good automatic support for that yet, but on a recent upstream kernel, you can try "pci=realloc". I doubt this option is in CentOS 6.3, though.

If an upstream kernel with "pci=realloc" still doesn't work, please post the entire dmesg log.
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Linux normally uses the resource assignments done by the BIOS, but it is possible for the kernel to reassign those. We don't have good automatic support for that yet, but on a recent upstream kernel, you can try "pci=realloc". I doubt this option is in CentOS 6.3, though

DellR710 with the latest BIOS should work fine for SR-IOV. My BIOS is v.6.3.0 and release date is 07/24/2012
Please check if you configured intel_iommu=on in the grub.conf file.
If you did, check your kernel .config file under Device Drivers-> IOMMU Hardware support->enable Support for Intel IOMMU using DMA remapping Devices, enable Intel DMA Remapping Devices by Default, enable Support for Interrupt Remapping.

-----Original Message-----
From: Jason Gao [mailto:[email protected]]
Sent: Tuesday, November 13, 2012 5:38 AM
To: [email protected]; Rose, Gregory V; Li, Sibai
Cc: [email protected]; Kirsher, Jeffrey T; linux-kernel; netdev; kvm; e1000- [email protected]; [email protected]; Yinghai Lu
Subject: Re: SR-IOV problem with Intel 82599EB (not enough MMIO resources for SR-IOV)

I'm very sorry for delayed reply.now SR-IOV works for me in Centos 6.3,thank all of you.

Quoted 4 lines · Bjorn Helgaas

Linux normally uses the resource assignments done by the BIOS, but it is possible for the kernel to reassign those. We don't have good automatic support for that yet, but on a recent upstream kernel, you can try "pci=realloc". I doubt this option is in CentOS 6.3, though

Thank you very much,I try "pci=realloc" in Centos 6.3,and now it works for me.

Quoted 2 lines · Sibai

DellR710 with the latest BIOS should work fine for SR-IOV. My BIOS is v.6.3.0 and release date is 07/24/2012 Please check if you configured

intel_iommu=on in the grub.conf file.

Quoted 1 line

If you did, check your kernel .config file under Device Drivers-> IOMMU

Hardware support->enable Support for Intel IOMMU using DMA remapping Devices, enable Intel DMA Remapping Devices by Default, enable Support for Interrupt Remapping.

thank you Sibai,Our server "Dell R710",its BIOS version is just v.6.3.0 and release date is 07/24/2012,and I also configured intel_iommu=on in the grub.conf file,but I can't find these IOMMU options in "Device Drivers" in my kernel(2.6.32-279) .config file , btw my os is Centos 6.3(RHEL6.3),although the problem solved,I'd like to know what's your os version ,kernel version?

Quoted 29 lines

-----Original Message-----
From: Jason Gao [mailto:[email protected]]
Sent: Tuesday, November 13, 2012 5:38 AM
To: [email protected]; Rose, Gregory V; Li, Sibai
Cc: [email protected]; Kirsher, Jeffrey T; linux-kernel; netdev; kvm; e1000- [email protected]; [email protected]; Yinghai Lu
Subject: Re: SR-IOV problem with Intel 82599EB (not enough MMIO resources for SR-IOV)

I'm very sorry for delayed reply.now SR-IOV works for me in Centos 6.3,thank all of you.

Quoted 4 lines · Bjorn Helgaas

Linux normally uses the resource assignments done by the BIOS, but it is possible for the kernel to reassign those. We don't have good automatic support for that yet, but on a recent upstream kernel, you can try "pci=realloc". I doubt this option is in CentOS 6.3, though

Thank you very much,I try "pci=realloc" in Centos 6.3,and now it works for me.

Quoted 2 lines · Sibai

DellR710 with the latest BIOS should work fine for SR-IOV. My BIOS is v.6.3.0 and release date is 07/24/2012 Please check if you configured

intel_iommu=on in the grub.conf file.

Quoted 8 lines

Quoted 1 line

If you did, check your kernel .config file under Device Drivers-> IOMMU

Hardware support->enable Support for Intel IOMMU using DMA remapping Devices, enable Intel DMA Remapping Devices by Default, enable Support for Interrupt Remapping.

thank you Sibai,Our server "Dell R710",its BIOS version is just v.6.3.0 and release date is 07/24/2012,and I also configured intel_iommu=on in the grub.conf file,but I can't find these IOMMU options in "Device Drivers" in my

Quoted 2 lines

kernel(2.6.32-279) .config file , btw my os is Centos 6.3(RHEL6.3),although the problem solved,I'd like to know what's your os version ,kernel version?

I am using RHEL6.3 with unstable kernel 3.7.0-rc
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 9 lines

Thank you very much,I try "pci=realloc" in Centos 6.3,and now it works for me.

thank you Sibai,Our server "Dell R710",its BIOS version is just v.6.3.0 and release date is 07/24/2012,and I also configured intel_iommu=on in the grub.conf file,but I can't find these IOMMU options in "Device Drivers" in my kernel(2.6.32-279) .config file , btw my os is Centos 6.3(RHEL6.3),although the problem solved,I'd like to know what's your os version ,kernel version?

I am using RHEL6.3 with unstable kernel 3.7.0-rc

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Yinghai Lu
Sent: Tuesday, November 13, 2012 10:17 AM
To: Li, Sibai
Cc: Jason Gao; [email protected]; Rose, Gregory V; [email protected]; Kirsher, Jeffrey T; linux-kernel; netdev; kvm; [email protected]; [email protected]
Subject: Re: SR-IOV problem with Intel 82599EB (not enough MMIO resources for SR-IOV)

Quoted 3 lines · Sibai

Quoted 3 lines

Thank you very much,I try "pci=realloc" in Centos 6.3,and now it works for

me.

Quoted 7 lines

Quoted 7 lines

thank you Sibai,Our server "Dell R710",its BIOS version is just v.6.3.0 and release date is 07/24/2012,and I also configured intel_iommu=on in the grub.conf file,but I can't find these IOMMU options in "Device Drivers" in my
kernel(2.6.32-279) .config file , btw my os is Centos 6.3(RHEL6.3),although the problem solved,I'd like to know what's your os

version ,kernel version?

Quoted 1 line

Quoted 1 line

I am using RHEL6.3 with unstable kernel 3.7.0-rc

that means that config has
CONFIG_PCI_REALLOC_ENABLE_AUTO=y

So you don't need to append "pci=realloc"

Yinghai

Never append "pci=realloc" for both kernel 2.6.32.279 and kernel 3.5.0 above.

-----Original Message-----
From: Jason Gao [mailto:[email protected]]
Sent: Thursday, November 08, 2012 8:00 PM
To: Rose, Gregory V
Cc: Kirsher, Jeffrey T; linux-kernel; netdev; kvm; e1000- [email protected]
Subject: Re: SR-IOV problem with Intel 82599EB (not enough MMIO resources for SR-IOV)

Quoted 1 line

The BIOS in your machine doesn't support SR-IOV. You'll need to ask the

manufacturer for a BIOS upgrade, if in fact one is available. Sometimes they're not.

very thanks Greg,my server Dell R710 with latest BIOS version and option for SR-IOV(SR-IOV Global Enable->Enabled) opened,I'm confused that Does R710 provide full support for SR-IOV, kernel or ixgbe driver's bug? but I'm not sure where the problem lies,anyone has any
experience about this? .

-----Original Message-----
From: Rose, Gregory V
Sent: Friday, November 09, 2012 9:16 AM
To: Jason Gao; Li, Sibai
Cc: Kirsher, Jeffrey T; linux-kernel; netdev; kvm
Subject: RE: SR-IOV problem with Intel 82599EB (not enough MMIO resources for SR-IOV)

Quoted 14 lines

-----Original Message-----
From: Jason Gao [mailto:[email protected]]
Sent: Thursday, November 08, 2012 8:00 PM
To: Rose, Gregory V
Cc: Kirsher, Jeffrey T; linux-kernel; netdev; kvm; e1000- [email protected]
Subject: Re: SR-IOV problem with Intel 82599EB (not enough MMIO resources for SR-IOV)

Quoted 2 lines

The BIOS in your machine doesn't support SR-IOV. You'll need to ask the

manufacturer for a BIOS upgrade, if in fact one is available. Sometimes they're not.

Quoted 6 lines

very thanks Greg,my server Dell R710 with latest BIOS version and option for SR-IOV(SR-IOV Global Enable->Enabled) opened,I'm confused that Does
R710 provide full support for SR-IOV, kernel or ixgbe driver's bug? but I'm not sure where the problem lies,anyone has any
experience about this? .

I use a Dell R710 for all my SR-IOV testing and it works fine but I had to acquire a special BIOS from Dell to get it to work. As I said, you'll want to contact them and make sure you've got the correct BIOS.

The error you're seeing, "not enough MMIO resources" has been an issue with the BIOS 100% of the times that we've run into it. In any case, it is NOT a driver bug. The driver has nothing to do with allocation of MMIO space reservation for VF devices. The message you're seeing pops up when the driver calls pci_enable_sriov(). There is nothing the driver can do to force that call to be successful.

I've added Sibai Li to this response. She should be able to get the BIOS version string off of one of our working Dell R710s.

Sibai,

Could you please reply to this message with that information?

Thanks,

- Greg