Skip to contentSkip to conversation

lenovo T530+Nvida GTX 660 TI eGPU via PE4H

DISCUSSION Joachim Schiele 3 participants 10 messages Mar 18, 2015 – Jun 15, 2015 PCI

lore
Share & tools
lore

Joachim Schiele

author
#0 lore raw

hi,

=== question ===
with the setup described below, i can't pass the graphics adapter into a KVM guest as described below.

is the BAR stuff in the listing an error i wuld have to fix in order to get my eGPU running? if so, what could i do about it? i've seen that the laptop will hang on boot if the eGPU is powered on system startup. i can see the adapter after hotplugging it using lspci and i never see an image on the external display.

thanks in advance,
joachim schiele

my config is this:
sudo qemu-system-x86_64 -enable-kvm \
-m 8192 \
-M q35 \
-cpu host,kvm=off \
-smp 4,sockets=1,cores=4,threads=1 \
-vga none \
-vnc none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=04:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=04:00.1,bus=root.1,addr=00.1 \
-drive file=/home/joachim/en_windows_7_professional_x64_dvd_X15-65805.iso,id=isocd -device ide-cd,bus=ide.1,drive=isocd \
-drive file=/var/lib/libvirt/images/win7.qcow2,format=qcow2,id=hdd -device ide-hd,bus=ide.0,drive=hdd \
-bios /usr/share/seabios/bios.bin \
-soundhw ac97

=== when hotplugging the eGPU ===
i want to use a Nvidia GTX 660 TI as an eGPU using PE4H and when i hotplug the ePCI adapter i get this log on a 3.18 ubuntu kernel:

[  960.145002] vgaarb: device changed decodes:
PCI:0000:00:02.0,olddecodes=none,decodes=io+mem:owns=none
[  966.574858] pci 0000:04:00.0: [10de:1183] type 00 class 0x030000
[  966.574892] pci 0000:04:00.0: reg 0x10: [mem 0x00000000-0x00ffffff]
[  966.574921] pci 0000:04:00.0: reg 0x14: [mem 0x00000000-0x07ffffff
64bit pref]
[  966.574950] pci 0000:04:00.0: reg 0x1c: [mem 0x00000000-0x01ffffff
64bit pref]
[  966.574963] pci 0000:04:00.0: reg 0x24: [io  0x0000-0x007f]
[  966.574974] pci 0000:04:00.0: reg 0x30: [mem 0x00000000-0x0007ffff pref]
[  966.575224] vgaarb: device added:
PCI:0000:04:00.0,decodes=io+mem,owns=none,locks=none
[  966.575227] vgaarb: device changed decodes:
PCI:0000:00:02.0,olddecodes=io+mem,decodes=none:owns=none
[  966.575327] pci 0000:04:00.1: [10de:0e0a] type 00 class 0x040300
[  966.575356] pci 0000:04:00.1: reg 0x10: [mem 0x00000000-0x00003fff]
[  966.998345] pci 0000:04:00.0: BAR 1: no space for [mem size
0x08000000 64bit pref]
[  966.998350] pci 0000:04:00.0: BAR 1: failed to assign [mem size
0x08000000 64bit pref]
[  966.998353] pci 0000:04:00.0: BAR 3: no space for [mem size
0x02000000 64bit pref]
[  966.998355] pci 0000:04:00.0: BAR 3: failed to assign [mem size
0x02000000 64bit pref]
[  966.998357] pci 0000:04:00.0: BAR 0: no space for [mem size 0x01000000]
[  966.998358] pci 0000:04:00.0: BAR 0: failed to assign [mem size
0x01000000]
[  966.998360] pci 0000:04:00.0: BAR 6: assigned [mem
0xf1400000-0xf147ffff pref]
[  966.998363] pci 0000:04:00.1: BAR 0: assigned [mem 0xf1480000-0xf1483fff]
[  966.998369] pci 0000:04:00.1: BAR 0: error updating (0xf1480000 !=
0xffffffff)
[  966.998373] pci 0000:04:00.0: BAR 5: assigned [io  0x3000-0x307f]
[  966.998379] pci 0000:04:00.0: BAR 5: error updating (0x003001 !=
0xffffffff)
[  966.998392] pcieport 0000:00:1c.2: Max Payload Size set to  128/ 128
(was  128), Max Read Rq  128
[  966.998403] pci 0000:04:00.0: Max Payload Size set to 16384/ 256 (was
16384), Max Read Rq 16384
[  966.998414] pci 0000:04:00.1: Max Payload Size set to 16384/ 256 (was
16384), Max Read Rq 16384
[  966.998472] pci-stub 0000:04:00.0: claimed by stub
[  966.998495] pci-stub 0000:04:00.1: claimed by stub
[  967.099040] pcieport 0000:00:1c.2: Max Payload Size set to  128/ 128
(was  128), Max Read Rq  128
[  967.099059] pci-stub 0000:04:00.0: Max Payload Size set to  128/ 256
(was  128), Max Read Rq  512
[  967.099075] pci-stub 0000:04:00.1: Max Payload Size set to  128/ 256
(was  128), Max Read Rq  512
[ 7735.938854] vgaarb: device changed decodes:

PCI:0000:00:02.0,olddecodes=none,decodes=io+mem:owns=none

more at:

https://lastlog.de/wiki/index.php/T530%2BNvida_GTX_660_TI_eGPU_via_PE4H

i derived the scripts i'm using from this guide:
http://www.pugetsystems.com/labs/articles/Multiheaded-NVIDIA-Gaming-using-Ubuntu-14-04-KVM-585/

maintainer
#1 lore raw

[+cc Alex]

Hi Joachim,

Thanks a lot for the bug report.

Quoted 8 lines · Joachim Schiele

hi,

=== question ===
with the setup described below, i can't pass the graphics adapter into a KVM guest as described below.

is the BAR stuff in the listing an error i wuld have to fix in order to get my eGPU running? if so, what could i do about it

Yes. If we can't assign resources to the BARs, we can't enable the device. If you boot with "pci=realloc", Linux will try harder to assign space (including moving other devices to make space if necessary).

Quoted 2 lines

i've seen that the
laptop will hang on boot if the eGPU is powered on system startup.

It's possible the BIOS is hanging because it tried to assign space for the eGPU but failed.

Quoted 3 lines

i can
see the adapter after hotplugging it using lspci and i never see an image on the external display.

The device will appear in lspci even if we can't assign space for it. In that case the device would not work, even though it appears in lspci.

So please try "pci=realloc". Also, please collect a complete dmesg log and "lspci -vv" output (with eGPU attached). That will contain the information needed to see whether it's possible even in principle to make this work.

Quoted 91 lines

my config is this:
sudo qemu-system-x86_64 -enable-kvm \

(quoted diff) +0 -7
-m 8192 \
-M q35 \
-cpu host,kvm=off \
-smp 4,sockets=1,cores=4,threads=1 \
-vga none \
-vnc none \
-device

ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=04:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \

(quoted diff) +0 -2
-device vfio-pci,host=04:00.1,bus=root.1,addr=00.1 \
-drive

file=/home/joachim/en_windows_7_professional_x64_dvd_X15-65805.iso,id=isocd

(quoted diff) +0 -5
-device ide-cd,bus=ide.1,drive=isocd \
-drive file=/var/lib/libvirt/images/win7.qcow2,format=qcow2,id=hdd
-device ide-hd,bus=ide.0,drive=hdd \
-bios /usr/share/seabios/bios.bin \
-soundhw ac97

=== when hotplugging the eGPU ===
i want to use a Nvidia GTX 660 TI as an eGPU using PE4H and when i hotplug the ePCI adapter i get this log on a 3.18 ubuntu kernel:

[  960.145002] vgaarb: device changed decodes:
PCI:0000:00:02.0,olddecodes=none,decodes=io+mem:owns=none
[  966.574858] pci 0000:04:00.0: [10de:1183] type 00 class 0x030000
[  966.574892] pci 0000:04:00.0: reg 0x10: [mem 0x00000000-0x00ffffff]
[  966.574921] pci 0000:04:00.0: reg 0x14: [mem 0x00000000-0x07ffffff
64bit pref]
[  966.574950] pci 0000:04:00.0: reg 0x1c: [mem 0x00000000-0x01ffffff
64bit pref]
[  966.574963] pci 0000:04:00.0: reg 0x24: [io  0x0000-0x007f]
[  966.574974] pci 0000:04:00.0: reg 0x30: [mem 0x00000000-0x0007ffff pref]
[  966.575224] vgaarb: device added:
PCI:0000:04:00.0,decodes=io+mem,owns=none,locks=none
[  966.575227] vgaarb: device changed decodes:
PCI:0000:00:02.0,olddecodes=io+mem,decodes=none:owns=none
[  966.575327] pci 0000:04:00.1: [10de:0e0a] type 00 class 0x040300
[  966.575356] pci 0000:04:00.1: reg 0x10: [mem 0x00000000-0x00003fff]
[  966.998345] pci 0000:04:00.0: BAR 1: no space for [mem size
0x08000000 64bit pref]
[  966.998350] pci 0000:04:00.0: BAR 1: failed to assign [mem size
0x08000000 64bit pref]
[  966.998353] pci 0000:04:00.0: BAR 3: no space for [mem size
0x02000000 64bit pref]
[  966.998355] pci 0000:04:00.0: BAR 3: failed to assign [mem size
0x02000000 64bit pref]
[  966.998357] pci 0000:04:00.0: BAR 0: no space for [mem size 0x01000000]
[  966.998358] pci 0000:04:00.0: BAR 0: failed to assign [mem size
0x01000000]
[  966.998360] pci 0000:04:00.0: BAR 6: assigned [mem
0xf1400000-0xf147ffff pref]
[  966.998363] pci 0000:04:00.1: BAR 0: assigned [mem 0xf1480000-0xf1483fff]
[  966.998369] pci 0000:04:00.1: BAR 0: error updating (0xf1480000 !=
0xffffffff)
[  966.998373] pci 0000:04:00.0: BAR 5: assigned [io  0x3000-0x307f]
[  966.998379] pci 0000:04:00.0: BAR 5: error updating (0x003001 !=
0xffffffff)
[  966.998392] pcieport 0000:00:1c.2: Max Payload Size set to  128/ 128
(was  128), Max Read Rq  128
[  966.998403] pci 0000:04:00.0: Max Payload Size set to 16384/ 256 (was
16384), Max Read Rq 16384
[  966.998414] pci 0000:04:00.1: Max Payload Size set to 16384/ 256 (was
16384), Max Read Rq 16384
[  966.998472] pci-stub 0000:04:00.0: claimed by stub
[  966.998495] pci-stub 0000:04:00.1: claimed by stub
[  967.099040] pcieport 0000:00:1c.2: Max Payload Size set to  128/ 128
(was  128), Max Read Rq  128
[  967.099059] pci-stub 0000:04:00.0: Max Payload Size set to  128/ 256
(was  128), Max Read Rq  512
[  967.099075] pci-stub 0000:04:00.1: Max Payload Size set to  128/ 256
(was  128), Max Read Rq  512
[ 7735.938854] vgaarb: device changed decodes:

PCI:0000:00:02.0,olddecodes=none,decodes=io+mem:owns=none

more at:

https://lastlog.de/wiki/index.php/T530%2BNvida_GTX_660_TI_eGPU_via_PE4H

i derived the scripts i'm using from this guide:
http://www.pugetsystems.com/labs/articles/Multiheaded-NVIDIA-Gaming-using-Ubuntu-14-04-KVM-585/

--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

author
#2 lore raw

hi,

i've run with pci=realloc two tests, the results are here (also the dmesg) outputs:

https://lastlog.de/wiki/index.php/T530%2BNvida_GTX_660_TI_eGPU_via_PE4H#debugging_using_pci.3Drealloc

thanks in advance,
joachim schiele

Quoted 131 lines · Bjorn Helgaas

[+cc Alex]

Hi Joachim,

Thanks a lot for the bug report.

Quoted 8 lines · Joachim Schiele

hi,

=== question ===
with the setup described below, i can't pass the graphics adapter into a KVM guest as described below.

is the BAR stuff in the listing an error i wuld have to fix in order to get my eGPU running? if so, what could i do about it

Yes. If we can't assign resources to the BARs, we can't enable the device. If you boot with "pci=realloc", Linux will try harder to assign space (including moving other devices to make space if necessary).

Quoted 2 lines

i've seen that the
laptop will hang on boot if the eGPU is powered on system startup.

It's possible the BIOS is hanging because it tried to assign space for the eGPU but failed.

Quoted 3 lines

i can
see the adapter after hotplugging it using lspci and i never see an image on the external display.

The device will appear in lspci even if we can't assign space for it. In that case the device would not work, even though it appears in lspci.

So please try "pci=realloc". Also, please collect a complete dmesg log and "lspci -vv" output (with eGPU attached). That will contain the information needed to see whether it's possible even in principle to make this work.

Quoted 91 lines

my config is this:
sudo qemu-system-x86_64 -enable-kvm \

(quoted diff) +0 -7
-m 8192 \
-M q35 \
-cpu host,kvm=off \
-smp 4,sockets=1,cores=4,threads=1 \
-vga none \
-vnc none \
-device

ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=04:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \

(quoted diff) +0 -2
-device vfio-pci,host=04:00.1,bus=root.1,addr=00.1 \
-drive

file=/home/joachim/en_windows_7_professional_x64_dvd_X15-65805.iso,id=isocd

(quoted diff) +0 -5
-device ide-cd,bus=ide.1,drive=isocd \
-drive file=/var/lib/libvirt/images/win7.qcow2,format=qcow2,id=hdd
-device ide-hd,bus=ide.0,drive=hdd \
-bios /usr/share/seabios/bios.bin \
-soundhw ac97

=== when hotplugging the eGPU ===
i want to use a Nvidia GTX 660 TI as an eGPU using PE4H and when i hotplug the ePCI adapter i get this log on a 3.18 ubuntu kernel:

[  960.145002] vgaarb: device changed decodes:
PCI:0000:00:02.0,olddecodes=none,decodes=io+mem:owns=none
[  966.574858] pci 0000:04:00.0: [10de:1183] type 00 class 0x030000
[  966.574892] pci 0000:04:00.0: reg 0x10: [mem 0x00000000-0x00ffffff]
[  966.574921] pci 0000:04:00.0: reg 0x14: [mem 0x00000000-0x07ffffff
64bit pref]
[  966.574950] pci 0000:04:00.0: reg 0x1c: [mem 0x00000000-0x01ffffff
64bit pref]
[  966.574963] pci 0000:04:00.0: reg 0x24: [io  0x0000-0x007f]
[  966.574974] pci 0000:04:00.0: reg 0x30: [mem 0x00000000-0x0007ffff pref]
[  966.575224] vgaarb: device added:
PCI:0000:04:00.0,decodes=io+mem,owns=none,locks=none
[  966.575227] vgaarb: device changed decodes:
PCI:0000:00:02.0,olddecodes=io+mem,decodes=none:owns=none
[  966.575327] pci 0000:04:00.1: [10de:0e0a] type 00 class 0x040300
[  966.575356] pci 0000:04:00.1: reg 0x10: [mem 0x00000000-0x00003fff]
[  966.998345] pci 0000:04:00.0: BAR 1: no space for [mem size
0x08000000 64bit pref]
[  966.998350] pci 0000:04:00.0: BAR 1: failed to assign [mem size
0x08000000 64bit pref]
[  966.998353] pci 0000:04:00.0: BAR 3: no space for [mem size
0x02000000 64bit pref]
[  966.998355] pci 0000:04:00.0: BAR 3: failed to assign [mem size
0x02000000 64bit pref]
[  966.998357] pci 0000:04:00.0: BAR 0: no space for [mem size 0x01000000]
[  966.998358] pci 0000:04:00.0: BAR 0: failed to assign [mem size
0x01000000]
[  966.998360] pci 0000:04:00.0: BAR 6: assigned [mem
0xf1400000-0xf147ffff pref]
[  966.998363] pci 0000:04:00.1: BAR 0: assigned [mem 0xf1480000-0xf1483fff]
[  966.998369] pci 0000:04:00.1: BAR 0: error updating (0xf1480000 !=
0xffffffff)
[  966.998373] pci 0000:04:00.0: BAR 5: assigned [io  0x3000-0x307f]
[  966.998379] pci 0000:04:00.0: BAR 5: error updating (0x003001 !=
0xffffffff)
[  966.998392] pcieport 0000:00:1c.2: Max Payload Size set to  128/ 128
(was  128), Max Read Rq  128
[  966.998403] pci 0000:04:00.0: Max Payload Size set to 16384/ 256 (was
16384), Max Read Rq 16384
[  966.998414] pci 0000:04:00.1: Max Payload Size set to 16384/ 256 (was
16384), Max Read Rq 16384
[  966.998472] pci-stub 0000:04:00.0: claimed by stub
[  966.998495] pci-stub 0000:04:00.1: claimed by stub
[  967.099040] pcieport 0000:00:1c.2: Max Payload Size set to  128/ 128
(was  128), Max Read Rq  128
[  967.099059] pci-stub 0000:04:00.0: Max Payload Size set to  128/ 256
(was  128), Max Read Rq  512
[  967.099075] pci-stub 0000:04:00.1: Max Payload Size set to  128/ 256
(was  128), Max Read Rq  512
[ 7735.938854] vgaarb: device changed decodes:

PCI:0000:00:02.0,olddecodes=none,decodes=io+mem:owns=none

more at:

https://lastlog.de/wiki/index.php/T530%2BNvida_GTX_660_TI_eGPU_via_PE4H

i derived the scripts i'm using from this guide:
http://www.pugetsystems.com/labs/articles/Multiheaded-NVIDIA-Gaming-using-Ubuntu-14-04-KVM-585/

--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

author
#3 lore raw

just wanted to report that i got the external graphics card to work with native windows 7 and setup 1.x + pci compaction!

see:
http://blog.lastlog.de/posts/setup-1.x/

i hope to get it also working in linux host using kvm+win7 as guest soon.

Quoted 145 lines · Joachim Schiele

hi,

i've run with pci=realloc two tests, the results are here (also the dmesg) outputs:

https://lastlog.de/wiki/index.php/T530%2BNvida_GTX_660_TI_eGPU_via_PE4H#debugging_using_pci.3Drealloc

thanks in advance,
joachim schiele

Quoted 131 lines · Bjorn Helgaas

[+cc Alex]

Hi Joachim,

Thanks a lot for the bug report.

Quoted 8 lines · Joachim Schiele

hi,

=== question ===
with the setup described below, i can't pass the graphics adapter into a KVM guest as described below.

is the BAR stuff in the listing an error i wuld have to fix in order to get my eGPU running? if so, what could i do about it

Yes. If we can't assign resources to the BARs, we can't enable the device. If you boot with "pci=realloc", Linux will try harder to assign space (including moving other devices to make space if necessary).

Quoted 2 lines

i've seen that the
laptop will hang on boot if the eGPU is powered on system startup.

It's possible the BIOS is hanging because it tried to assign space for the eGPU but failed.

Quoted 3 lines

i can
see the adapter after hotplugging it using lspci and i never see an image on the external display.

The device will appear in lspci even if we can't assign space for it. In that case the device would not work, even though it appears in lspci.

So please try "pci=realloc". Also, please collect a complete dmesg log and "lspci -vv" output (with eGPU attached). That will contain the information needed to see whether it's possible even in principle to make this work.

Quoted 91 lines

my config is this:
sudo qemu-system-x86_64 -enable-kvm \

(quoted diff) +0 -7
-m 8192 \
-M q35 \
-cpu host,kvm=off \
-smp 4,sockets=1,cores=4,threads=1 \
-vga none \
-vnc none \
-device

ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=04:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \

(quoted diff) +0 -2
-device vfio-pci,host=04:00.1,bus=root.1,addr=00.1 \
-drive

file=/home/joachim/en_windows_7_professional_x64_dvd_X15-65805.iso,id=isocd

(quoted diff) +0 -5
-device ide-cd,bus=ide.1,drive=isocd \
-drive file=/var/lib/libvirt/images/win7.qcow2,format=qcow2,id=hdd
-device ide-hd,bus=ide.0,drive=hdd \
-bios /usr/share/seabios/bios.bin \
-soundhw ac97

=== when hotplugging the eGPU ===
i want to use a Nvidia GTX 660 TI as an eGPU using PE4H and when i hotplug the ePCI adapter i get this log on a 3.18 ubuntu kernel:

[  960.145002] vgaarb: device changed decodes:
PCI:0000:00:02.0,olddecodes=none,decodes=io+mem:owns=none
[  966.574858] pci 0000:04:00.0: [10de:1183] type 00 class 0x030000
[  966.574892] pci 0000:04:00.0: reg 0x10: [mem 0x00000000-0x00ffffff]
[  966.574921] pci 0000:04:00.0: reg 0x14: [mem 0x00000000-0x07ffffff
64bit pref]
[  966.574950] pci 0000:04:00.0: reg 0x1c: [mem 0x00000000-0x01ffffff
64bit pref]
[  966.574963] pci 0000:04:00.0: reg 0x24: [io  0x0000-0x007f]
[  966.574974] pci 0000:04:00.0: reg 0x30: [mem 0x00000000-0x0007ffff pref]
[  966.575224] vgaarb: device added:
PCI:0000:04:00.0,decodes=io+mem,owns=none,locks=none
[  966.575227] vgaarb: device changed decodes:
PCI:0000:00:02.0,olddecodes=io+mem,decodes=none:owns=none
[  966.575327] pci 0000:04:00.1: [10de:0e0a] type 00 class 0x040300
[  966.575356] pci 0000:04:00.1: reg 0x10: [mem 0x00000000-0x00003fff]
[  966.998345] pci 0000:04:00.0: BAR 1: no space for [mem size
0x08000000 64bit pref]
[  966.998350] pci 0000:04:00.0: BAR 1: failed to assign [mem size
0x08000000 64bit pref]
[  966.998353] pci 0000:04:00.0: BAR 3: no space for [mem size
0x02000000 64bit pref]
[  966.998355] pci 0000:04:00.0: BAR 3: failed to assign [mem size
0x02000000 64bit pref]
[  966.998357] pci 0000:04:00.0: BAR 0: no space for [mem size 0x01000000]
[  966.998358] pci 0000:04:00.0: BAR 0: failed to assign [mem size
0x01000000]
[  966.998360] pci 0000:04:00.0: BAR 6: assigned [mem
0xf1400000-0xf147ffff pref]
[  966.998363] pci 0000:04:00.1: BAR 0: assigned [mem 0xf1480000-0xf1483fff]
[  966.998369] pci 0000:04:00.1: BAR 0: error updating (0xf1480000 !=
0xffffffff)
[  966.998373] pci 0000:04:00.0: BAR 5: assigned [io  0x3000-0x307f]
[  966.998379] pci 0000:04:00.0: BAR 5: error updating (0x003001 !=
0xffffffff)
[  966.998392] pcieport 0000:00:1c.2: Max Payload Size set to  128/ 128
(was  128), Max Read Rq  128
[  966.998403] pci 0000:04:00.0: Max Payload Size set to 16384/ 256 (was
16384), Max Read Rq 16384
[  966.998414] pci 0000:04:00.1: Max Payload Size set to 16384/ 256 (was
16384), Max Read Rq 16384
[  966.998472] pci-stub 0000:04:00.0: claimed by stub
[  966.998495] pci-stub 0000:04:00.1: claimed by stub
[  967.099040] pcieport 0000:00:1c.2: Max Payload Size set to  128/ 128
(was  128), Max Read Rq  128
[  967.099059] pci-stub 0000:04:00.0: Max Payload Size set to  128/ 256
(was  128), Max Read Rq  512
[  967.099075] pci-stub 0000:04:00.1: Max Payload Size set to  128/ 256
(was  128), Max Read Rq  512
[ 7735.938854] vgaarb: device changed decodes:

PCI:0000:00:02.0,olddecodes=none,decodes=io+mem:owns=none

more at:

https://lastlog.de/wiki/index.php/T530%2BNvida_GTX_660_TI_eGPU_via_PE4H

i derived the scripts i'm using from this guide:
http://www.pugetsystems.com/labs/articles/Multiheaded-NVIDIA-Gaming-using-Ubuntu-14-04-KVM-585/

--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#4 lore raw
Quoted 7 lines · Joachim Schiele

just wanted to report that i got the external graphics card to work with native windows 7 and setup 1.x + pci compaction!

see:
http://blog.lastlog.de/posts/setup-1.x/

i hope to get it also working in linux host using kvm+win7 as guest soon.

I just setup something similar, using the EXP GDC "Beast" rather than the PE4H. My Lenovo T530 only leaves 390MB for MMIO space and doesn't provide any 64bit space via ACPI. I also need to fiddle with the switches to delay the card so the laptop doesn't hang during BIOS boot. I pass pci=nocrs,realloc on the kernel commandline in order to get the GPU programmed. I've only actually tested it with KVM/VFIO GPU assignment and it seems to work well, or at least as well as can be expected over a x1 5GT/s link. Thanks,

Alex

Quoted 152 lines · Joachim Schiele
Quoted 136 lines

hi,

i've run with pci=realloc two tests, the results are here (also the dmesg) outputs:

https://lastlog.de/wiki/index.php/T530%2BNvida_GTX_660_TI_eGPU_via_PE4H#debugging_using_pci.3Drealloc

thanks in advance,
joachim schiele

Quoted 124 lines · Bjorn Helgaas

[+cc Alex]

Hi Joachim,

Thanks a lot for the bug report.

Quoted 8 lines · Joachim Schiele

hi,

=== question ===
with the setup described below, i can't pass the graphics adapter into a KVM guest as described below.

is the BAR stuff in the listing an error i wuld have to fix in order to get my eGPU running? if so, what could i do about it

Yes. If we can't assign resources to the BARs, we can't enable the device. If you boot with "pci=realloc", Linux will try harder to assign space (including moving other devices to make space if necessary).

Quoted 2 lines

i've seen that the
laptop will hang on boot if the eGPU is powered on system startup.

It's possible the BIOS is hanging because it tried to assign space for the eGPU but failed.

Quoted 3 lines

i can
see the adapter after hotplugging it using lspci and i never see an image on the external display.

The device will appear in lspci even if we can't assign space for it. In that case the device would not work, even though it appears in lspci.

So please try "pci=realloc". Also, please collect a complete dmesg log and "lspci -vv" output (with eGPU attached). That will contain the information needed to see whether it's possible even in principle to make this work.

Quoted 84 lines

my config is this:
sudo qemu-system-x86_64 -enable-kvm \

(quoted diff) +0 -7
-m 8192 \
-M q35 \
-cpu host,kvm=off \
-smp 4,sockets=1,cores=4,threads=1 \
-vga none \
-vnc none \
-device

ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=04:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \

(quoted diff) +0 -2
-device vfio-pci,host=04:00.1,bus=root.1,addr=00.1 \
-drive

file=/home/joachim/en_windows_7_professional_x64_dvd_X15-65805.iso,id=isocd

(quoted diff) +0 -5
-device ide-cd,bus=ide.1,drive=isocd \
-drive file=/var/lib/libvirt/images/win7.qcow2,format=qcow2,id=hdd
-device ide-hd,bus=ide.0,drive=hdd \
-bios /usr/share/seabios/bios.bin \
-soundhw ac97

=== when hotplugging the eGPU ===
i want to use a Nvidia GTX 660 TI as an eGPU using PE4H and when i hotplug the ePCI adapter i get this log on a 3.18 ubuntu kernel:

[  960.145002] vgaarb: device changed decodes:
PCI:0000:00:02.0,olddecodes=none,decodes=io+mem:owns=none
[  966.574858] pci 0000:04:00.0: [10de:1183] type 00 class 0x030000
[  966.574892] pci 0000:04:00.0: reg 0x10: [mem 0x00000000-0x00ffffff]
[  966.574921] pci 0000:04:00.0: reg 0x14: [mem 0x00000000-0x07ffffff
64bit pref]
[  966.574950] pci 0000:04:00.0: reg 0x1c: [mem 0x00000000-0x01ffffff
64bit pref]
[  966.574963] pci 0000:04:00.0: reg 0x24: [io  0x0000-0x007f]
[  966.574974] pci 0000:04:00.0: reg 0x30: [mem 0x00000000-0x0007ffff pref]
[  966.575224] vgaarb: device added:
PCI:0000:04:00.0,decodes=io+mem,owns=none,locks=none
[  966.575227] vgaarb: device changed decodes:
PCI:0000:00:02.0,olddecodes=io+mem,decodes=none:owns=none
[  966.575327] pci 0000:04:00.1: [10de:0e0a] type 00 class 0x040300
[  966.575356] pci 0000:04:00.1: reg 0x10: [mem 0x00000000-0x00003fff]
[  966.998345] pci 0000:04:00.0: BAR 1: no space for [mem size
0x08000000 64bit pref]
[  966.998350] pci 0000:04:00.0: BAR 1: failed to assign [mem size
0x08000000 64bit pref]
[  966.998353] pci 0000:04:00.0: BAR 3: no space for [mem size
0x02000000 64bit pref]
[  966.998355] pci 0000:04:00.0: BAR 3: failed to assign [mem size
0x02000000 64bit pref]
[  966.998357] pci 0000:04:00.0: BAR 0: no space for [mem size 0x01000000]
[  966.998358] pci 0000:04:00.0: BAR 0: failed to assign [mem size
0x01000000]
[  966.998360] pci 0000:04:00.0: BAR 6: assigned [mem
0xf1400000-0xf147ffff pref]
[  966.998363] pci 0000:04:00.1: BAR 0: assigned [mem 0xf1480000-0xf1483fff]
[  966.998369] pci 0000:04:00.1: BAR 0: error updating (0xf1480000 !=
0xffffffff)
[  966.998373] pci 0000:04:00.0: BAR 5: assigned [io  0x3000-0x307f]
[  966.998379] pci 0000:04:00.0: BAR 5: error updating (0x003001 !=
0xffffffff)
[  966.998392] pcieport 0000:00:1c.2: Max Payload Size set to  128/ 128
(was  128), Max Read Rq  128
[  966.998403] pci 0000:04:00.0: Max Payload Size set to 16384/ 256 (was
16384), Max Read Rq 16384
[  966.998414] pci 0000:04:00.1: Max Payload Size set to 16384/ 256 (was
16384), Max Read Rq 16384
[  966.998472] pci-stub 0000:04:00.0: claimed by stub
[  966.998495] pci-stub 0000:04:00.1: claimed by stub
[  967.099040] pcieport 0000:00:1c.2: Max Payload Size set to  128/ 128
(was  128), Max Read Rq  128
[  967.099059] pci-stub 0000:04:00.0: Max Payload Size set to  128/ 256
(was  128), Max Read Rq  512
[  967.099075] pci-stub 0000:04:00.1: Max Payload Size set to  128/ 256
(was  128), Max Read Rq  512
[ 7735.938854] vgaarb: device changed decodes:

PCI:0000:00:02.0,olddecodes=none,decodes=io+mem:owns=none

more at:

https://lastlog.de/wiki/index.php/T530%2BNvida_GTX_660_TI_eGPU_via_PE4H

Quoted 9 lines
Quoted 7 lines
Quoted 7 lines

i derived the scripts i'm using from this guide:
http://www.pugetsystems.com/labs/articles/Multiheaded-NVIDIA-Gaming-using-Ubuntu-14-04-KVM-585/

--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

author
#5 lore raw

ah nice! thanks for reporting back. i'm looking forward to testing your cmdline this weekend.

Quoted 175 lines · Alex Williamson
Quoted 7 lines · Joachim Schiele

just wanted to report that i got the external graphics card to work with native windows 7 and setup 1.x + pci compaction!

see:
http://blog.lastlog.de/posts/setup-1.x/

i hope to get it also working in linux host using kvm+win7 as guest soon.

I just setup something similar, using the EXP GDC "Beast" rather than the PE4H. My Lenovo T530 only leaves 390MB for MMIO space and doesn't provide any 64bit space via ACPI. I also need to fiddle with the switches to delay the card so the laptop doesn't hang during BIOS boot. I pass pci=nocrs,realloc on the kernel commandline in order to get the GPU programmed. I've only actually tested it with KVM/VFIO GPU assignment and it seems to work well, or at least as well as can be expected over a x1 5GT/s link. Thanks,

Alex

Quoted 152 lines · Joachim Schiele
Quoted 136 lines

hi,

i've run with pci=realloc two tests, the results are here (also the dmesg) outputs:

https://lastlog.de/wiki/index.php/T530%2BNvida_GTX_660_TI_eGPU_via_PE4H#debugging_using_pci.3Drealloc

thanks in advance,
joachim schiele

Quoted 124 lines · Bjorn Helgaas

[+cc Alex]

Hi Joachim,

Thanks a lot for the bug report.

Quoted 8 lines · Joachim Schiele

hi,

=== question ===
with the setup described below, i can't pass the graphics adapter into a KVM guest as described below.

is the BAR stuff in the listing an error i wuld have to fix in order to get my eGPU running? if so, what could i do about it

Yes. If we can't assign resources to the BARs, we can't enable the device. If you boot with "pci=realloc", Linux will try harder to assign space (including moving other devices to make space if necessary).

Quoted 2 lines

i've seen that the
laptop will hang on boot if the eGPU is powered on system startup.

It's possible the BIOS is hanging because it tried to assign space for the eGPU but failed.

Quoted 3 lines

i can
see the adapter after hotplugging it using lspci and i never see an image on the external display.

The device will appear in lspci even if we can't assign space for it. In that case the device would not work, even though it appears in lspci.

So please try "pci=realloc". Also, please collect a complete dmesg log and "lspci -vv" output (with eGPU attached). That will contain the information needed to see whether it's possible even in principle to make this work.

Quoted 84 lines

my config is this:
sudo qemu-system-x86_64 -enable-kvm \

(quoted diff) +0 -7
-m 8192 \
-M q35 \
-cpu host,kvm=off \
-smp 4,sockets=1,cores=4,threads=1 \
-vga none \
-vnc none \
-device

ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=04:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \

(quoted diff) +0 -2
-device vfio-pci,host=04:00.1,bus=root.1,addr=00.1 \
-drive

file=/home/joachim/en_windows_7_professional_x64_dvd_X15-65805.iso,id=isocd

(quoted diff) +0 -5
-device ide-cd,bus=ide.1,drive=isocd \
-drive file=/var/lib/libvirt/images/win7.qcow2,format=qcow2,id=hdd
-device ide-hd,bus=ide.0,drive=hdd \
-bios /usr/share/seabios/bios.bin \
-soundhw ac97

=== when hotplugging the eGPU ===
i want to use a Nvidia GTX 660 TI as an eGPU using PE4H and when i hotplug the ePCI adapter i get this log on a 3.18 ubuntu kernel:

[  960.145002] vgaarb: device changed decodes:
PCI:0000:00:02.0,olddecodes=none,decodes=io+mem:owns=none
[  966.574858] pci 0000:04:00.0: [10de:1183] type 00 class 0x030000
[  966.574892] pci 0000:04:00.0: reg 0x10: [mem 0x00000000-0x00ffffff]
[  966.574921] pci 0000:04:00.0: reg 0x14: [mem 0x00000000-0x07ffffff
64bit pref]
[  966.574950] pci 0000:04:00.0: reg 0x1c: [mem 0x00000000-0x01ffffff
64bit pref]
[  966.574963] pci 0000:04:00.0: reg 0x24: [io  0x0000-0x007f]
[  966.574974] pci 0000:04:00.0: reg 0x30: [mem 0x00000000-0x0007ffff pref]
[  966.575224] vgaarb: device added:
PCI:0000:04:00.0,decodes=io+mem,owns=none,locks=none
[  966.575227] vgaarb: device changed decodes:
PCI:0000:00:02.0,olddecodes=io+mem,decodes=none:owns=none
[  966.575327] pci 0000:04:00.1: [10de:0e0a] type 00 class 0x040300
[  966.575356] pci 0000:04:00.1: reg 0x10: [mem 0x00000000-0x00003fff]
[  966.998345] pci 0000:04:00.0: BAR 1: no space for [mem size
0x08000000 64bit pref]
[  966.998350] pci 0000:04:00.0: BAR 1: failed to assign [mem size
0x08000000 64bit pref]
[  966.998353] pci 0000:04:00.0: BAR 3: no space for [mem size
0x02000000 64bit pref]
[  966.998355] pci 0000:04:00.0: BAR 3: failed to assign [mem size
0x02000000 64bit pref]
[  966.998357] pci 0000:04:00.0: BAR 0: no space for [mem size 0x01000000]
[  966.998358] pci 0000:04:00.0: BAR 0: failed to assign [mem size
0x01000000]
[  966.998360] pci 0000:04:00.0: BAR 6: assigned [mem
0xf1400000-0xf147ffff pref]
[  966.998363] pci 0000:04:00.1: BAR 0: assigned [mem 0xf1480000-0xf1483fff]
[  966.998369] pci 0000:04:00.1: BAR 0: error updating (0xf1480000 !=
0xffffffff)
[  966.998373] pci 0000:04:00.0: BAR 5: assigned [io  0x3000-0x307f]
[  966.998379] pci 0000:04:00.0: BAR 5: error updating (0x003001 !=
0xffffffff)
[  966.998392] pcieport 0000:00:1c.2: Max Payload Size set to  128/ 128
(was  128), Max Read Rq  128
[  966.998403] pci 0000:04:00.0: Max Payload Size set to 16384/ 256 (was
16384), Max Read Rq 16384
[  966.998414] pci 0000:04:00.1: Max Payload Size set to 16384/ 256 (was
16384), Max Read Rq 16384
[  966.998472] pci-stub 0000:04:00.0: claimed by stub
[  966.998495] pci-stub 0000:04:00.1: claimed by stub
[  967.099040] pcieport 0000:00:1c.2: Max Payload Size set to  128/ 128
(was  128), Max Read Rq  128
[  967.099059] pci-stub 0000:04:00.0: Max Payload Size set to  128/ 256
(was  128), Max Read Rq  512
[  967.099075] pci-stub 0000:04:00.1: Max Payload Size set to  128/ 256
(was  128), Max Read Rq  512
[ 7735.938854] vgaarb: device changed decodes:

PCI:0000:00:02.0,olddecodes=none,decodes=io+mem:owns=none

more at:

https://lastlog.de/wiki/index.php/T530%2BNvida_GTX_660_TI_eGPU_via_PE4H

Quoted 9 lines
Quoted 7 lines
Quoted 7 lines

i derived the scripts i'm using from this guide:
http://www.pugetsystems.com/labs/articles/Multiheaded-NVIDIA-Gaming-using-Ubuntu-14-04-KVM-585/

--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 6 lines
Joachim Schiele

blog: http://blog.lastlog.de
wiki: http://lastlog.de
jabber: [email protected]
GPG: C6AC8770
author
#6 lore raw
In reply to Alex Williamson
Quoted 7 lines · Alex Williamson
Quoted 6 lines · Joachim Schiele

just wanted to report that i got the external graphics card to work with native windows 7 and setup 1.x + pci compaction!

see:
http://blog.lastlog.de/posts/setup-1.x/

Quoted 6 lines
Quoted 1 line

i hope to get it also working in linux host using kvm+win7 as guest soon.

I just setup something similar, using the EXP GDC "Beast" rather than the PE4H. My Lenovo T530 only leaves 390MB for MMIO space and doesn't provide any 64bit space via ACPI. I also need to fiddle with the switches to delay the card so the laptop doesn't hang during BIOS boot.

are you using the 'nvidia gtx 660 ti' or 'nvidia gtx 660' or something else?

what i'm curious about is why i get the 'nvidia gtx 660' working but not the 'nvidia gtx 660 ti'. both have 2gb of ram.

maybe there is some white/blacklisting of pcie IDs in the kernel somewhere and the 'nvidia gtx 660 ti' didn't make it there yet?

all i get to see is stuff like:

(quote from an earlier attempt to get the 'nvidia gtx 660 ti' to work):

[  966.575224] vgaarb: device added:
PCI:0000:04:00.0,decodes=io+mem,owns=none,locks=none
[  966.575227] vgaarb: device changed decodes:
PCI:0000:00:02.0,olddecodes=io+mem,decodes=none:owns=none
[  966.575327] pci 0000:04:00.1: [10de:0e0a] type 00 class 0x040300
[  966.575356] pci 0000:04:00.1: reg 0x10: [mem 0x00000000-0x00003fff]
[  966.998345] pci 0000:04:00.0: BAR 1: no space for [mem size
0x08000000 64bit pref]
[  966.998350] pci 0000:04:00.0: BAR 1: failed to assign [mem size
0x08000000 64bit pref]
[  966.998353] pci 0000:04:00.0: BAR 3: no space for [mem size
0x02000000 64bit pref]
[  966.998355] pci 0000:04:00.0: BAR 3: failed to assign [mem size
0x02000000 64bit pref]
[  966.998357] pci 0000:04:00.0: BAR 0: no space for [mem size 0x01000000]
[  966.998358] pci 0000:04:00.0: BAR 0: failed to assign [mem size
0x01000000]
[  966.998360] pci 0000:04:00.0: BAR 6: assigned [mem
0xf1400000-0xf147ffff pref]
[  966.998363] pci 0000:04:00.1: BAR 0: assigned [mem 0xf1480000-0xf1483fff]
[  966.998369] pci 0000:04:00.1: BAR 0: error updating (0xf1480000 !=
0xffffffff)
[  966.998373] pci 0000:04:00.0: BAR 5: assigned [io  0x3000-0x307f]
[  966.998379] pci 0000:04:00.0: BAR 5: error updating (0x003001 !=

0xffffffff)

Quoted 4 lines

I pass pci=nocrs,realloc on the kernel commandline in order to get the GPU programmed. I've only actually tested it with KVM/VFIO GPU assignment and it seems to work well, or at least as well as can be expected over a x1 5GT/s link. Thanks,

i've been playing with pci=nocrs,realloc a lot lately but that didn't make it work.

Quoted 157 lines

Alex

Quoted 152 lines · Joachim Schiele
Quoted 136 lines

hi,

i've run with pci=realloc two tests, the results are here (also the dmesg) outputs:

https://lastlog.de/wiki/index.php/T530%2BNvida_GTX_660_TI_eGPU_via_PE4H#debugging_using_pci.3Drealloc

thanks in advance,
joachim schiele

Quoted 124 lines · Bjorn Helgaas

[+cc Alex]

Hi Joachim,

Thanks a lot for the bug report.

Quoted 8 lines · Joachim Schiele

hi,

=== question ===
with the setup described below, i can't pass the graphics adapter into a KVM guest as described below.

is the BAR stuff in the listing an error i wuld have to fix in order to get my eGPU running? if so, what could i do about it

Yes. If we can't assign resources to the BARs, we can't enable the device. If you boot with "pci=realloc", Linux will try harder to assign space (including moving other devices to make space if necessary).

Quoted 2 lines

i've seen that the
laptop will hang on boot if the eGPU is powered on system startup.

It's possible the BIOS is hanging because it tried to assign space for the eGPU but failed.

Quoted 3 lines

i can
see the adapter after hotplugging it using lspci and i never see an image on the external display.

The device will appear in lspci even if we can't assign space for it. In that case the device would not work, even though it appears in lspci.

So please try "pci=realloc". Also, please collect a complete dmesg log and "lspci -vv" output (with eGPU attached). That will contain the information needed to see whether it's possible even in principle to make this work.

Quoted 84 lines

my config is this:
sudo qemu-system-x86_64 -enable-kvm \

(quoted diff) +0 -7
-m 8192 \
-M q35 \
-cpu host,kvm=off \
-smp 4,sockets=1,cores=4,threads=1 \
-vga none \
-vnc none \
-device

ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=04:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \

(quoted diff) +0 -2
-device vfio-pci,host=04:00.1,bus=root.1,addr=00.1 \
-drive

file=/home/joachim/en_windows_7_professional_x64_dvd_X15-65805.iso,id=isocd

(quoted diff) +0 -5
-device ide-cd,bus=ide.1,drive=isocd \
-drive file=/var/lib/libvirt/images/win7.qcow2,format=qcow2,id=hdd
-device ide-hd,bus=ide.0,drive=hdd \
-bios /usr/share/seabios/bios.bin \
-soundhw ac97

=== when hotplugging the eGPU ===
i want to use a Nvidia GTX 660 TI as an eGPU using PE4H and when i hotplug the ePCI adapter i get this log on a 3.18 ubuntu kernel:

[  960.145002] vgaarb: device changed decodes:
PCI:0000:00:02.0,olddecodes=none,decodes=io+mem:owns=none
[  966.574858] pci 0000:04:00.0: [10de:1183] type 00 class 0x030000
[  966.574892] pci 0000:04:00.0: reg 0x10: [mem 0x00000000-0x00ffffff]
[  966.574921] pci 0000:04:00.0: reg 0x14: [mem 0x00000000-0x07ffffff
64bit pref]
[  966.574950] pci 0000:04:00.0: reg 0x1c: [mem 0x00000000-0x01ffffff
64bit pref]
[  966.574963] pci 0000:04:00.0: reg 0x24: [io  0x0000-0x007f]
[  966.574974] pci 0000:04:00.0: reg 0x30: [mem 0x00000000-0x0007ffff pref]
[  966.575224] vgaarb: device added:
PCI:0000:04:00.0,decodes=io+mem,owns=none,locks=none
[  966.575227] vgaarb: device changed decodes:
PCI:0000:00:02.0,olddecodes=io+mem,decodes=none:owns=none
[  966.575327] pci 0000:04:00.1: [10de:0e0a] type 00 class 0x040300
[  966.575356] pci 0000:04:00.1: reg 0x10: [mem 0x00000000-0x00003fff]
[  966.998345] pci 0000:04:00.0: BAR 1: no space for [mem size
0x08000000 64bit pref]
[  966.998350] pci 0000:04:00.0: BAR 1: failed to assign [mem size
0x08000000 64bit pref]
[  966.998353] pci 0000:04:00.0: BAR 3: no space for [mem size
0x02000000 64bit pref]
[  966.998355] pci 0000:04:00.0: BAR 3: failed to assign [mem size
0x02000000 64bit pref]
[  966.998357] pci 0000:04:00.0: BAR 0: no space for [mem size 0x01000000]
[  966.998358] pci 0000:04:00.0: BAR 0: failed to assign [mem size
0x01000000]
[  966.998360] pci 0000:04:00.0: BAR 6: assigned [mem
0xf1400000-0xf147ffff pref]
[  966.998363] pci 0000:04:00.1: BAR 0: assigned [mem 0xf1480000-0xf1483fff]
[  966.998369] pci 0000:04:00.1: BAR 0: error updating (0xf1480000 !=
0xffffffff)
[  966.998373] pci 0000:04:00.0: BAR 5: assigned [io  0x3000-0x307f]
[  966.998379] pci 0000:04:00.0: BAR 5: error updating (0x003001 !=
0xffffffff)
[  966.998392] pcieport 0000:00:1c.2: Max Payload Size set to  128/ 128
(was  128), Max Read Rq  128
[  966.998403] pci 0000:04:00.0: Max Payload Size set to 16384/ 256 (was
16384), Max Read Rq 16384
[  966.998414] pci 0000:04:00.1: Max Payload Size set to 16384/ 256 (was
16384), Max Read Rq 16384
[  966.998472] pci-stub 0000:04:00.0: claimed by stub
[  966.998495] pci-stub 0000:04:00.1: claimed by stub
[  967.099040] pcieport 0000:00:1c.2: Max Payload Size set to  128/ 128
(was  128), Max Read Rq  128
[  967.099059] pci-stub 0000:04:00.0: Max Payload Size set to  128/ 256
(was  128), Max Read Rq  512
[  967.099075] pci-stub 0000:04:00.1: Max Payload Size set to  128/ 256
(was  128), Max Read Rq  512
[ 7735.938854] vgaarb: device changed decodes:

PCI:0000:00:02.0,olddecodes=none,decodes=io+mem:owns=none

more at:

https://lastlog.de/wiki/index.php/T530%2BNvida_GTX_660_TI_eGPU_via_PE4H

Quoted 9 lines
Quoted 7 lines
Quoted 7 lines

i derived the scripts i'm using from this guide:
http://www.pugetsystems.com/labs/articles/Multiheaded-NVIDIA-Gaming-using-Ubuntu-14-04-KVM-585/

--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#7 lore raw
Quoted 53 lines · Joachim Schiele
Quoted 7 lines · Alex Williamson
Quoted 6 lines · Joachim Schiele

just wanted to report that i got the external graphics card to work with native windows 7 and setup 1.x + pci compaction!

see:
http://blog.lastlog.de/posts/setup-1.x/

Quoted 6 lines
Quoted 1 line

i hope to get it also working in linux host using kvm+win7 as guest soon.

I just setup something similar, using the EXP GDC "Beast" rather than the PE4H. My Lenovo T530 only leaves 390MB for MMIO space and doesn't provide any 64bit space via ACPI. I also need to fiddle with the switches to delay the card so the laptop doesn't hang during BIOS boot.

are you using the 'nvidia gtx 660 ti' or 'nvidia gtx 660' or something else?

what i'm curious about is why i get the 'nvidia gtx 660' working but not the 'nvidia gtx 660 ti'. both have 2gb of ram.

maybe there is some white/blacklisting of pcie IDs in the kernel somewhere and the 'nvidia gtx 660 ti' didn't make it there yet?

all i get to see is stuff like:

(quote from an earlier attempt to get the 'nvidia gtx 660 ti' to work):

[  966.575224] vgaarb: device added:
PCI:0000:04:00.0,decodes=io+mem,owns=none,locks=none
[  966.575227] vgaarb: device changed decodes:
PCI:0000:00:02.0,olddecodes=io+mem,decodes=none:owns=none
[  966.575327] pci 0000:04:00.1: [10de:0e0a] type 00 class 0x040300
[  966.575356] pci 0000:04:00.1: reg 0x10: [mem 0x00000000-0x00003fff]
[  966.998345] pci 0000:04:00.0: BAR 1: no space for [mem size
0x08000000 64bit pref]
[  966.998350] pci 0000:04:00.0: BAR 1: failed to assign [mem size
0x08000000 64bit pref]
[  966.998353] pci 0000:04:00.0: BAR 3: no space for [mem size
0x02000000 64bit pref]
[  966.998355] pci 0000:04:00.0: BAR 3: failed to assign [mem size
0x02000000 64bit pref]
[  966.998357] pci 0000:04:00.0: BAR 0: no space for [mem size 0x01000000]
[  966.998358] pci 0000:04:00.0: BAR 0: failed to assign [mem size
0x01000000]
[  966.998360] pci 0000:04:00.0: BAR 6: assigned [mem
0xf1400000-0xf147ffff pref]
[  966.998363] pci 0000:04:00.1: BAR 0: assigned [mem 0xf1480000-0xf1483fff]
[  966.998369] pci 0000:04:00.1: BAR 0: error updating (0xf1480000 !=
0xffffffff)
[  966.998373] pci 0000:04:00.0: BAR 5: assigned [io  0x3000-0x307f]
[  966.998379] pci 0000:04:00.0: BAR 5: error updating (0x003001 !=

0xffffffff)

Quoted 7 lines
Quoted 4 lines

I pass pci=nocrs,realloc on the kernel commandline in order to get the GPU programmed. I've only actually tested it with KVM/VFIO GPU assignment and it seems to work well, or at least as well as can be expected over a x1 5GT/s link. Thanks,

i've been playing with pci=nocrs,realloc a lot lately but that didn't make it work.

You need to boot with the card attached, perhaps even stalling at the grub boot loader to make sure you've passed the delay configured by the onboard switches that hide the card past bios boot. Your dmesg timestamps above seem to imply you're trying to hot-plug the card later, but there's not sufficient space for it.

I tested with a lowly GT 635, but I guarantee there are no white/blacklist in the kernel for basic resource assignment of devices. Thanks,

Alex

Quoted 161 lines
Quoted 148 lines
Quoted 148 lines · Joachim Schiele
Quoted 145 lines

hi,

i've run with pci=realloc two tests, the results are here (also the dmesg) outputs:

https://lastlog.de/wiki/index.php/T530%2BNvida_GTX_660_TI_eGPU_via_PE4H#debugging_using_pci.3Drealloc

thanks in advance,
joachim schiele

Quoted 131 lines · Bjorn Helgaas

[+cc Alex]

Hi Joachim,

Thanks a lot for the bug report.

Quoted 8 lines · Joachim Schiele

hi,

=== question ===
with the setup described below, i can't pass the graphics adapter into a KVM guest as described below.

is the BAR stuff in the listing an error i wuld have to fix in order to get my eGPU running? if so, what could i do about it

Yes. If we can't assign resources to the BARs, we can't enable the device. If you boot with "pci=realloc", Linux will try harder to assign space (including moving other devices to make space if necessary).

Quoted 2 lines

i've seen that the
laptop will hang on boot if the eGPU is powered on system startup.

It's possible the BIOS is hanging because it tried to assign space for the eGPU but failed.

Quoted 3 lines

i can
see the adapter after hotplugging it using lspci and i never see an image on the external display.

The device will appear in lspci even if we can't assign space for it. In that case the device would not work, even though it appears in lspci.

So please try "pci=realloc". Also, please collect a complete dmesg log and "lspci -vv" output (with eGPU attached). That will contain the information needed to see whether it's possible even in principle to make this work.

Quoted 91 lines

my config is this:
sudo qemu-system-x86_64 -enable-kvm \

(quoted diff) +0 -7
-m 8192 \
-M q35 \
-cpu host,kvm=off \
-smp 4,sockets=1,cores=4,threads=1 \
-vga none \
-vnc none \
-device

ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=04:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \

(quoted diff) +0 -2
-device vfio-pci,host=04:00.1,bus=root.1,addr=00.1 \
-drive

file=/home/joachim/en_windows_7_professional_x64_dvd_X15-65805.iso,id=isocd

(quoted diff) +0 -5
-device ide-cd,bus=ide.1,drive=isocd \
-drive file=/var/lib/libvirt/images/win7.qcow2,format=qcow2,id=hdd
-device ide-hd,bus=ide.0,drive=hdd \
-bios /usr/share/seabios/bios.bin \
-soundhw ac97

=== when hotplugging the eGPU ===
i want to use a Nvidia GTX 660 TI as an eGPU using PE4H and when i hotplug the ePCI adapter i get this log on a 3.18 ubuntu kernel:

[  960.145002] vgaarb: device changed decodes:
PCI:0000:00:02.0,olddecodes=none,decodes=io+mem:owns=none
[  966.574858] pci 0000:04:00.0: [10de:1183] type 00 class 0x030000
[  966.574892] pci 0000:04:00.0: reg 0x10: [mem 0x00000000-0x00ffffff]
[  966.574921] pci 0000:04:00.0: reg 0x14: [mem 0x00000000-0x07ffffff
64bit pref]
[  966.574950] pci 0000:04:00.0: reg 0x1c: [mem 0x00000000-0x01ffffff
64bit pref]
[  966.574963] pci 0000:04:00.0: reg 0x24: [io  0x0000-0x007f]
[  966.574974] pci 0000:04:00.0: reg 0x30: [mem 0x00000000-0x0007ffff pref]
[  966.575224] vgaarb: device added:
PCI:0000:04:00.0,decodes=io+mem,owns=none,locks=none
[  966.575227] vgaarb: device changed decodes:
PCI:0000:00:02.0,olddecodes=io+mem,decodes=none:owns=none
[  966.575327] pci 0000:04:00.1: [10de:0e0a] type 00 class 0x040300
[  966.575356] pci 0000:04:00.1: reg 0x10: [mem 0x00000000-0x00003fff]
[  966.998345] pci 0000:04:00.0: BAR 1: no space for [mem size
0x08000000 64bit pref]
[  966.998350] pci 0000:04:00.0: BAR 1: failed to assign [mem size
0x08000000 64bit pref]
[  966.998353] pci 0000:04:00.0: BAR 3: no space for [mem size
0x02000000 64bit pref]
[  966.998355] pci 0000:04:00.0: BAR 3: failed to assign [mem size
0x02000000 64bit pref]
[  966.998357] pci 0000:04:00.0: BAR 0: no space for [mem size 0x01000000]
[  966.998358] pci 0000:04:00.0: BAR 0: failed to assign [mem size
0x01000000]
[  966.998360] pci 0000:04:00.0: BAR 6: assigned [mem
0xf1400000-0xf147ffff pref]
[  966.998363] pci 0000:04:00.1: BAR 0: assigned [mem 0xf1480000-0xf1483fff]
[  966.998369] pci 0000:04:00.1: BAR 0: error updating (0xf1480000 !=
0xffffffff)
[  966.998373] pci 0000:04:00.0: BAR 5: assigned [io  0x3000-0x307f]
[  966.998379] pci 0000:04:00.0: BAR 5: error updating (0x003001 !=
0xffffffff)
[  966.998392] pcieport 0000:00:1c.2: Max Payload Size set to  128/ 128
(was  128), Max Read Rq  128
[  966.998403] pci 0000:04:00.0: Max Payload Size set to 16384/ 256 (was
16384), Max Read Rq 16384
[  966.998414] pci 0000:04:00.1: Max Payload Size set to 16384/ 256 (was
16384), Max Read Rq 16384
[  966.998472] pci-stub 0000:04:00.0: claimed by stub
[  966.998495] pci-stub 0000:04:00.1: claimed by stub
[  967.099040] pcieport 0000:00:1c.2: Max Payload Size set to  128/ 128
(was  128), Max Read Rq  128
[  967.099059] pci-stub 0000:04:00.0: Max Payload Size set to  128/ 256
(was  128), Max Read Rq  512
[  967.099075] pci-stub 0000:04:00.1: Max Payload Size set to  128/ 256
(was  128), Max Read Rq  512
[ 7735.938854] vgaarb: device changed decodes:

PCI:0000:00:02.0,olddecodes=none,decodes=io+mem:owns=none

more at:

https://lastlog.de/wiki/index.php/T530%2BNvida_GTX_660_TI_eGPU_via_PE4H

i derived the scripts i'm using from this guide:
http://www.pugetsystems.com/labs/articles/Multiheaded-NVIDIA-Gaming-using-Ubuntu-14-04-KVM-585/

--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 7 lines
Quoted 4 lines

--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

author
#8 lore raw
Quoted 62 lines · Alex Williamson
Quoted 53 lines · Joachim Schiele
Quoted 7 lines · Alex Williamson
Quoted 6 lines · Joachim Schiele

just wanted to report that i got the external graphics card to work with native windows 7 and setup 1.x + pci compaction!

see:
http://blog.lastlog.de/posts/setup-1.x/

Quoted 6 lines
Quoted 1 line

i hope to get it also working in linux host using kvm+win7 as guest soon.

I just setup something similar, using the EXP GDC "Beast" rather than the PE4H. My Lenovo T530 only leaves 390MB for MMIO space and doesn't provide any 64bit space via ACPI. I also need to fiddle with the switches to delay the card so the laptop doesn't hang during BIOS boot.

are you using the 'nvidia gtx 660 ti' or 'nvidia gtx 660' or something else?

what i'm curious about is why i get the 'nvidia gtx 660' working but not the 'nvidia gtx 660 ti'. both have 2gb of ram.

maybe there is some white/blacklisting of pcie IDs in the kernel somewhere and the 'nvidia gtx 660 ti' didn't make it there yet?

all i get to see is stuff like:

(quote from an earlier attempt to get the 'nvidia gtx 660 ti' to work):

[  966.575224] vgaarb: device added:
PCI:0000:04:00.0,decodes=io+mem,owns=none,locks=none
[  966.575227] vgaarb: device changed decodes:
PCI:0000:00:02.0,olddecodes=io+mem,decodes=none:owns=none
[  966.575327] pci 0000:04:00.1: [10de:0e0a] type 00 class 0x040300
[  966.575356] pci 0000:04:00.1: reg 0x10: [mem 0x00000000-0x00003fff]
[  966.998345] pci 0000:04:00.0: BAR 1: no space for [mem size
0x08000000 64bit pref]
[  966.998350] pci 0000:04:00.0: BAR 1: failed to assign [mem size
0x08000000 64bit pref]
[  966.998353] pci 0000:04:00.0: BAR 3: no space for [mem size
0x02000000 64bit pref]
[  966.998355] pci 0000:04:00.0: BAR 3: failed to assign [mem size
0x02000000 64bit pref]
[  966.998357] pci 0000:04:00.0: BAR 0: no space for [mem size 0x01000000]
[  966.998358] pci 0000:04:00.0: BAR 0: failed to assign [mem size
0x01000000]
[  966.998360] pci 0000:04:00.0: BAR 6: assigned [mem
0xf1400000-0xf147ffff pref]
[  966.998363] pci 0000:04:00.1: BAR 0: assigned [mem 0xf1480000-0xf1483fff]
[  966.998369] pci 0000:04:00.1: BAR 0: error updating (0xf1480000 !=
0xffffffff)
[  966.998373] pci 0000:04:00.0: BAR 5: assigned [io  0x3000-0x307f]
[  966.998379] pci 0000:04:00.0: BAR 5: error updating (0x003001 !=

0xffffffff)

Quoted 7 lines
Quoted 4 lines

I pass pci=nocrs,realloc on the kernel commandline in order to get the GPU programmed. I've only actually tested it with KVM/VFIO GPU assignment and it seems to work well, or at least as well as can be expected over a x1 5GT/s link. Thanks,

i've been playing with pci=nocrs,realloc a lot lately but that didn't make it work.

Quoted 7 lines

You need to boot with the card attached, perhaps even stalling at the grub boot loader to make sure you've passed the delay configured by the onboard switches that hide the card past bios boot. Your dmesg timestamps above seem to imply you're trying to hot-plug the card later, but there's not sufficient space for it.

I tested with a lowly GT 635,

good to know

Quoted 3 lines

but I guarantee there are no
white/blacklist in the kernel for basic resource assignment of devices. Thanks,

ah fine. then the problem is probably card/bios specific

Quoted 164 lines
Quoted 161 lines
Quoted 148 lines
Quoted 148 lines · Joachim Schiele
Quoted 145 lines

hi,

i've run with pci=realloc two tests, the results are here (also the dmesg) outputs:

https://lastlog.de/wiki/index.php/T530%2BNvida_GTX_660_TI_eGPU_via_PE4H#debugging_using_pci.3Drealloc

thanks in advance,
joachim schiele

Quoted 131 lines · Bjorn Helgaas

[+cc Alex]

Hi Joachim,

Thanks a lot for the bug report.

Quoted 8 lines · Joachim Schiele

hi,

=== question ===
with the setup described below, i can't pass the graphics adapter into a KVM guest as described below.

is the BAR stuff in the listing an error i wuld have to fix in order to get my eGPU running? if so, what could i do about it

Yes. If we can't assign resources to the BARs, we can't enable the device. If you boot with "pci=realloc", Linux will try harder to assign space (including moving other devices to make space if necessary).

Quoted 2 lines

i've seen that the
laptop will hang on boot if the eGPU is powered on system startup.

It's possible the BIOS is hanging because it tried to assign space for the eGPU but failed.

Quoted 3 lines

i can
see the adapter after hotplugging it using lspci and i never see an image on the external display.

The device will appear in lspci even if we can't assign space for it. In that case the device would not work, even though it appears in lspci.

So please try "pci=realloc". Also, please collect a complete dmesg log and "lspci -vv" output (with eGPU attached). That will contain the information needed to see whether it's possible even in principle to make this work.

Quoted 91 lines

my config is this:
sudo qemu-system-x86_64 -enable-kvm \

(quoted diff) +0 -7
-m 8192 \
-M q35 \
-cpu host,kvm=off \
-smp 4,sockets=1,cores=4,threads=1 \
-vga none \
-vnc none \
-device

ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=04:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \

(quoted diff) +0 -2
-device vfio-pci,host=04:00.1,bus=root.1,addr=00.1 \
-drive

file=/home/joachim/en_windows_7_professional_x64_dvd_X15-65805.iso,id=isocd

(quoted diff) +0 -5
-device ide-cd,bus=ide.1,drive=isocd \
-drive file=/var/lib/libvirt/images/win7.qcow2,format=qcow2,id=hdd
-device ide-hd,bus=ide.0,drive=hdd \
-bios /usr/share/seabios/bios.bin \
-soundhw ac97

=== when hotplugging the eGPU ===
i want to use a Nvidia GTX 660 TI as an eGPU using PE4H and when i hotplug the ePCI adapter i get this log on a 3.18 ubuntu kernel:

[  960.145002] vgaarb: device changed decodes:
PCI:0000:00:02.0,olddecodes=none,decodes=io+mem:owns=none
[  966.574858] pci 0000:04:00.0: [10de:1183] type 00 class 0x030000
[  966.574892] pci 0000:04:00.0: reg 0x10: [mem 0x00000000-0x00ffffff]
[  966.574921] pci 0000:04:00.0: reg 0x14: [mem 0x00000000-0x07ffffff
64bit pref]
[  966.574950] pci 0000:04:00.0: reg 0x1c: [mem 0x00000000-0x01ffffff
64bit pref]
[  966.574963] pci 0000:04:00.0: reg 0x24: [io  0x0000-0x007f]
[  966.574974] pci 0000:04:00.0: reg 0x30: [mem 0x00000000-0x0007ffff pref]
[  966.575224] vgaarb: device added:
PCI:0000:04:00.0,decodes=io+mem,owns=none,locks=none
[  966.575227] vgaarb: device changed decodes:
PCI:0000:00:02.0,olddecodes=io+mem,decodes=none:owns=none
[  966.575327] pci 0000:04:00.1: [10de:0e0a] type 00 class 0x040300
[  966.575356] pci 0000:04:00.1: reg 0x10: [mem 0x00000000-0x00003fff]
[  966.998345] pci 0000:04:00.0: BAR 1: no space for [mem size
0x08000000 64bit pref]
[  966.998350] pci 0000:04:00.0: BAR 1: failed to assign [mem size
0x08000000 64bit pref]
[  966.998353] pci 0000:04:00.0: BAR 3: no space for [mem size
0x02000000 64bit pref]
[  966.998355] pci 0000:04:00.0: BAR 3: failed to assign [mem size
0x02000000 64bit pref]
[  966.998357] pci 0000:04:00.0: BAR 0: no space for [mem size 0x01000000]
[  966.998358] pci 0000:04:00.0: BAR 0: failed to assign [mem size
0x01000000]
[  966.998360] pci 0000:04:00.0: BAR 6: assigned [mem
0xf1400000-0xf147ffff pref]
[  966.998363] pci 0000:04:00.1: BAR 0: assigned [mem 0xf1480000-0xf1483fff]
[  966.998369] pci 0000:04:00.1: BAR 0: error updating (0xf1480000 !=
0xffffffff)
[  966.998373] pci 0000:04:00.0: BAR 5: assigned [io  0x3000-0x307f]
[  966.998379] pci 0000:04:00.0: BAR 5: error updating (0x003001 !=
0xffffffff)
[  966.998392] pcieport 0000:00:1c.2: Max Payload Size set to  128/ 128
(was  128), Max Read Rq  128
[  966.998403] pci 0000:04:00.0: Max Payload Size set to 16384/ 256 (was
16384), Max Read Rq 16384
[  966.998414] pci 0000:04:00.1: Max Payload Size set to 16384/ 256 (was
16384), Max Read Rq 16384
[  966.998472] pci-stub 0000:04:00.0: claimed by stub
[  966.998495] pci-stub 0000:04:00.1: claimed by stub
[  967.099040] pcieport 0000:00:1c.2: Max Payload Size set to  128/ 128
(was  128), Max Read Rq  128
[  967.099059] pci-stub 0000:04:00.0: Max Payload Size set to  128/ 256
(was  128), Max Read Rq  512
[  967.099075] pci-stub 0000:04:00.1: Max Payload Size set to  128/ 256
(was  128), Max Read Rq  512
[ 7735.938854] vgaarb: device changed decodes:

PCI:0000:00:02.0,olddecodes=none,decodes=io+mem:owns=none

more at:

https://lastlog.de/wiki/index.php/T530%2BNvida_GTX_660_TI_eGPU_via_PE4H

i derived the scripts i'm using from this guide:
http://www.pugetsystems.com/labs/articles/Multiheaded-NVIDIA-Gaming-using-Ubuntu-14-04-KVM-585/

--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 7 lines
Quoted 4 lines

--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 6 lines
Joachim Schiele

blog: http://blog.lastlog.de
wiki: http://lastlog.de
jabber: [email protected]
GPG: C6AC8770
author
#9 lore raw
In reply to Alex Williamson
Quoted 62 lines · Alex Williamson
Quoted 53 lines · Joachim Schiele
Quoted 7 lines · Alex Williamson
Quoted 6 lines · Joachim Schiele

just wanted to report that i got the external graphics card to work with native windows 7 and setup 1.x + pci compaction!

see:
http://blog.lastlog.de/posts/setup-1.x/

Quoted 6 lines
Quoted 1 line

i hope to get it also working in linux host using kvm+win7 as guest soon.

I just setup something similar, using the EXP GDC "Beast" rather than the PE4H. My Lenovo T530 only leaves 390MB for MMIO space and doesn't provide any 64bit space via ACPI. I also need to fiddle with the switches to delay the card so the laptop doesn't hang during BIOS boot.

are you using the 'nvidia gtx 660 ti' or 'nvidia gtx 660' or something else?

what i'm curious about is why i get the 'nvidia gtx 660' working but not the 'nvidia gtx 660 ti'. both have 2gb of ram.

maybe there is some white/blacklisting of pcie IDs in the kernel somewhere and the 'nvidia gtx 660 ti' didn't make it there yet?

all i get to see is stuff like:

(quote from an earlier attempt to get the 'nvidia gtx 660 ti' to work):

[  966.575224] vgaarb: device added:
PCI:0000:04:00.0,decodes=io+mem,owns=none,locks=none
[  966.575227] vgaarb: device changed decodes:
PCI:0000:00:02.0,olddecodes=io+mem,decodes=none:owns=none
[  966.575327] pci 0000:04:00.1: [10de:0e0a] type 00 class 0x040300
[  966.575356] pci 0000:04:00.1: reg 0x10: [mem 0x00000000-0x00003fff]
[  966.998345] pci 0000:04:00.0: BAR 1: no space for [mem size
0x08000000 64bit pref]
[  966.998350] pci 0000:04:00.0: BAR 1: failed to assign [mem size
0x08000000 64bit pref]
[  966.998353] pci 0000:04:00.0: BAR 3: no space for [mem size
0x02000000 64bit pref]
[  966.998355] pci 0000:04:00.0: BAR 3: failed to assign [mem size
0x02000000 64bit pref]
[  966.998357] pci 0000:04:00.0: BAR 0: no space for [mem size 0x01000000]
[  966.998358] pci 0000:04:00.0: BAR 0: failed to assign [mem size
0x01000000]
[  966.998360] pci 0000:04:00.0: BAR 6: assigned [mem
0xf1400000-0xf147ffff pref]
[  966.998363] pci 0000:04:00.1: BAR 0: assigned [mem 0xf1480000-0xf1483fff]
[  966.998369] pci 0000:04:00.1: BAR 0: error updating (0xf1480000 !=
0xffffffff)
[  966.998373] pci 0000:04:00.0: BAR 5: assigned [io  0x3000-0x307f]
[  966.998379] pci 0000:04:00.0: BAR 5: error updating (0x003001 !=

0xffffffff)

Quoted 7 lines
Quoted 4 lines

I pass pci=nocrs,realloc on the kernel commandline in order to get the GPU programmed. I've only actually tested it with KVM/VFIO GPU assignment and it seems to work well, or at least as well as can be expected over a x1 5GT/s link. Thanks,

i've been playing with pci=nocrs,realloc a lot lately but that didn't make it work.

Quoted 8 lines

You need to boot with the card attached, perhaps even stalling at the grub boot loader to make sure you've passed the delay configured by the onboard switches that hide the card past bios boot. Your dmesg timestamps above seem to imply you're trying to hot-plug the card later, but there's not sufficient space for it.

I tested with a lowly GT 635, but I guarantee there are no white/blacklist in the kernel for basic resource assignment of devices.

i've now also tested a gtx 680 (2gb ram also) with no success either using KVM (win7 crashed when installing the driver in the KVM instance). does not work either with my native win7 which has the patched DSDT and also by trying it with various PCI compaction attemts using setup 1.x

just for the record.

Quoted 164 lines
Quoted 161 lines
Quoted 148 lines
Quoted 148 lines · Joachim Schiele
Quoted 145 lines

hi,

i've run with pci=realloc two tests, the results are here (also the dmesg) outputs:

https://lastlog.de/wiki/index.php/T530%2BNvida_GTX_660_TI_eGPU_via_PE4H#debugging_using_pci.3Drealloc

thanks in advance,
joachim schiele

Quoted 131 lines · Bjorn Helgaas

[+cc Alex]

Hi Joachim,

Thanks a lot for the bug report.

Quoted 8 lines · Joachim Schiele

hi,

=== question ===
with the setup described below, i can't pass the graphics adapter into a KVM guest as described below.

is the BAR stuff in the listing an error i wuld have to fix in order to get my eGPU running? if so, what could i do about it

Yes. If we can't assign resources to the BARs, we can't enable the device. If you boot with "pci=realloc", Linux will try harder to assign space (including moving other devices to make space if necessary).

Quoted 2 lines

i've seen that the
laptop will hang on boot if the eGPU is powered on system startup.

It's possible the BIOS is hanging because it tried to assign space for the eGPU but failed.

Quoted 3 lines

i can
see the adapter after hotplugging it using lspci and i never see an image on the external display.

The device will appear in lspci even if we can't assign space for it. In that case the device would not work, even though it appears in lspci.

So please try "pci=realloc". Also, please collect a complete dmesg log and "lspci -vv" output (with eGPU attached). That will contain the information needed to see whether it's possible even in principle to make this work.

Quoted 91 lines

my config is this:
sudo qemu-system-x86_64 -enable-kvm \

(quoted diff) +0 -7
-m 8192 \
-M q35 \
-cpu host,kvm=off \
-smp 4,sockets=1,cores=4,threads=1 \
-vga none \
-vnc none \
-device

ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=04:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \

(quoted diff) +0 -2
-device vfio-pci,host=04:00.1,bus=root.1,addr=00.1 \
-drive

file=/home/joachim/en_windows_7_professional_x64_dvd_X15-65805.iso,id=isocd

(quoted diff) +0 -5
-device ide-cd,bus=ide.1,drive=isocd \
-drive file=/var/lib/libvirt/images/win7.qcow2,format=qcow2,id=hdd
-device ide-hd,bus=ide.0,drive=hdd \
-bios /usr/share/seabios/bios.bin \
-soundhw ac97

=== when hotplugging the eGPU ===
i want to use a Nvidia GTX 660 TI as an eGPU using PE4H and when i hotplug the ePCI adapter i get this log on a 3.18 ubuntu kernel:

[  960.145002] vgaarb: device changed decodes:
PCI:0000:00:02.0,olddecodes=none,decodes=io+mem:owns=none
[  966.574858] pci 0000:04:00.0: [10de:1183] type 00 class 0x030000
[  966.574892] pci 0000:04:00.0: reg 0x10: [mem 0x00000000-0x00ffffff]
[  966.574921] pci 0000:04:00.0: reg 0x14: [mem 0x00000000-0x07ffffff
64bit pref]
[  966.574950] pci 0000:04:00.0: reg 0x1c: [mem 0x00000000-0x01ffffff
64bit pref]
[  966.574963] pci 0000:04:00.0: reg 0x24: [io  0x0000-0x007f]
[  966.574974] pci 0000:04:00.0: reg 0x30: [mem 0x00000000-0x0007ffff pref]
[  966.575224] vgaarb: device added:
PCI:0000:04:00.0,decodes=io+mem,owns=none,locks=none
[  966.575227] vgaarb: device changed decodes:
PCI:0000:00:02.0,olddecodes=io+mem,decodes=none:owns=none
[  966.575327] pci 0000:04:00.1: [10de:0e0a] type 00 class 0x040300
[  966.575356] pci 0000:04:00.1: reg 0x10: [mem 0x00000000-0x00003fff]
[  966.998345] pci 0000:04:00.0: BAR 1: no space for [mem size
0x08000000 64bit pref]
[  966.998350] pci 0000:04:00.0: BAR 1: failed to assign [mem size
0x08000000 64bit pref]
[  966.998353] pci 0000:04:00.0: BAR 3: no space for [mem size
0x02000000 64bit pref]
[  966.998355] pci 0000:04:00.0: BAR 3: failed to assign [mem size
0x02000000 64bit pref]
[  966.998357] pci 0000:04:00.0: BAR 0: no space for [mem size 0x01000000]
[  966.998358] pci 0000:04:00.0: BAR 0: failed to assign [mem size
0x01000000]
[  966.998360] pci 0000:04:00.0: BAR 6: assigned [mem
0xf1400000-0xf147ffff pref]
[  966.998363] pci 0000:04:00.1: BAR 0: assigned [mem 0xf1480000-0xf1483fff]
[  966.998369] pci 0000:04:00.1: BAR 0: error updating (0xf1480000 !=
0xffffffff)
[  966.998373] pci 0000:04:00.0: BAR 5: assigned [io  0x3000-0x307f]
[  966.998379] pci 0000:04:00.0: BAR 5: error updating (0x003001 !=
0xffffffff)
[  966.998392] pcieport 0000:00:1c.2: Max Payload Size set to  128/ 128
(was  128), Max Read Rq  128
[  966.998403] pci 0000:04:00.0: Max Payload Size set to 16384/ 256 (was
16384), Max Read Rq 16384
[  966.998414] pci 0000:04:00.1: Max Payload Size set to 16384/ 256 (was
16384), Max Read Rq 16384
[  966.998472] pci-stub 0000:04:00.0: claimed by stub
[  966.998495] pci-stub 0000:04:00.1: claimed by stub
[  967.099040] pcieport 0000:00:1c.2: Max Payload Size set to  128/ 128
(was  128), Max Read Rq  128
[  967.099059] pci-stub 0000:04:00.0: Max Payload Size set to  128/ 256
(was  128), Max Read Rq  512
[  967.099075] pci-stub 0000:04:00.1: Max Payload Size set to  128/ 256
(was  128), Max Read Rq  512
[ 7735.938854] vgaarb: device changed decodes:

PCI:0000:00:02.0,olddecodes=none,decodes=io+mem:owns=none

more at:

https://lastlog.de/wiki/index.php/T530%2BNvida_GTX_660_TI_eGPU_via_PE4H

i derived the scripts i'm using from this guide:
http://www.pugetsystems.com/labs/articles/Multiheaded-NVIDIA-Gaming-using-Ubuntu-14-04-KVM-585/

--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 7 lines
Quoted 4 lines

--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html