Skip to contentSkip to conversation

PCI rescan issue

DISCUSSION Jon Baker 3 participants 5 messages Jun 19, 2014 – Jun 20, 2014 2 messages not in the archive PCI

lore
Share & tools
lore

Jon Baker

author
#0 lore raw

I am trying to find a solution to a Linux PCI rescan problem.

I have CentOS 6.5

[jbaker@server0 ~]$ uname -a
Linux server0 2.6.32-431.5.1.el6.x86_64 #1 SMP Wed Feb 12 00:41:43
UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

I have an Altera FPGA PCIe protoboard I am trying to test in a TYAN FT77A-B7059.

We are trying to solve some hotplug/FPGA reload issues; when we reload the FPGA the PCI interface reloads confusing linux pci code.

After bootup the board is seen by the kernel; lscpi shows Altera FPGA PCIe protoboard settings and config space. We can load our driver and all is well.

When we reload the FPGA lscpi display shows all "ff" for config space. As root I tried

echo 1 > /sys/bus/pci/rescan

But no change to lspci output.

I tried some variants

    echo 1 > /sys/bus/pci/devices/0000:8a:00.0/rescan

no change to lspci output.

echo 1 > /sys/bus/pci/devices/0000:8a:00.0/remove echo 1 > /sys/bus/pci/rescan

This removed the device and it did not show up again in lscpi.

Kernel has PCI HOTPLUG enabled:

    [jbaker@server0 ~]$ cat /boot/config-2.6.32-431.el6.x86_64 | grep

HOTPLUG
CONFIG_HOTPLUG=y
CONFIG_MEMORY_HOTPLUG=y
CONFIG_MEMORY_HOTPLUG_SPARSE=y
CONFIG_HOTPLUG_CPU=y
CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
CONFIG_ACPI_HOTPLUG_CPU=y
CONFIG_ACPI_HOTPLUG_MEMORY=y
CONFIG_ACPI_HOTPLUG_MEMORY_AUTO_ONLINE=y
CONFIG_HOTPLUG_PCI_PCIE=y
CONFIG_HOTPLUG_PCI=y
CONFIG_HOTPLUG_PCI_FAKE=m
CONFIG_HOTPLUG_PCI_ACPI=y
CONFIG_HOTPLUG_PCI_ACPI_IBM=m
# CONFIG_HOTPLUG_PCI_CPCI is not set
CONFIG_HOTPLUG_PCI_SHPC=m
[jbaker@server0 ~]$

The kernel has pciehp module loaded, I see pciehp messages in /var/log/messages.

In other distros, on other platforms, echo 1 > /sys/bus/pci/rescan has worked but does not appear to work here for CentOS 6.5 on this hardware.

This problem is much discussed on the web but I have yet to see a solution discussed. I am thinking the 2.6.32-431.5.1.el6.x86_64 kernel may require a patch(es). The rescan may not be resetting the PCIe card?

Any ideas how to get rescan to work?

Thank you,

Jon Baker

Signature · 9 lines
===================================
Jon Baker
Software Engineer
ViaSat Inc.
Cleveland OH
216-706-7800
[email protected]
www.viasat.com
===================================
#1 lore raw
Quoted 6 lines · Jon Baker

I am trying to find a solution to a Linux PCI rescan problem.

I have CentOS 6.5

[jbaker@server0 ~]$ uname -a
Linux server0 2.6.32-431.5.1.el6.x86_64 #1 SMP Wed Feb 12 00:41:43 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

Can you test it in the latest kernel?

Quoted 8 lines

I have an Altera FPGA PCIe protoboard I am trying to test in a TYAN FT77A-B7059.

We are trying to solve some hotplug/FPGA reload issues; when we reload the FPGA the PCI interface reloads confusing linux pci code.

After bootup the board is seen by the kernel; lscpi shows Altera FPGA PCIe protoboard settings and config space. We can load our driver and all is well.

When we reload the FPGA lscpi display shows all "ff" for config space. As root I tried

What happened during you reload the FPGA? reset PCI device or other anything?

Quoted 5 lines

echo 1 > /sys/bus/pci/rescan

But no change to lspci output.

Quoted 8 lines

I tried some variants

    echo 1 > /sys/bus/pci/devices/0000:8a:00.0/rescan

no change to lspci output.

echo 1 > /sys/bus/pci/devices/0000:8a:00.0/remove echo 1 > /sys/bus/pci/rescan

I guess the device link to your board was disconnect. Maybe the power to device is off or link error. So OS can not access pci device config space and return nothing found.

If you provide more detailed info is better. like lspci -vvvxxx, dmesg etc. before and after your operations.

Quoted 35 lines

This removed the device and it did not show up again in lscpi.

Kernel has PCI HOTPLUG enabled:

    [jbaker@server0 ~]$ cat /boot/config-2.6.32-431.el6.x86_64 | grep HOTPLUG
    CONFIG_HOTPLUG=y
    CONFIG_MEMORY_HOTPLUG=y
    CONFIG_MEMORY_HOTPLUG_SPARSE=y
    CONFIG_HOTPLUG_CPU=y
    CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
    CONFIG_ACPI_HOTPLUG_CPU=y
    CONFIG_ACPI_HOTPLUG_MEMORY=y
    CONFIG_ACPI_HOTPLUG_MEMORY_AUTO_ONLINE=y
    CONFIG_HOTPLUG_PCI_PCIE=y
    CONFIG_HOTPLUG_PCI=y
    CONFIG_HOTPLUG_PCI_FAKE=m
    CONFIG_HOTPLUG_PCI_ACPI=y
    CONFIG_HOTPLUG_PCI_ACPI_IBM=m
    # CONFIG_HOTPLUG_PCI_CPCI is not set
    CONFIG_HOTPLUG_PCI_SHPC=m
    [jbaker@server0 ~]$

The kernel has pciehp module loaded, I see pciehp messages in /var/log/messages.

In other distros, on other platforms, echo 1 > /sys/bus/pci/rescan has worked but does not appear to work here for CentOS 6.5 on this hardware.

This problem is much discussed on the web but I have yet to see a solution discussed. I am thinking the 2.6.32-431.5.1.el6.x86_64 kernel may require a patch(es). The rescan may not be resetting the PCIe card?

Any ideas how to get rescan to work?

Thank you,

Jon Baker

Signature · 2 lines
Thanks!
Yijing

A message that is not in the archive (sent off-list or not yet received): [email protected]

#3 lore raw
Quoted 15 lines · Jon Baker
Quoted 8 lines · Yijing Wang
Quoted 6 lines · Jon Baker

I am trying to find a solution to a Linux PCI rescan problem.

I have CentOS 6.5

[jbaker@server0 ~]$ uname -a
Linux server0 2.6.32-431.5.1.el6.x86_64 #1 SMP Wed Feb 12 00:41:43 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

Can you test it in the latest kernel?

Not easily, this is a system shared amongst several developers and containing some specific tools and applications. And unfortunately we know this works with a 3.X kernel on an embedded target used by another group at our company. - Jon

Quoted 3 lines
Quoted 3 lines

I have an Altera FPGA PCIe protoboard I am trying to test in a TYAN FT77A-B7059.

Quoted 7 lines
Quoted 6 lines
Quoted 5 lines

We are trying to solve some hotplug/FPGA reload issues; when we reload the FPGA the PCI interface reloads confusing linux pci code.

After bootup the board is seen by the kernel; lscpi shows Altera FPGA PCIe protoboard settings and config space. We can load our driver and all is well.

When we reload the FPGA lscpi display shows all "ff" for config space. As root I tried

What happened during you reload the FPGA? reset PCI device or other anything?

I do not know, we have not monitored this. The PCI interface is contained within the FPGA so I don't really know for sure how that behaves on reload. - Jon

If you find lspci PCI device return all ff after FPGA reload, and you do nothing during reload. I think the PCI device can not found after your FPAG reload, as I mentioned, maybe the PCI device power or link has problems. I think this is not a Linux problem.

You can try this:
1. echo 1 > /sys/bus/pci/devices/0000:8a:00.0/remove
2. Reload FPGA board.
3. echo 1 > /sys/bus/pci/rescan
4. lspci to show your PCI device

Here make your device removed before FPGA board reload, avoid some stall device info in OS.

or

You can go to /sys/bus/pci/slots/ to find whether your PCI device support hotplug. If yes, you can
echo 0 > /sys/bus/pci/slots/$device/power
echo 1 > /sys/bus/pci/slots/$device/power
reset the PCI device, then lspci show device whether is found.

Quoted 22 lines
Quoted 9 lines
Quoted 9 lines

echo 1 > /sys/bus/pci/rescan

But no change to lspci output.

I tried some variants

    echo 1 > /sys/bus/pci/devices/0000:8a:00.0/rescan
Quoted 8 lines
Quoted 4 lines

no change to lspci output.

echo 1 > /sys/bus/pci/devices/0000:8a:00.0/remove echo 1 > /sys/bus/pci/rescan

I guess the device link to your board was disconnect. Maybe the power to device is off or link error. So OS can not access pci device config space and return nothing found.

If you provide more detailed info is better. like lspci -vvvxxx, dmesg etc. before and after your operations.

I have logged this...

After system power up (case where everything works expected):

Quoted 13 lines
    $ lspci -vvv -xx -s 13:00.0
    13:00.0 Non-VGA unclassified device: Device 1b83:f002 (rev 01)
            Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
            Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
            Latency: 0, Cache Line Size: 64 bytes
            Interrupt: pin A routed to IRQ 11
            Region 0: Memory at dcc00000 (32-bit, non-prefetchable) [size=256K]
            Region 1: Memory at dcb00000 (32-bit, non-prefetchable) [size=1M]
            Capabilities: <access denied>
    00: 83 1b 02 f0 06 00 10 00 01 00 00 00 10 00 00 00
    10: 00 00 c0 dc 00 00 b0 dc 00 00 00 00 00 00 00 00
    20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
    30: 00 00 00 00 50 00 00 00 00 00 00 00 0b 01 00 00

Can you show the upstream port device lspci info ? If 13:00.0 device support hotplug, upstream device got a register to control the slot power.

Quoted 63 lines

After FPGA reload:

    $ lspci -vvv -xx -s 13:00.0
    13:00.0 Non-VGA unclassified device: Device 1b83:f002 (rev ff) (prog-if ff)
            !!! Unknown header type 7f
    00: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
    10: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
    20: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
    30: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff

- Jon

Quoted 36 lines
Quoted 35 lines

This removed the device and it did not show up again in lscpi.

Kernel has PCI HOTPLUG enabled:

    [jbaker@server0 ~]$ cat /boot/config-2.6.32-431.el6.x86_64 | grep HOTPLUG
    CONFIG_HOTPLUG=y
    CONFIG_MEMORY_HOTPLUG=y
    CONFIG_MEMORY_HOTPLUG_SPARSE=y
    CONFIG_HOTPLUG_CPU=y
    CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
    CONFIG_ACPI_HOTPLUG_CPU=y
    CONFIG_ACPI_HOTPLUG_MEMORY=y
    CONFIG_ACPI_HOTPLUG_MEMORY_AUTO_ONLINE=y
    CONFIG_HOTPLUG_PCI_PCIE=y
    CONFIG_HOTPLUG_PCI=y
    CONFIG_HOTPLUG_PCI_FAKE=m
    CONFIG_HOTPLUG_PCI_ACPI=y
    CONFIG_HOTPLUG_PCI_ACPI_IBM=m
    # CONFIG_HOTPLUG_PCI_CPCI is not set
    CONFIG_HOTPLUG_PCI_SHPC=m
    [jbaker@server0 ~]$

The kernel has pciehp module loaded, I see pciehp messages in /var/log/messages.

In other distros, on other platforms, echo 1 > /sys/bus/pci/rescan has worked but does not appear to work here for CentOS 6.5 on this hardware.

This problem is much discussed on the web but I have yet to see a solution discussed. I am thinking the 2.6.32-431.5.1.el6.x86_64 kernel may require a patch(es). The rescan may not be resetting the PCIe card?

Any ideas how to get rescan to work?

Thank you,

Jon Baker

Signature · 9 lines
===================================
Jon Baker
Software Engineer
ViaSat Inc.
Cleveland OH
216-706-7800
[email protected]
www.viasat.com
===================================
Signature · 2 lines
Thanks!
Yijing
#4 lore raw
In reply to Jon Baker
Quoted 8 lines · Jon Baker

I am trying to find a solution to a Linux PCI rescan problem.

I have CentOS 6.5

[jbaker@server0 ~]$ uname -a
Linux server0 2.6.32-431.5.1.el6.x86_64 #1 SMP Wed Feb 12 00:41:43
UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

Quoted 7 lines

I have an Altera FPGA PCIe protoboard I am trying to test in a TYAN FT77A-B7059.

We are trying to solve some hotplug/FPGA reload issues; when we reload the FPGA the PCI interface reloads confusing linux pci code.

After bootup the board is seen by the kernel; lscpi shows Altera FPGA PCIe protoboard settings and config space. We can load our driver and all is well.

When we reload the FPGA lscpi display shows all "ff" for config space. As

Are you saying all values in the config space are 0xff? If yes, this mean the vendor/device ids are invalid, so I'm just curious how OS identify this device, and you really can see that with lspci?

Tiejun

Quoted 50 lines

root I tried

echo 1 > /sys/bus/pci/rescan

But no change to lspci output.

I tried some variants

    echo 1 > /sys/bus/pci/devices/0000:8a:00.0/rescan

no change to lspci output.

echo 1 > /sys/bus/pci/devices/0000:8a:00.0/remove echo 1 > /sys/bus/pci/rescan

This removed the device and it did not show up again in lscpi.

Kernel has PCI HOTPLUG enabled:

    [jbaker@server0 ~]$ cat /boot/config-2.6.32-431.el6.x86_64 | grep

HOTPLUG
CONFIG_HOTPLUG=y
CONFIG_MEMORY_HOTPLUG=y
CONFIG_MEMORY_HOTPLUG_SPARSE=y
CONFIG_HOTPLUG_CPU=y
CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
CONFIG_ACPI_HOTPLUG_CPU=y
CONFIG_ACPI_HOTPLUG_MEMORY=y
CONFIG_ACPI_HOTPLUG_MEMORY_AUTO_ONLINE=y
CONFIG_HOTPLUG_PCI_PCIE=y
CONFIG_HOTPLUG_PCI=y
CONFIG_HOTPLUG_PCI_FAKE=m
CONFIG_HOTPLUG_PCI_ACPI=y
CONFIG_HOTPLUG_PCI_ACPI_IBM=m
# CONFIG_HOTPLUG_PCI_CPCI is not set
CONFIG_HOTPLUG_PCI_SHPC=m
[jbaker@server0 ~]$

The kernel has pciehp module loaded, I see pciehp messages in /var/log/messages.

In other distros, on other platforms, echo 1 > /sys/bus/pci/rescan has worked but does not appear to work here for CentOS 6.5 on this hardware.

This problem is much discussed on the web but I have yet to see a solution discussed. I am thinking the 2.6.32-431.5.1.el6.x86_64 kernel may require a patch(es). The rescan may not be resetting the PCIe card?

Any ideas how to get rescan to work?

Thank you,

Jon Baker

A message that is not in the archive (sent off-list or not yet received): [email protected]

#6 lore raw
Quoted 47 lines · Jon Baker
Quoted 13 lines · Tiejun
Quoted 12 lines · Jon Baker

I am trying to find a solution to a Linux PCI rescan problem.

I have CentOS 6.5

[jbaker@server0 ~]$ uname -a
Linux server0 2.6.32-431.5.1.el6.x86_64 #1 SMP Wed Feb 12
00:41:43 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

I have an Altera FPGA PCIe protoboard I am trying to test in a TYAN FT77A-B7059.

We are trying to solve some hotplug/FPGA reload issues; when we reload the FPGA the PCI interface reloads confusing linux pci code.

Quoted 7 lines
Quoted 3 lines

After bootup the board is seen by the kernel; lscpi shows Altera FPGA PCIe protoboard settings and config space. We can load our driver and all is well.

When we reload the FPGA lscpi display shows all "ff" for config space. As

Are you saying all values in the config space are 0xff? If yes, this mean the vendor/device ids are invalid, so I'm just curious how OS identify this device, and you really can see that with lspci?

Tiejun

Yes.

After system power up (case where everything works expected):

    $ lspci -vvv -xx -s 13:00.0
    13:00.0 Non-VGA unclassified device: Device 1b83:f002 (rev 01)
            Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV-
    VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
            Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast
Quoted 1 line

TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

            Latency: 0, Cache Line Size: 64 bytes
            Interrupt: pin A routed to IRQ 11
            Region 0: Memory at dcc00000 (32-bit, non-prefetchable)
    [size=256K]
            Region 1: Memory at dcb00000 (32-bit, non-prefetchable)
    [size=1M]
            Capabilities: <access denied>
    00: 83 1b 02 f0 06 00 10 00 01 00 00 00 10 00 00 00
    10: 00 00 c0 dc 00 00 b0 dc 00 00 00 00 00 00 00 00
    20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
    30: 00 00 00 00 50 00 00 00 00 00 00 00 0b 01 00 00

After FPGA reload:

Quoted 9 lines
    $ lspci -vvv -xx -s 13:00.0
    13:00.0 Non-VGA unclassified device: Device 1b83:f002 (rev ff)
    (prog-if ff)
            !!! Unknown header type 7f
    00: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
    10: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
    20: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
    30: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff

Looks you rescan directly with any first remove action. But seems you can't find your device again after you remove that firstly. So firstly you need to check if the host and device support hotplug before you validate hotplug feature.

If rescan manually, you may need to check if the link is established before rescan. If yes, probably the FPGA doesn't response any read config from bus actually, so all value are 0xff. You can use PCIE analyzer to check this.

Tiejun