PCI card not accessible; I/O ports at <ignored>

Hi folks,

I'm maintaining the driver software for the PCI cards manufactured by our company, Meinberg Funkuhren in Germany. Basically our Linux driver supports all our PCI cards on all Linux kernels 2.6.x and 3.x. The PCI cards are e.g. GPS receivers, radio clocks, or IRIG time code receivers.

Recently I've run into a problem with our PCI Express cards in a certain server PC: There are 8 PCI Express slots on the mainboard, and in some of the slots the cards work properly, but in some other slots on the same machine they don't work since the cards are not accessible by the kernel driver.

The cards are accessed via legacy I/O ports. I *know* memory mapped registers should be used preferably, but some older types of card don't support this. In cases where the cards are not accessible lspci -v reports for the I/O base address 0:

Region 0: I/O ports at <ignored>

First I thought this was a problem with our cards, with the chipset on the mainboard, with the BIOS, or whatever. Then I tried to boot Windows and FreeDOS on this machine and found that the cards could always be accessed in all slots without problems.

Finally I tried different Linux distributions with different kernel versions and found that the same card in the same slot with the same driver works properly under kernel 2.6.32 and earlier, but doesn't work anymore under kernel 2.6.36 and later, including kernel 3.1.4. So there must have been some code changes in the newer kernel versions which prevent the card from being accessed via I/O, and let lspci say "I/O ports at <ignored>".

Here is the full output of lspci under kernel 3.1.4 in case where the card is *not* accessible:

root@MMS:~# lspci -vvv -x -d1360:*
87:00.0 System peripheral: Meinberg Funkuhren GPS180PEX GPS Receiver (PCI Express) (rev 01)
        Subsystem: Meinberg Funkuhren GPS180PEX GPS Receiver (PCI Express)
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- 
ParErr+ Stepping- SERR+ FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- 
<TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 256 bytes
        Interrupt: pin A routed to IRQ 54
        Region 0: I/O ports at <ignored>
        Region 1: Memory at f8cc0000 (32-bit, non-prefetchable) [size=256K]
        Capabilities: [50] MSI: Enable- Count=1/4 Maskable- 64bit+
                Address: 0000000000000000  Data: 0000
        Capabilities: [78] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA 
PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [80] Express (v1) Legacy Endpoint, MSI 00
                DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s 
<64ns, L1 <1us
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
                DevCtl: Report errors: Correctable- Non-Fatal- Fatal- 
Unsupported-
                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 256 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- 
AuxPwr- TransPend-
                LnkCap: Port #1, Speed 2.5GT/s, Width x1, ASPM L0s, 
Latency L0 unlimited, L1 unlimited
                        ClockPM- Surprise- LLActRep- BwNot-
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- 
CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ 
DLActive- BWMgmt- ABWMgmt-
        Capabilities: [100 v1] Virtual Channel
                Caps:   LPEVC=0 RefClk=100ns PATEntryBits=1
                Arb:    Fixed- WRR32- WRR64- WRR128-
                Ctrl:   ArbSelect=Fixed
                Status: InProgress-
                VC0:    Caps:   PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
                        Arb:    Fixed- WRR32- WRR64- WRR128- TWRR128- 
WRR256-
                        Ctrl:   Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
                        Status: NegoPending- InProgress-
        Kernel modules: mbgclock
00: 60 13 06 02 47 01 10 00 01 00 80 08 40 00 00 00
10: 01 fc 00 00 00 00 cc f8 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 60 13 06 02
30: 00 00 00 00 50 00 00 00 00 00 00 00 0a 01 00 00

The interesting thing here is the hex dump of the configuration space above, which shows that actually I/O address 0xFC00 *has* been assigned to base address register 0 at offset 0x10. However, when the kernel calls the driver's probe routine then this I/O range is *not* in the resource list of the device retrieved by the probe routine.

Here is the lspci output if the card is installed in a different slot where it can be accessed:

root@MMS:~# lspci -vvv -x -d1360:*
03:00.0 System peripheral: Meinberg Funkuhren GPS180PEX GPS Receiver (PCI Express) (rev 01)
        Subsystem: Meinberg Funkuhren GPS180PEX GPS Receiver (PCI Express)
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- 
ParErr+ Stepping- SERR+ FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- 
<TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 256 bytes
        Interrupt: pin A routed to IRQ 26
        Region 0: I/O ports at dc00 [size=256]
        Region 1: Memory at faec0000 (32-bit, non-prefetchable) [size=256K]
        Capabilities: [50] MSI: Enable- Count=1/4 Maskable- 64bit+
                Address: 0000000000000000  Data: 0000
        Capabilities: [78] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA 
PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [80] Express (v1) Legacy Endpoint, MSI 00
                DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s 
<64ns, L1 <1us
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
                DevCtl: Report errors: Correctable- Non-Fatal- Fatal- 
Unsupported-
                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 256 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- 
AuxPwr- TransPend-
                LnkCap: Port #1, Speed 2.5GT/s, Width x1, ASPM L0s, 
Latency L0 unlimited, L1 unlimited
                        ClockPM- Surprise- LLActRep- BwNot-
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- 
CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ 
DLActive- BWMgmt- ABWMgmt-
        Capabilities: [100 v1] Virtual Channel
                Caps:   LPEVC=0 RefClk=100ns PATEntryBits=1
                Arb:    Fixed- WRR32- WRR64- WRR128-
                Ctrl:   ArbSelect=Fixed
                Status: InProgress-
                VC0:    Caps:   PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
                        Arb:    Fixed- WRR32- WRR64- WRR128- TWRR128- 
WRR256-
                        Ctrl:   Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
                        Status: NegoPending- InProgress-
        Kernel driver in use: mbgclock
        Kernel modules: mbgclock
00: 60 13 06 02 47 01 10 00 01 00 80 08 40 00 00 00
10: 01 dc 00 00 00 00 ec fa 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 60 13 06 02
30: 00 00 00 00 50 00 00 00 00 00 00 00 0a 01 00 00

Here the hex dump of the configuration space shows I/O address 0xDC00 for base address range 0, lspsi does not say <ignored>, and the card is accessible.

Can someone please shed some light on this and tell me:

- What makes lspci say "I/O ports at <ignored>"?

- What could be the reason why base address register 0 is not added to the resource list for this PCI device when the kernel scans the bus and detects the device?

- What has been the reason to implement this behaviour in newer kernel versions whereas this seems to work without problems under older kernels?

If any more information is required to fiddle this out, please let me know.

Thanks for any input on this.

Martin

Martin Burnicki

Meinberg Funkuhren
Bad Pyrmont
Germany

Nice, any reason why these drivers aren't in the main kernel.org tree?

thanks,

greg k-h

Yes, you wouldn't like the coding style of our driver ;-)

Seriously, I have discussed this with some Linux guys before, and also with some folks from *BSD who also wanted to pick up the driver into their *BSD source trees.

The disadvantages of this approach were:

- If someone wanted to use a new type of PCI card then he was forced to upgrade the kernel to get support for it. We have quite a number of customers who want to stick with a specific Linux distribution version (and thus kernel version) for some reason, so they wouldn't be able to use a newer PCI card model unless they backport a newer driver to an older kernel. E.g. we have a number of customers who still run some old RHEL versions with kernels around 2.6.16 or so, but they are using current PCI card models.

- Sometimes it happens that new features are implemented in the firmware of the PCI cards, often even on request of a customer. To benefit from such new features the appropriate changes needed to be made in the driver code, and again this would require some backporting if the user is not willing to upgrade the kernel.

- The driver code would have to be reformatted for Linux to comply with the Linux coding style, for FreeBSD to comply with the FreeBSD coding style, etc.

Our cards are often used as reference time source for ntpd, but also for measurement applications which access the cards directly. In any way, the cards provide usually quite a number of configuration parameters which can be or need to be set up properly, depending on the requirements.

API calls required for these cards include reading high resolution,high accurate timestamps from the card, but also lots of configuration stuff, e.g. reading which IRIG code formats are supported by a given IRIG receiver card, and configuring the selected IRIG code to be decoded by the card. The huge amount of API calls is very specific to our cards.

Actually our driver code is divided into some pieces of OS-specific code and a huge piece of code which is shared for all supported operating systems, including Linux, Windows, FreeBSD, NetBSD, Novell Netware, DOS, formerly OS/2, and even by the firmware for the microcontrollers on the PCI devices.

There is one driver package for Linux, which can be used with all 2.6.x and 3.x kernels, one package for FreeBSD, a single driver for all Windows versions from WinNT up to Win7 and Server 2008. All these drivers share a lot of source code, and in fact the API calls available to applications are source code compatible across all supported operating systems. We have had quite a number of customers who started to use our cards under Windows, and then switched to Linux, and didn't have to change the way API calls for our PCI cards are used by their application.

Since many years our Linux driver is available as source code on our download page:
http://www.meinberg.de/english/sw/#linux

I must admit that the driver on this page is a little bit outdated. However, we are currently preparing a new release:
http://www.meinberg.de/download/drivers/mbgtools-lx-dev-2011-12-19.tar.gz

There's also a corresponding driver version for FreeBSD, the first release of which will be available with the next version of the Linux driver:
http://www.meinberg.de/download/drivers/mbgtools-fbsd-dev-2011-12-21.tar.gz

If you look into both archives you'll see that the user space tools and mostly all the stuff below the mbglib directory is shared by both driver packages.

For example, there is one common source code module which implements the basic functions usually required by kernel drivers, e.g.:

- check if a particular device is supported

- start the device and detect the features it provides, which depend on the type of card but may also depend on the firmware version actually installed on the card

- provide different low level functions required to access cards depending on the PCI interface chip assembled on the detected card

In addition, there is an OS-specific skeleton driver which calls those routines if e.g. the kernel calls the probe or ioctl routine for a given device.

So the advantages of our approach to maintain the drivers out of the Linux or *BSD source trees are:

- we can implement new features and support new card types by making changes *once* in the source code, and the new feature is available immediately for all supported operating systems.

- it provides most possible flexibility for the folks who use our PCI cards since they can use our latest driver for the newest cards even with older distributions/kernels. Just unpack the archive, type make && make install
and that's it.

I'd like to point out explicitely that our approach is *not* because we want to keep some stuff as secret. Everything is available as source code, and you can grab a copy of the code and merge it into the kernel tree, if you want. However, maintaining the driver code would be much harder then since for a new feature the Linux tree needed to be updated, *BSD trees needed to be updated, and we'd need to maintain our local code base anyway to support those operating systems which are not open source.

BTW, we are currently also working on a GUI application which is based on wxWidgets and can be used to view and modify configuration parameters on our different PCI card types under different operating systems. This project also shares much of the common code, so once more it is easier to provide common driver packages than having a specific driver in the kernel only.

Sorry for the lengthy reply, but I've tried to make our point as clear as possible, and hope you understand and accept the reasoning.

Martin

Martin Burnicki

Meinberg Funkuhren
Bad Pyrmont
Germany

Am Mittwoch 18 Januar 2012, 19:22:19 schrieben Sie:

I've had such a driver before. Trust me, it was a great relief once I got rid of it because I did not have to maintain the os abstraction layer anymore. And many of the lower level functions you need to do by hand for all systems can be done with fewer calls on nearly all systems. Just the borders of these calls differ between the systems, so you can't abstract this away in the wrapper layer.

And noone stops you to have your driver available for older kernel releases too. But plugging something in and it just works is just cool ;)

Eike

<snip>

That all makes sense, thanks. It's your decision, but note, everyone who has ever taken the time to get their code into the main kernel tree, FreeBSD included, has had less time and energy maintaining it over the long-run.

Plus, you get support for your customers from Red Hat and SUSE and others, and you don't void their warranty by loading unsupported kernels :)

But it's your business decision, if this is what you want to do, great, best of luck.

thanks,

greg k-h

This may be true e.g. for NIC drivers, etc, which need to talk to the NIC hardware at the downside and be compliant to the network stack API upwards.

However, for the kind of cards we are dealing with there is no standardized API (like the network stack) in the kernel which the kernel could use to access the hardware.

Based on a simple standard skeleton driver providing some methods for loading, unloading, ioctl, etc. the major changes required over the last years to support different kernel versions were to account for changes in the kernel API calls which need to be called by the driver, e.g. around 2.6.27 the function device_create() started to expect one more parameter than in earlier kernel versions.

We don't even use DMA or some other enhanced techniques since the amount of data to be transferred is usually marginal. The main

Appliances for our cards are usually to let applications read timestamps from the card as exactly as possible, and with as low latency and execution time as possible.

If new ways are found to improve this then the hardware/firmware needs to support this, the kernel driver needs to support this, etc.

As said earlier, with our approach we can make the required changes once in the source code and the new features are available under all supported operating systems and kernel versions.

If the code was in the kernel trees of several open source operating systems we either had to explain to every maintainer of every OS what they needed to change, or do it by ourselves individually for every OS and submit a number of patches.

In all the years we have supported Linux there have not been any stability problems due to our driver, even though it is not officially supported by distributions. ;-)

Thanks. Again I hope I could make the reasining for our approach clear enough, and I also hope that I'm anyway allowed to ask for some technical information here on this list, especially where my actual question is not even directly related to our driver, but just to changes in the kernel which cause existing PCI hardware to be treated differently as before.

Regards,

Martin

Martin Burnicki

Meinberg Funkuhren
Bad Pyrmont
Germany

this happens when the BAR contains zero.

Please open a bug report at http://bugzilla.kernel.org, category Drivers/PCI, mark it as a regression, add me to the CC: list, and attach complete dmesg logs from 2.6.32 and 3.1.4 with the card in the slot that no longer works.

Bjorn

Thanks for your reply. I've just sent a lengthy explanation to this list ;-)

I'll care about the bug report tomorrow.

Martin

Martin Burnicki

Meinberg Funkuhren
Bad Pyrmont
Germany

This must be the BAR value saved inside the kernel's device structure. The confguration space dump of lspci -x shows that even in this case a usual I/O address has been assigned to BAR0 on the card.

Done:
https://bugzilla.kernel.org/show_bug.cgi?id=42606

Thanks again!

Martin

Martin Burnicki

Meinberg Funkuhren
Bad Pyrmont
Germany