Skip to contentSkip to conversation

PCI Express support for 2.4 kernel

PATCH Vladimir Kondratiev 4 participants 8 messages Dec 14, 2003 – Dec 16, 2003 PCI

lore
Share & tools
lore
#0 lore raw

Hi,
PCI-Express platforms will soon appear on the market. It is worth to support it.

Following is patch for 2.4.23 kernel. I tested it on my host, it works properly. I did it for i386 only, I have no other architecture to test.

It was patch on the same subject from* Seshadri, Harinarayanan* (/[email protected]/ <mailto:[email protected]>) http://www.cs.helsinki.fi/linux/linux-kernel/2003-17/0247.html
My version differ in several aspects: it is for 2.4 (vs. 2.6); it do not ioremap/unmap page for each transaction.

How about inclusion in 2.4.24?

I am not subscribed to lkml, thus please CC me (Vladimir Kondratiev <[email protected]>) in replies.

Vladimir.

Enable PCI Express access method for configuration space
This path includes:
* access routines itself
* command line argument "pci=exp" to force PCI Express, similar to "conf1" and "conf2"
* full 4k config accessed through /proc/bus/pci/...

How it works:

With PCI-E, config space accessed through memory. Each device gets its own 4k memory mapped config, total 256M for all devices.

At init time, I map whole region to not spent time for mapping later.

For /proc/bus/pci/..., I changed PCI_CFG_SPACE_SIZE to variable and changed it to 4k for PCI-E.

It is tested on 1 platform.

Author: "Vladimir Kondratiev" <[email protected]>

arch/i386/kernel/pci-i386.h +1 -0
diff -bBdur linux-2.4.23/arch/i386/kernel/pci-i386.h linux-2.4.23-pciexp/arch/i386/kernel/pci-i386.h
@@ -15,6 +15,7 @@
 #define PCI_PROBE_BIOS		0x0001
 #define PCI_PROBE_CONF1		0x0002
 #define PCI_PROBE_CONF2		0x0004
+#define PCI_PROBE_EXP		0x0008
 #define PCI_NO_SORT		0x0100
 #define PCI_BIOS_SORT		0x0200
 #define PCI_NO_CHECKS		0x0400
arch/i386/kernel/pci-pc.c +188 -4

229 lines not on this page · raw message

drivers/pci/proc.c +1 -1
diff -bBdur linux-2.4.23/drivers/pci/proc.c linux-2.4.23-pciexp/drivers/pci/proc.c
@@ -16,7 +16,7 @@
 #include <asm/uaccess.h>
 #include <asm/byteorder.h>
 
-#define PCI_CFG_SPACE_SIZE 256
+int PCI_CFG_SPACE_SIZE=256;
 
 static loff_t
 proc_bus_pci_lseek(struct file *file, loff_t off, int whence)
#1 lore raw
Quoted 7 lines · Vladimir Kondratiev

Hi,
PCI-Express platforms will soon appear on the market. It is worth to support it.

Following is patch for 2.4.23 kernel. I tested it on my host, it works properly.
I did it for i386 only, I have no other architecture to test.

Patch looks okay (except for totally broken indentation, that needs fixing)

It should go into 2.6 first. though. And while you're at it add your copyright info to the top of the file instead of the middle.

#2 lore raw
In reply to Vladimir Kondratiev

This really isn't appropriate. The

With PCI-E, config space accessed through memory. Each device gets its own 4k memory mapped config, total 256M for all devices.

thing _really_ does not work on x86, since 256M of IO mapping is _way_ way too much.

You _really_ need to allocate a FIXMAP entry (just one), and then use

	set_fixmap_nocache(FIX_PCIEXPRESS, phys);

to set it up for each device.

That's actually going to be a lot simpler than what you do now.

Linus

#3 lore raw
Quoted 3 lines · Linus Torvalds

You _really_ need to allocate a FIXMAP entry (just one), and then use

	set_fixmap_nocache(FIX_PCIEXPRESS, phys);

neat. dumb question though... how portable is set_fixmap_nocache()?

I only see it on four architectures, and I'm sure PCI Express will appear on more than that eventually.

Jeff

#4 lore raw
Quoted 2 lines · Jeff Garzik

neat. dumb question though... how portable is set_fixmap_nocache()?

Not very. Although it should generally be trivial to port if you need it.

Quoted 2 lines

I only see it on four architectures, and I'm sure PCI Express will appear on more than that eventually.

On 64-bit architectures you're not likely to ever need to worry about it, and then you can just map the whole thing directly (and use some special large-page mapping for it, at that).

Linus

#5 lore raw
Quoted 4 lines · Linus Torvalds
Quoted 1 line · Jeff Garzik

neat. dumb question though... how portable is set_fixmap_nocache()?

Not very. Although it should generally be trivial to port if you need it.

[...]

Quoted 3 lines

On 64-bit architectures you're not likely to ever need to worry about it, and then you can just map the whole thing directly (and use some special large-page mapping for it, at that).

Yeah, good point. IA64 with its 1001 different types of mappings and zones can go crazy, for example.

Sounds like PCI express needs a per-architecture policy for mapping the 256MB region. Some [64b] arches can direct-map the entire area. IA32 and others can go another route.

Jeff

#6 lore raw
In reply to Linus Torvalds

Actually, besides discussion on how to save 4 bytes in executable size, I see 2 real problems with my PCI-E patch.

First problem is those addressed by Linus (thanks a lot!). I looked for proper way to do mapping, but it was either wasting lots of virtual space, or overhead for ioremap/unmap per transaction. I though 1-st variant is preferable and used it.

Linus, FIXMAP helps, it is lighter then ioremap, but it still requires page table walk. In addition, since several operations should be done atomically, lock/unlock required as well. Can it be faster method, without page table walk for each transaction? To what extent should one concern about performance here?

As alternative between 1 page and 256M, I see also lazy allocation on per-bus basis: when bus is first accessed, ioremap 1Mb for it. On real system, it is no more then 3-4 buses. This way, we will end with several 1MB mappings. Finer granularity do not looks feasible, since bus scanning procedure tries to access all devices.

I am going to re-do PCI-E stuff with FIXMAP. To save extra page table walks, I'll keep last accessed page to not set_fixmap if we access the same device several times sequentially.

Second problem is generic way (lack of it) to recognize PCI-E presence and RRBAR physical address. Default is 0xe0000000, but theoretically it may be set to any 256M aligned. I used this default address and sanity_check for presence recognition. It is really not nice place in this patch. And worse, problem seems to be in PCI-E definition. I am working to address this problem before it is too late. What I want is some standard PCI capability that will be required in 00:00.0 device, its presence will indicate it is PCI-E, and its content will include RRBAR (and optionally any other appropriate stuff). Any alternative ideas?

Vladimir.

Quoted 20 lines · Linus Torvalds

This really isn't appropriate. The

With PCI-E, config space accessed through memory. Each device gets its own 4k memory mapped config, total 256M for all devices.

thing _really_ does not work on x86, since 256M of IO mapping is _way_ way too much.

You _really_ need to allocate a FIXMAP entry (just one), and then use

	set_fixmap_nocache(FIX_PCIEXPRESS, phys);

to set it up for each device.

That's actually going to be a lot simpler than what you do now.

Linus

#7 lore raw
Quoted 3 lines · Vladimir Kondratiev

Linus, FIXMAP helps, it is lighter then ioremap, but it still requires page table walk. In addition, since several operations should be done atomically, lock/unlock required as well. Can it be faster method, without page table walk for each transaction? To what extent should one concern about performance here?

As alternative between 1 page and 256M, I see also lazy allocation on per-bus basis: when bus is first accessed, ioremap 1Mb for it. On real system, it is no more then 3-4 buses. This way, we will end with several 1MB mappings. Finer granularity do not looks feasible, since bus scanning procedure tries to access all devices.

Well, two things to consider:
* probing is not a performance-intensive operation
* even with a system loaded down with many PCI Express devices, I doubt you will need more than 1-2MB mapped total, during runtime

So, one alternative could be to keep a cache of ioremap() regions -- smaller than 1MB in my opinion -- and update that as needed.

Anybody doing large numbers of PCI config register reads/writes in a hot path should be shot (or the h/w designers, depending on situation), and PCI Express doesn't change that ;-)

Jeff