Skip to contentSkip to conversation

PCI Express support for 2.4 kernel

PATCH Vladimir Kondratiev 12 participants 41 messages Dec 14, 2003 – Dec 17, 2003

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Please, ignore previous submission with the same subject. Patch file attached was wrong one. Now correct patch attached.

Hi,
PCI-Express platforms will soon appear on the market. It is worth to support it.

Following is patch for 2.4.23 kernel. I tested it on my host, it works properly.
I did it for i386 only, I have no other architecture to test.

It was patch on the same subject from* Seshadri, Harinarayanan* (/[email protected]/ <mailto:[email protected]>)
http://www.cs.helsinki.fi/linux/linux-kernel/2003-17/0247.html
My version differ in several aspects: it is for 2.4 (vs. 2.6); it do not ioremap/unmap page for each transaction.

How about inclusion in 2.4.24?

I am not subscribed to lkml, thus please CC me (Vladimir Kondratiev <[email protected]>) in replies.

Vladimir.

Enable PCI Express access method for configuration space
This path includes:
* access routines itself
* command line argument "pci=exp" to force PCI Express, similar to "conf1" and "conf2"
* full 4k config accessed through /proc/bus/pci/...

How it works:

With PCI-E, config space accessed through memory. Each device gets its own 4k memory mapped config, total 256M for all devices.

At init time, I map whole region to not spent time for mapping later.

For /proc/bus/pci/..., I changed PCI_CFG_SPACE_SIZE to variable and changed it to 4k for PCI-E.

It is tested on 1 platform.

Author: "Vladimir Kondratiev" <[email protected]>

arch/i386/kernel/pci-i386.h +1 -0
diff -bBdur linux-2.4.23/arch/i386/kernel/pci-i386.h linux-2.4.23-pciexp/arch/i386/kernel/pci-i386.h
@@ -15,6 +15,7 @@
 #define PCI_PROBE_BIOS		0x0001
 #define PCI_PROBE_CONF1		0x0002
 #define PCI_PROBE_CONF2		0x0004
+#define PCI_PROBE_EXP		0x0008
 #define PCI_NO_SORT		0x0100
 #define PCI_BIOS_SORT		0x0200
 #define PCI_NO_CHECKS		0x0400
arch/i386/kernel/pci-pc.c +174 -4

215 lines not on this page · raw message

drivers/pci/proc.c +1 -1
diff -bBdur linux-2.4.23/drivers/pci/proc.c linux-2.4.23-pciexp/drivers/pci/proc.c
@@ -16,7 +16,7 @@
 #include <asm/uaccess.h>
 #include <asm/byteorder.h>
 
-#define PCI_CFG_SPACE_SIZE 256
+int PCI_CFG_SPACE_SIZE=256;
 
 static loff_t
 proc_bus_pci_lseek(struct file *file, loff_t off, int whence)
#1 lore raw
Quoted 17 lines · Vladimir Kondratiev

Please, ignore previous submission with the same subject. Patch file attached was wrong one. Now correct patch attached.

Hi,
PCI-Express platforms will soon appear on the market. It is worth to support it.

Following is patch for 2.4.23 kernel. I tested it on my host, it works properly.
I did it for i386 only, I have no other architecture to test.

It was patch on the same subject from* Seshadri, Harinarayanan* (/[email protected]/ <mailto:[email protected]>)
http://www.cs.helsinki.fi/linux/linux-kernel/2003-17/0247.html
My version differ in several aspects: it is for 2.4 (vs. 2.6); it do not ioremap/unmap page for each transaction.

Quoted 4 lines

How about inclusion in 2.4.24?

I am not subscribed to lkml, thus please CC me (Vladimir Kondratiev <[email protected]>) in replies.

The patch could use some cleaning up, for coding style. The general style is documented in CodingStyle file in the kernel source tree. But also, you should consider the "golden rule of style" for programmers: When modifying a file, follow the same formatting as is found in the rest of the file.

Particularly:
* use a single tab to indent at each level
* use whitespace liberally. avoid "foo=bar". prefer "foo = bar".

Quoted 5 lines
arch/i386/kernel/pci-pc.c +5 -0
+/**
+ * PCI Express routines
+ * "Vladimir Kondratiev" <[email protected]>
+ */
+/**

arch/i386/kernel/pci-pc.c line 444

Quoted 10 lines
arch/i386/kernel/pci-pc.c +10 -0
+ * RRBAR (memory base for PCI-E config space) resides here.
+ * Initialized to default address. Actually, it is platform specific, and
+ * value may vary.
+ * I don't know how to detect it properly, it is chipset specific.
+ */
+static u32 rrbar_phys=0xe0000000UL;
+/**
+ * RRBAR is always 256M
+ */
+static u32 rrbar_size=0x10000000UL;

Insert spaces in the assignment statements.

arch/i386/kernel/pci-pc.c line 448

Quoted 4 lines
arch/i386/kernel/pci-pc.c +4 -0
+/**
+ * Virtual address for RRBAR
+ */
+static void* rrbar_virt=NULL;

Do not bother initializing static variables to zero. This just wastes bss space, since these variables are automatically zeroed for you, anyway.

arch/i386/kernel/pci-pc.c line 452

Quoted 4 lines
arch/i386/kernel/pci-pc.c +4 -0
+/**
+ * It used to be #define, but I am going to change it.
+ */
+extern int PCI_CFG_SPACE_SIZE;

Hopefully this is temporary... variables should not be ALL CAPS. If you change it to a variable, also change its case.

Quoted 6 lines
arch/i386/kernel/pci-pc.c +6 -0
+/**
+ * Initializes PCI Express method for config space access.
+ * 
+ * There is no standard method to recognize presence of PCI Express,
+ * thus we will assume it is PCI-E, and rely on sanity check to
+ * deassert PCI-E presense. If PCI-E not present,

arch/i386/kernel/pci-pc.c line 469

Quoted 10 lines
arch/i386/kernel/pci-pc.c +10 -0
+ * there is no physical RAM on RRBAR address, and we should read
+ * something like 0xff.
+ * 
+ * Creates mapping for whole 256M area.
+ * 
+ * @return 1 if OK, 0 if error
+ */
+static int pci_express_init(void)
+{
+  /* TODO: check PCI-Ex presense */

Um, this sounds critical. We should not merge this patch until this 'TODO' is complete, in my opinion.

Quoted 1 line

+ rrbar_virt=ioremap(rrbar_phys,rrbar_size);

add whitespace

Quoted 1 line

+ if (!rrbar_virt) return 0;

add a line break between two statements

Quoted 4 lines
arch/i386/kernel/pci-pc.c +4 -0
+  return 1;
+}
+
+/**

arch/i386/kernel/pci-pc.c line 485

Quoted 10 lines
arch/i386/kernel/pci-pc.c +10 -0
+ * Shuts down PCI-E resources.
+ */
+static void pci_express_fini(void)
+{
+  if (rrbar_virt) {
+    iounmap(rrbar_virt);
+  }
+}
+
+static int pci_exp_read (int seg, int bus, int dev, int fn, int reg, int len, u32 *value)

I think these arguments should be 'unsigned int', not 'int'.

Let's be proactive, and protect against signed/unsigned problems now, before they appear.

Review

Quoted 2 lines
(quoted diff) +2 -0
+{
+  void* addr=rrbar_virt+(bus << 20)+(dev << 15)+(fn << 12)+reg;

Why do you prefer '+' to '|' here?

'|' normally has less side effects.

Quoted 28 lines
arch/i386/kernel/pci-pc.c +28 -0
+  if ((bus > 255 || dev > 31 || fn > 7 || reg > 4095)) 
+    return -EINVAL;
+  switch (len) {
+  case 1:
+    *value=readb(addr);
+    break;
+  case 2:
+    if (reg&1) return -EINVAL;
+    *value=readw(addr);
+    break;
+  case 4:
+    if (reg&3) return -EINVAL;
+    *value=readl(addr);
+    break;
+  }
+  return 0;
+}
+
+static int pci_exp_write (int seg, int bus, int dev, int fn, int reg, int len, u32 value)
+{
+  void* addr=rrbar_virt+(bus << 20)+(dev << 15)+(fn << 12)+reg;
+  if ((bus > 255 || dev > 31 || fn > 7 || reg > 4095)) 
+    return -EINVAL;
+  switch (len) {
+  case 1:
+    writeb(value,addr);
+    break;
+  case 2:

arch/i386/kernel/pci-pc.c line 525

Quoted 10 lines
arch/i386/kernel/pci-pc.c +10 -0
+    if (reg&1) return -EINVAL;
+    writew(value,addr);
+    break;
+  case 4:
+    if (reg&3) return -EINVAL;
+    writel(value,addr);
+    break;
+  }
+  return 0;
+}

ditto, comments above: formatting, whitespace, signed/unsigned, logical or.

Further, PCI posting: a writeb() / writew() / writel() will not be flushed immediately to the processor. The CPU and/or PCI bridge may post (delay/combine) such writes. I do not think this is a desireable effect, for PCI config register accesses.

Quoted 33 lines
arch/i386/kernel/pci-pc.c +33 -0
+static int pci_exp_read_config_byte(struct pci_dev *dev, int where, u8 *value)
+{
+  int result; 
+  u32 data;
+  result = pci_exp_read(0, dev->bus->number, PCI_SLOT(dev->devfn), 
+			PCI_FUNC(dev->devfn), where, 1, &data);
+  *value = (u8)data;
+  return result;
+}
+
+static int pci_exp_read_config_word(struct pci_dev *dev, int where, u16 *value)
+{
+  int result; 
+  u32 data;
+  result = pci_exp_read(0, dev->bus->number, PCI_SLOT(dev->devfn), 
+			PCI_FUNC(dev->devfn), where, 2, &data);
+  *value = (u16)data;
+  return result;
+}
+
+static int pci_exp_read_config_dword(struct pci_dev *dev, int where, u32 *value)
+{
+  return pci_exp_read(0, dev->bus->number, PCI_SLOT(dev->devfn), 
+		      PCI_FUNC(dev->devfn), where, 4, value);
+}
+
+static int pci_exp_write_config_byte(struct pci_dev *dev, int where, u8 value)
+{
+  return pci_exp_write(0, dev->bus->number, PCI_SLOT(dev->devfn), 
+		       PCI_FUNC(dev->devfn), where, 1, value);
+}
+
+static int pci_exp_write_config_word(struct pci_dev *dev, int where, u16 value)

arch/i386/kernel/pci-pc.c line 569

Quoted 10 lines
arch/i386/kernel/pci-pc.c +10 -0
+{
+  return pci_exp_write(0, dev->bus->number, PCI_SLOT(dev->devfn), 
+		       PCI_FUNC(dev->devfn), where, 2, value);
+}
+
+static int pci_exp_write_config_dword(struct pci_dev *dev, int where, u32 value)
+{
+  return pci_exp_write(0, dev->bus->number, PCI_SLOT(dev->devfn), 
+		       PCI_FUNC(dev->devfn), where, 4, value);
+}

As with other code, define a macro that creates these functions. Then add a series of lines such as

DEF_PCIEX_READ(word, 2)
DEF_PCIEX_READ(dword, 4)
...

Quoted 70 lines
arch/i386/kernel/pci-pc.c +32 -3
+static struct pci_ops pci_express_conf = {
+  pci_exp_read_config_byte,
+  pci_exp_read_config_word,
+  pci_exp_read_config_dword,
+  pci_exp_write_config_byte,
+  pci_exp_write_config_word,
+  pci_exp_write_config_dword
+};
 
 /*
  * Before we decide to use direct hardware access mechanisms, we try to do some
@@ -465,6 +614,21 @@
 
 	__save_flags(flags); __cli();
 
+    /**
+     * Check if PCI-express access work
+     */
+    if (pci_express_init()) {
+        if (pci_sanity_check(&pci_express_conf)) {
+            PCI_CFG_SPACE_SIZE=4096;
+			__restore_flags(flags);
+			printk(KERN_INFO "PCI: Using configuration type PCI Express\n");
+			request_mem_region(rrbar_phys, rrbar_size, "PCI-Express config space");
+			return &pci_express_conf;
+        } else {
+            pci_express_fini();
+        }
+    }
+
 	/*
 	 * Check if configuration type 1 works.
 	 */
@@ -1398,16 +1562,18 @@
 #endif
 
 #ifdef CONFIG_PCI_DIRECT
-	if ((pci_probe & (PCI_PROBE_CONF1 | PCI_PROBE_CONF2)) 
+	if ((pci_probe & (PCI_PROBE_CONF1 | PCI_PROBE_CONF2 | PCI_PROBE_EXP)) 
 		&& (tmp = pci_check_direct())) {
 		pci_root_ops = tmp;
 		if (pci_root_ops == &pci_direct_conf1) {
 			pci_config_read = pci_conf1_read;
 			pci_config_write = pci_conf1_write;
-		}
-		else {
+		} else if (pci_root_ops == &pci_direct_conf2) {
 			pci_config_read = pci_conf2_read;
 			pci_config_write = pci_conf2_write;
+		} else if (pci_root_ops == &pci_express_conf) {
+			pci_config_read = pci_exp_read;
+			pci_config_write = pci_exp_write;
 		}
 	}
 #endif
@@ -1489,6 +1655,10 @@
 		pci_probe = PCI_PROBE_CONF2 | PCI_NO_CHECKS;
 		return NULL;
 	}
+	else if (!strcmp(str, "exp")) {
+		pci_probe = PCI_PROBE_EXP | PCI_NO_CHECKS;
+		return NULL;
+	}
 #endif
 	else if (!strcmp(str, "rom")) {
 		pci_probe |= PCI_ASSIGN_ROMS;

diff -bBdur linux-2.4.23/drivers/pci/proc.c linux-2.4.23-pciexp/drivers/pci/proc.c

drivers/pci/proc.c line 19

Quoted 5 lines
drivers/pci/proc.c +1 -1
@@ -16,7 +16,7 @@
 #include <asm/uaccess.h>
 #include <asm/byteorder.h>
 
-#define PCI_CFG_SPACE_SIZE 256
+int PCI_CFG_SPACE_SIZE=256;

ditto comment at the top... when changing to a variable, make it lowercase.

Jeff

#2 lore raw

Hi all,

Thanks for comments. Jeff, I specially appreciate time you spent walking through code.

I am not subscribed to lkml, thus please CC me (Vladimir Kondratiev <[email protected]>) in replies.

To keep posting short, I skipped most of original mail.
I attached fixed version for the patch. Sure, I tested it prior to posting.

Quoted 3 lines

<Jeff>
The patch could use some cleaning up, for coding style. The general style is documented in CodingStyle file in the kernel source tree. But also, you should consider the "golden rule of style" for programmers: When modifying a file, follow the same formatting as is found in the rest of the file.

Agree. It was my fault to not do it from the beginning. I habited to trust Emacs. Fixed.

Quoted 7 lines

Review

Quoted 4 lines
(quoted diff) +4 -0
+/**
+ * Virtual address for RRBAR
+ */
+static void* rrbar_virt=NULL;

Do not bother initializing static variables to zero. This just wastes bss space, since these variables are automatically zeroed for you, anyway.

I did not found this feature in standard. More, future versions of gcc will give at least warning, if not error, like "use of uninitialized variable". Many good sources also say it is good practice to initialize all variables. I rely on its value later. I' ll keep it as is unless really strong arguments provided.

Quoted 7 lines

Review

Quoted 4 lines
(quoted diff) +4 -0
+/**
+ * It used to be #define, but I am going to change it.
+ */
+extern int PCI_CFG_SPACE_SIZE;

Hopefully this is temporary... variables should not be ALL CAPS. If you change it to a variable, also change its case.

Agree. Fixed.

Quoted 9 lines
Quoted 9 lines

+/**
+ * Initializes PCI Express method for config space access.
+ * + * There is no standard method to recognize presence of PCI Express,
+ * thus we will assume it is PCI-E, and rely on sanity check to
+ * deassert PCI-E presense. If PCI-E not present,
+ * there is no physical RAM on RRBAR address, and we should read
+ * something like 0xff.
+ * + * Creates mapping for whole 256M area.

Quoted 8 lines

Review

Quoted 5 lines

+ * + * @return 1 if OK, 0 if error
+ */

(quoted diff) +3 -0
+static int pci_express_init(void)
+{
+  /* TODO: check PCI-Ex presense */

Um, this sounds critical. We should not merge this patch until this 'TODO' is complete, in my opinion.

As I explained in comment to function, it is not really critical. The problem is, there is no generic way (or I don't know it) to recognize PCI-E. One suggest to go over all devices and see whether PCI-E capability present for at least one of them. I don't think it is good way to do. Sanity check do pretty good job here. If it is not PCI-E platform, this address in physical memory will not be connected to anything real. You will get 0xff's.

Quoted 7 lines
Quoted 2 lines

+static int pci_exp_read (int seg, int bus, int dev, int fn, int reg, int len, u32 *value)

I think these arguments should be 'unsigned int', not 'int'.

Let's be proactive, and protect against signed/unsigned problems now, before they appear.

Sound reasonable. But this is how this function prototyped:

int (*pci_config_read)(int seg, int bus, int dev, int fn, int reg, int len, u32 *value) = NULL; int (*pci_config_write)(int seg, int bus, int dev, int fn, int reg, int len, u32 value) = NULL;

I think it is better to ask author (Martin) of this code, may be he had some reason for this prototype? I added him to CC list.

Quoted 7 lines
Quoted 1 line

+ void* addr=rrbar_virt+(bus << 20)+(dev << 15)+(fn << 12)+reg;

Why do you prefer '+' to '|' here?

'|' normally has less side effects.

Agree. Fixed, except 1-st "+", since virtual addres may be not aligned on 256M boundary

Quoted 2 lines

Further, PCI posting: a writeb() / writew() / writel() will not be flushed immediately to the processor. The CPU and/or PCI bridge may post (delay/combine) such writes. I do not think this is a desireable effect, for PCI config register accesses.

Good point. Fixed.

Quoted 6 lines

As with other code, define a macro that creates these functions. Then add a series of lines such as

DEF_PCIEX_READ(word, 2)
DEF_PCIEX_READ(dword, 4)
...

2 'read' functions (one do not follow pattern) and 3 'write' functions do not worth 2 macros. Also, coding style for the rest of file is different. Other methods defined w/o macros.

<Pete>

Quoted 10 lines

+ if (rrbar_virt) { <====== Not trusting own code

Review

Quoted 2 lines
(quoted diff) +2 -0
+        iounmap(rrbar_virt);
+    }

All in all, raw. Also, a healthy dose of Itanium is prescribed.

Extra check is here for the case code will be modified. In not frequently executed code, it is better to check twice rather then forget to check.

<Christoph>

Quoted 3 lines

It should go into 2.6 first. though. And while you're at it add your copyright info to the top of the file instead of the middle.

Copyright - done, thanks for input.
Regarding 2.6 - you are right, but now I need it for 2.4, and I have no 2.6 handy. I will do it in a week or 2.

Vladimir.

Enable PCI Express access method for configuration space
This path includes:
* access routines itself
* command line argument "pci=exp" to force PCI Express, similar to "conf1" and "conf2"
* full 4k config accessed through /proc/bus/pci/...

How it works:

With PCI-E, config space accessed through memory. Each device gets its own 4k memory mapped config, total 256M for all devices.

At init time, I map whole region to not spent time for mapping later.

For /proc/bus/pci/..., I changed PCI_CFG_SPACE_SIZE to variable and changed it to 4k for PCI-E.

It is tested on 1 platform.

Author: "Vladimir Kondratiev" <[email protected]>

arch/i386/kernel/pci-i386.h +1 -0
diff -dur linux-2.4.23/arch/i386/kernel/pci-i386.h linux-2.4.23-pciexp/arch/i386/kernel/pci-i386.h
@@ -15,6 +15,7 @@
 #define PCI_PROBE_BIOS		0x0001
 #define PCI_PROBE_CONF1		0x0002
 #define PCI_PROBE_CONF2		0x0004
+#define PCI_PROBE_EXP		0x0008
 #define PCI_NO_SORT		0x0100
 #define PCI_BIOS_SORT		0x0200
 #define PCI_NO_CHECKS		0x0400
arch/i386/kernel/pci-pc.c +182 -4

229 lines not on this page · raw message

drivers/pci/proc.c +13 -10
diff -dur linux-2.4.23/drivers/pci/proc.c linux-2.4.23-pciexp/drivers/pci/proc.c
@@ -16,7 +16,10 @@
 #include <asm/uaccess.h>
 #include <asm/byteorder.h>
 
-#define PCI_CFG_SPACE_SIZE 256
+/**
+ * For PCI Express, it will be set to 4096 during PCI init
+ */
+int pci_cfg_space_size=256;
 
 static loff_t
 proc_bus_pci_lseek(struct file *file, loff_t off, int whence)
@@ -31,12 +34,12 @@
 		new = file->f_pos + off;
 		break;
 	case 2:
-		new = PCI_CFG_SPACE_SIZE + off;
+		new = pci_cfg_space_size + off;
 		break;
 	default:
 		return -EINVAL;
 	}
-	if (new < 0 || new > PCI_CFG_SPACE_SIZE)
+	if (new < 0 || new > pci_cfg_space_size)
 		return -EINVAL;
 	return (file->f_pos = new);
 }
@@ -57,7 +60,7 @@
 	 */
 
 	if (capable(CAP_SYS_ADMIN))
-		size = PCI_CFG_SPACE_SIZE;
+		size = pci_cfg_space_size;
 	else if (dev->hdr_type == PCI_HEADER_TYPE_CARDBUS)
 		size = 128;
 	else
@@ -132,12 +135,12 @@
 	int pos = *ppos;
 	int cnt;
 
-	if (pos >= PCI_CFG_SPACE_SIZE)
+	if (pos >= pci_cfg_space_size)
 		return 0;
-	if (nbytes >= PCI_CFG_SPACE_SIZE)
-		nbytes = PCI_CFG_SPACE_SIZE;
-	if (pos + nbytes > PCI_CFG_SPACE_SIZE)
-		nbytes = PCI_CFG_SPACE_SIZE - pos;
+	if (nbytes >= pci_cfg_space_size)
+		nbytes = pci_cfg_space_size;
+	if (pos + nbytes > pci_cfg_space_size)
+		nbytes = pci_cfg_space_size - pos;
 	cnt = nbytes;
 
 	if (!access_ok(VERIFY_READ, buf, cnt))
@@ -389,7 +392,7 @@
 		return -ENOMEM;
 	e->proc_fops = &proc_bus_pci_operations;
 	e->data = dev;
-	e->size = PCI_CFG_SPACE_SIZE;
+	e->size = pci_cfg_space_size;
 	return 0;
 }
 
#3 lore raw
	Hi, 
Quoted 19 lines · Vladimir Kondratiev
Quoted 10 lines
Quoted 10 lines

+/**
+ * Initializes PCI Express method for config space access.
+ * + * There is no standard method to recognize presence of PCI Express,
+ * thus we will assume it is PCI-E, and rely on sanity check to
+ * deassert PCI-E presense. If PCI-E not present,
+ * there is no physical RAM on RRBAR address, and we should read
+ * something like 0xff.
+ * + * Creates mapping for whole 256M area.
+ * + * @return 1 if OK, 0 if error

Quoted 8 lines

arch/i386/kernel/pci-pc.c

Quoted 4 lines

+ */

arch/i386/kernel/pci-pc.c +3 -0
+static int pci_express_init(void)
+{
+  /* TODO: check PCI-Ex presense */

Um, this sounds critical. We should not merge this patch until this 'TODO' is complete, in my opinion.

Quoted 7 lines

As I explained in comment to function, it is not really critical. The problem is, there is no generic way (or I don't know it) to recognize PCI-E. One suggest to go over all devices and see whether PCI-E capability present for at least one of them. I don't think it is good way to do. Sanity check do pretty good job here. If it is not PCI-E platform, this address in physical memory will not be connected to anything real. You will get 0xff's.

Or a machine check on some architectures which make you know that they don't like a bus cycle which terminates on a master abort.

Quoted 2 lines

Agree. Fixed, except 1-st "+", since virtual addres may be not aligned on 256M boundary

The major problem I see is that using up 256 Mb of kernel virtual address space for accessing PCI config space is gross. Besides that it won't work for 32 bit machines with 768 Mb of RAM or more.

I believe that it would be better to use kmap/kunmap like tricks, especially for something which is relatively infrequent. You could even reserve a fixmap entry for this and keep somewhere a pointer to the PTE, which would be modified on every config space access (config space access was already properly serialized last time I looked, I believe that all you need is a TLB flush after the PTE is updated).

I have no strong opinion on how to handle 64 bit archs.

Quoted 7 lines
Quoted 6 lines

Further, PCI posting: a writeb() / writew() / writel() will not be flushed immediately to the processor. The CPU and/or PCI bridge may post (delay/combine) such writes. I do not think this is a desireable effect, for PCI config register accesses.

Good point. Fixed.

Here I'm somehwat lost. Writes to uncacheable RAM will be in program order and never combined. The bridge itself should not post writes to config space. So it's a matter of pushing the write to the processor bus, a PCI read looks very heavy for this. Isn't there a more lightweight solution ?

Regards,
Gabriel

#4 lore raw
Quoted 7 lines · Gabriel Paubert
Quoted 3 lines

Sanity check do pretty good job here. If it is not PCI-E platform, this address in physical memory will not be connected to anything real. You will get 0xff's.

Or a machine check on some architectures which make you know that they don't like a bus cycle which terminates on a master abort.

Could you elaborate a bit? This code is for pci-pc.c, it is platform specific. Are there surprises on pc platform? What is visible behavour in this case?

Quoted 10 lines

The major problem I see is that using up 256 Mb of kernel virtual address space for accessing PCI config space is gross. Besides that it won't work for 32 bit machines with 768 Mb of RAM or more.

I believe that it would be better to use kmap/kunmap like tricks, especially for something which is relatively infrequent. You could even reserve a fixmap entry for this and keep somewhere a pointer to the PTE, which would be modified on every config space access (config space access was already properly serialized last time I looked, I believe that all you need is a TLB flush after the PTE is updated).

I have no strong opinion on how to handle 64 bit archs.

I should be missing something here. You have 256M of physical address space at 0xe0000000 occupied. You can do nothing with it, it is simply present. Then, ioremap maps it somewhere in high memory. It should not conflict with kernel RAM, for which trivial mapping (+3G) used.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

Further, PCI posting: a writeb() / writew() / writel() will not be flushed immediately to the processor. The CPU and/or PCI bridge may post (delay/combine) such writes. I do not think this is a desireable effect, for PCI config register accesses.

Quoted 9 lines
Quoted 3 lines

Good point. Fixed.

Here I'm somehwat lost. Writes to uncacheable RAM will be in program order and never combined. The bridge itself should not post writes to config space. So it's a matter of pushing the write to the processor bus, a PCI read looks very heavy for this. Isn't there a more
lightweight solution ?

I am not expert here. I know that PCI-E bridge completes its transaction before data actually get to memory. Each intermediate bridge may do its own buffering. Besides read, I know nothing that will make you sure data reaches final destination. I'd appreciate if someone who knows better will step in. Config write is not very often procedure, I don't think extra read will make any difference. I'll look for qualified person around, to consult on this issue.

Vladimir.

#5 lore raw
Quoted 7 lines

I should be missing something here. You have 256M of physical address space at 0xe0000000 occupied.
You can do nothing with it, it is simply present. Then, ioremap maps it somewhere in high memory.
It should not conflict with kernel RAM, for which trivial mapping (+3G) used.

the thing is that typically you have a maximum of 168Mb or so of ioremap/vmalloc space (they share the same pool). That is, ff your system has >= 1Gb of ram, if it has less ran the ioremap/vmalloc space is bigger....

#6 lore raw

Got it.
Should I understand it this way: for system with >=1Gb RAM, I will be unable to ioremap 256Mb region? It looks confusing. On my test system (don't ask details, I am not alowed to share this info), I see video controller with 256Mb BAR. Does it mean this controller will not work as well?

There is alternative solution, for each transaction to ioremap/unmap corresponded page. I don't like it, it involves huge overhead.

I thought about remapping only pages that have actual PCI devices behind,
but this is problematic: access to config goes not always through pci_exp_read_config_xxx and alike, raw access with bus/dev/fn numbers used as well. And in 2.6, correct me if I wrong, raw access using bus/dev/fn numbers goes to be the only way. Per-device access replaced with per-bus, at least.

I can statically remap only region for existing buses, this will be huge save. It is 1MB per bus, this lead to typical 2-3MB instead of 256. To be sure I can do this, I need to know that new bus can't be added on run time. I don't think it is true, isn't it? Or do we have single point to capture hot plug for new bus?

Vladimir.

Quoted 13 lines · Arjan van de Ven
Quoted 5 lines

I should be missing something here. You have 256M of physical address space at 0xe0000000 occupied. You can do nothing with it, it is simply present. Then, ioremap maps it somewhere in high memory. It should not conflict with kernel RAM, for which trivial mapping (+3G) used.

the thing is that typically you have a maximum of 168Mb or so of ioremap/vmalloc space (they share the same pool). That is, ff your system has >= 1Gb of ram, if it has less ran the ioremap/vmalloc space is bigger....

#7 lore raw
Quoted 7 lines · Vladimir Kondratiev

Got it.
Should I understand it this way: for system with >=1Gb RAM, I will be unable to ioremap 256Mb region?
It looks confusing. On my test system (don't ask details, I am not alowed to share this info), I see
video controller with 256Mb BAR. Does it mean this controller will not work as well?

Video memory is generally not ioremap'd by the kernel (it may be mapped into XFree's address space though)

Quoted 6 lines

I thought about remapping only pages that have actual PCI devices behind, but this is problematic: access to config goes not always through pci_exp_read_config_xxx and alike, raw access with bus/dev/fn numbers used as well. And in 2.6, correct me if I wrong, raw access using bus/dev/fn numbers goes to be the only way. Per-device access replaced with per-bus, at least.

I would suspect you want to store the ioremap cookie for the config space in the pci device struct; longer term that struct maybe needs to grow a few function pointers to access config space too...

#8 lore raw
In reply to Vladimir Kondratiev
Quoted 4 lines · Vladimir Kondratiev

Got it.
Should I understand it this way: for system with >=1Gb RAM, I will be unable to ioremap 256Mb region? It looks confusing. On my test system (don't ask details, I am not alowed to share this info), I see video controller with 256Mb BAR. Does it mean this controller will not work as well?

But that video card's BAR is not ioremapped into the kernel (XFree86 will mmap it into userspace).

--
Brian Gerst

#9 lore raw
In reply to Vladimir Kondratiev
Quoted 5 lines · Vladimir Kondratiev

I can statically remap only region for existing buses, this will be huge save. It is 1MB per bus, this lead to typical 2-3MB instead of 256. To be sure I can do this, I need to know that new bus can't be added on run time. I don't think it is true, isn't it? Or do we have single point to capture hot plug for new bus?

New busses can be added at run time, remember hotplug PCI systems.

thanks,

greg k-h

#10 lore raw
In reply to Vladimir Kondratiev
Quoted 4 lines

It looks confusing. On my test system (don't ask details, I am not alowed to share this info), I see
video controller with 256Mb BAR. Does it mean this controller will not work as well?

The 256Mb BAR turns up in a variety of graphic cards, and 512Mb in a few. The kernel maps just enough memory for the frame buffer + cache. There isn't enough space to map it all. User space apps may well have the 512Mb mapped into their address space, but that isnt the kernel one

Quoted 3 lines

There is alternative solution, for each transaction to ioremap/unmap corresponded page.
I don't like it, it involves huge overhead.

ioremap means you can't use pci_exp_* from an interrupt so kmap is probably needed. It is overhead but the design of the 386 left us with that problem on x86 platforms and it isnt going to go away. On ia64/x86_64 etc its a non issue

#11 lore raw
In reply to Vladimir Kondratiev
Quoted 3 lines · Vladimir Kondratiev

Should I understand it this way: for system with >=1Gb RAM, I will be unable to ioremap 256Mb region?

Yes.

Quoted 4 lines

It looks confusing. On my test system (don't ask details, I am not alowed to share this info), I see
video controller with 256Mb BAR. Does it mean this controller will not work as well?

It works, but that's because the kernel doesn't even _try_ to map it into the kernel virtual address space.

XFree86 maps it into its user address space (which is 3GB, and doesn't have to try to map much else into it).

Quoted 2 lines

There is alternative solution, for each transaction to ioremap/unmap corresponded page.

Nope, you can't do that. It's not really supported from interrupts and early boot - both of which will want to access PCI config space.

Quoted 1 line

I don't like it, it involves huge overhead.

Indeed it would. But using fixmaps does _not_.

But a fixmap will be easy to use, and reasonable efficient: it will allocate just one virtual page, and then it will force a TLB flush when it switches over the mapping.

You can improve the efficiency by caching the "last fixmap entry" and only doing "set_fixmap()" when changing devices.

You could also do a per-cpu fixmap, which would help further, but that ends up being more work too. Probably not worth it, especially as it is entirely possible that the hardware requires single-threaded access anyway (ie I would not be surprised at all if the southbridge got very confused if you tried to do overlapping config accesses)..

Linus

#12 lore raw

I re-worked the patch. This time it uses fixmap; I added config parameters to disable this code and to handle non-default base address.
Unless generic solution for auto detection found, it seems appropriate way to go.

Autodetection problem isolated in 2 places: is_pcie_platform and, while it is not do real auto detection, configuration option for custom base.

Vladimir.

Quoted 26 lines · Linus Torvalds
Quoted 4 lines · Vladimir Kondratiev

There is alternative solution, for each transaction to ioremap/unmap corresponded page.

Nope, you can't do that. It's not really supported from interrupts and early boot - both of which will want to access PCI config space.

Quoted 3 lines

I don't like it, it involves huge overhead.

Indeed it would. But using fixmaps does _not_.

But a fixmap will be easy to use, and reasonable efficient: it will allocate just one virtual page, and then it will force a TLB flush when it switches over the mapping.

You can improve the efficiency by caching the "last fixmap entry" and only doing "set_fixmap()" when changing devices.

Quoted 9 lines

You could also do a per-cpu fixmap, which would help further, but that ends up being more work too. Probably not worth it, especially as it is entirely possible that the hardware requires single-threaded access anyway (ie I would not be surprised at all if the southbridge got very confused if you tried to do overlapping config accesses)..

Linus

Enable PCI Express access method for configuration space
This path includes:
* configuration options
* access routines itself
* command line argument "pci=exp" to force PCI Express, similar to "conf1" and "conf2"
* full 4k config accessed through /proc/bus/pci/...

How it works:

With PCI-E, config space accessed through memory. Each device gets its own 4k memory mapped config, total 256M for all devices.

At init time, fixmap used to create 4k window within config space. For each access, window set to proper physical address. Last physical address cached, to save page walk if the same device is accessed

For /proc/bus/pci/..., I changed PCI_CFG_SPACE_SIZE to variable and changed it to 4k for PCI-E.

It is tested on 1 platform, with PCI-E enabled and disabled.

Author: "Vladimir Kondratiev" <[email protected]>

Documentation/Configure.help +24 -0
diff -dur linux-2.4.23/Documentation/Configure.help linux-2.4.23-pciexp/Documentation/Configure.help
@@ -4131,6 +4131,30 @@
   if that doesn't work. If unsure, go with the default, which is
   "Any".
 
+PCI-Express support
+CONFIG_PCI_EXPRESS
+  PCI-Express is standard for devices interconnection. It is compatible
+  with PCI. If your system is PCI-Express, and you want support for
+  extended PCI-Express features, set this option to Y. PCI-Express
+  is auto detected, thus saying Y here is safe.
+
+  This option provides access to 4k extended config space and uses
+  memory-mapped method to access config space. Kernel option
+  "pci=exp" allows to force PCI-Express mode.
+
+Enable PCI-E custom base address
+CONFIG_PCIEXP_USE_CUSTOM_BASE
+  Override autodetected base address for PCI-Express config space.
+
+  If you don't know really well what are you doing, say "N".
+
+PCI-Express base address
+CONFIG_PCI_EXPRESS_BASE
+  Override autodetected base address for PCI-Express config space.
+  Base will be this number * 0x10000000 (256Mb).
+
+  Default value is "0xe".
+
 PCI device name database
 CONFIG_PCI_NAMES
   By default, the kernel contains a database of all known PCI device
arch/i386/config.in +9 -0
diff -dur linux-2.4.23/arch/i386/config.in linux-2.4.23-pciexp/arch/i386/config.in
@@ -292,6 +292,15 @@
       fi
       if [ "$CONFIG_PCI_GODIRECT" = "y" -o "$CONFIG_PCI_GOANY" = "y" ]; then
          define_bool CONFIG_PCI_DIRECT y
+         bool 'PCI-Express support' CONFIG_PCI_EXPRESS
+         if [ "$CONFIG_PCI_EXPRESS" = "y" ]; then
+            bool 'Enable PCI-E custom base address' CONFIG_PCIEXP_USE_CUSTOM_BASE
+            if [ "$CONFIG_PCIEXP_USE_CUSTOM_BASE" = "y" ]; then
+               hex 'PCI-Express base address' CONFIG_PCI_EXPRESS_BASE 0xe
+            else
+               define_hex CONFIG_PCI_EXPRESS_BASE 0xe
+            fi
+         fi
       fi
    fi
    bool 'ISA bus support' CONFIG_ISA
arch/i386/defconfig +3 -0
diff -dur linux-2.4.23/arch/i386/defconfig linux-2.4.23-pciexp/arch/i386/defconfig
@@ -79,6 +79,9 @@
 CONFIG_PCI_GOANY=y
 CONFIG_PCI_BIOS=y
 CONFIG_PCI_DIRECT=y
+# CONFIG_PCI_EXPRESS is not set
+# CONFIG_PCIEXP_USE_CUSTOM_BASE is not set
+CONFIG_PCI_EXPRESS_BASE=0xe
 CONFIG_PCI_NAMES=y
 # CONFIG_EISA is not set
 # CONFIG_MCA is not set
arch/i386/kernel/pci-i386.h +5 -0
diff -dur linux-2.4.23/arch/i386/kernel/pci-i386.h linux-2.4.23-pciexp/arch/i386/kernel/pci-i386.h
@@ -15,6 +15,11 @@
 #define PCI_PROBE_BIOS		0x0001
 #define PCI_PROBE_CONF1		0x0002
 #define PCI_PROBE_CONF2		0x0004
+
+#ifdef CONFIG_PCI_EXPRESS
+#define PCI_PROBE_EXP		0x0008
+#endif
+
 #define PCI_NO_SORT		0x0100
 #define PCI_BIOS_SORT		0x0200
 #define PCI_NO_CHECKS		0x0400
arch/i386/kernel/pci-pc.c +254 -5

304 lines not on this page · raw message

drivers/pci/proc.c +13 -10
diff -dur linux-2.4.23/drivers/pci/proc.c linux-2.4.23-pciexp/drivers/pci/proc.c
@@ -16,7 +16,10 @@
 #include <asm/uaccess.h>
 #include <asm/byteorder.h>
 
-#define PCI_CFG_SPACE_SIZE 256
+/**
+ * For PCI Express, it will be set to 4096 during PCI init
+ */
+int pci_cfg_space_size=256;
 
 static loff_t
 proc_bus_pci_lseek(struct file *file, loff_t off, int whence)
@@ -31,12 +34,12 @@
 		new = file->f_pos + off;
 		break;
 	case 2:
-		new = PCI_CFG_SPACE_SIZE + off;
+		new = pci_cfg_space_size + off;
 		break;
 	default:
 		return -EINVAL;
 	}
-	if (new < 0 || new > PCI_CFG_SPACE_SIZE)
+	if (new < 0 || new > pci_cfg_space_size)
 		return -EINVAL;
 	return (file->f_pos = new);
 }
@@ -57,7 +60,7 @@
 	 */
 
 	if (capable(CAP_SYS_ADMIN))
-		size = PCI_CFG_SPACE_SIZE;
+		size = pci_cfg_space_size;
 	else if (dev->hdr_type == PCI_HEADER_TYPE_CARDBUS)
 		size = 128;
 	else
@@ -132,12 +135,12 @@
 	int pos = *ppos;
 	int cnt;
 
-	if (pos >= PCI_CFG_SPACE_SIZE)
+	if (pos >= pci_cfg_space_size)
 		return 0;
-	if (nbytes >= PCI_CFG_SPACE_SIZE)
-		nbytes = PCI_CFG_SPACE_SIZE;
-	if (pos + nbytes > PCI_CFG_SPACE_SIZE)
-		nbytes = PCI_CFG_SPACE_SIZE - pos;
+	if (nbytes >= pci_cfg_space_size)
+		nbytes = pci_cfg_space_size;
+	if (pos + nbytes > pci_cfg_space_size)
+		nbytes = pci_cfg_space_size - pos;
 	cnt = nbytes;
 
 	if (!access_ok(VERIFY_READ, buf, cnt))
@@ -389,7 +392,7 @@
 		return -ENOMEM;
 	e->proc_fops = &proc_bus_pci_operations;
 	e->data = dev;
-	e->size = PCI_CFG_SPACE_SIZE;
+	e->size = pci_cfg_space_size;
 	return 0;
 }
 
include/asm-i386/fixmap.h +3 -0
diff -dur linux-2.4.23/include/asm-i386/fixmap.h linux-2.4.23-pciexp/include/asm-i386/fixmap.h
@@ -76,6 +76,9 @@
 	FIX_ACPI_BEGIN,
 	FIX_ACPI_END = FIX_ACPI_BEGIN + FIX_ACPI_PAGES - 1,
 #endif
+#ifdef CONFIG_PCI_EXPRESS
+	FIX_PCIE_CONFIG, /* Window within PCI-E config space*/
+#endif 
 	__end_of_permanent_fixed_addresses,
 	/* temporary boot-time mappings, used before ioremap() is functional */
 #define NR_FIX_BTMAPS	16
#13 lore raw
Quoted 3 lines · Vladimir Kondratiev

I re-worked the patch. This time it uses fixmap; I added config parameters to disable this code and to handle non-default base address.

Ok, this looks sane to me.

However, I have a totally independent question: do new PCI Express host bridges really not just suppor the old PCI config access interface?

Quite frankly, if I was a manager in charge of a PCI Express host bridge, and it didn't support the old C8C IO access patterns, I'd be so ashamed of myself that I'd kill my whole development team with rat poison, and then blame them for the mistake(*).

Do we know of any sudden suspicious death waves inside certain groups at Intel?

Linus

(*) That's how managers work, after all. Long gone are the days when personal shame caused you to take personal responsibility.

#14 lore raw

Linus,

Hopefully, they (or we, should I count myself?) are not completely brain dead. Old method still works.
All game is about to give full 4k. It will be required for new deices. If all you need is PCI devices behind PCI-E/PCI brigde,
it will work fine with old method.

Vladimir.

Quoted 27 lines · Linus Torvalds
Quoted 4 lines · Vladimir Kondratiev

I re-worked the patch. This time it uses fixmap; I added config parameters to disable this code and to handle non-default base address.

Ok, this looks sane to me.

However, I have a totally independent question: do new PCI Express host bridges really not just suppor the old PCI config access interface?

Quite frankly, if I was a manager in charge of a PCI Express host bridge, and it didn't support the old C8C IO access patterns, I'd be so ashamed of myself that I'd kill my whole development team with rat poison, and then blame them for the mistake(*).

Do we know of any sudden suspicious death waves inside certain groups at Intel?

Linus

(*) That's how managers work, after all. Long gone are the days when personal shame caused you to take personal responsibility.

#15 lore raw
Quoted 3 lines · Vladimir Kondratiev

Hopefully, they (or we, should I count myself?) are not completely brain dead. Old method still works.

Good. I just wanted to check from a timing perspective - it means that this won't be an issue for most people for a while (ie until we start seeing actual PCI-X-specific hardware and drivers rather than just the support chipsets - and I obviously have no idea how long that will take)

So if this will only matter for PCI-X drivers and not for discovery etc, I wonder if it wouldn't make sense to have this as a totally separate function? Instead of trying to make the existing "pci_config_xxxx()" stuff work with PCI-X, wouldn't it be nicer to have the driver just map its config space on probe?

That way there would be no scalability issues (only as many pages mapped as there are actual physical PCI-X devices that care) _and_ there would be no performance issues (the users would not need to walk page tables or flush the TLB - they'd just have a direct-mapped pointer).

You could do it with just ioremap(), but you'd really want to abstract it out a bit, and have a "[un]map_pcix_config()" function?

Linus

#16 lore raw
Quoted 3 lines · Linus Torvalds

So if this will only matter for PCI-X drivers and not for discovery etc, I
wonder if it wouldn't make sense to have this as a totally separate
function? Instead of trying to make the existing "pci_config_xxxx()" stuff work with PCI-X, wouldn't it be nicer to have the driver just map its config space on probe?

Not a bad idea... After posting yesterday on this thread, I had the thought: Just like PCI has readl() and sbus has sbus_readl(), why not pciex_cfg_readl() ?

Any PCI-Ex drivers would obviously _know_ they are PCI Ex, and they could communicate that by virtue of simply using new functions. Older drivers for older hardware would use the old API and not care... Further, PCI-Ex operations are already basically readl/writel anyway, so going through the forest of pci_cfg_ops pointers and such would just add needless layering.

Quoted 1 line

You could do it with just ioremap(), but you'd really want to abstract it out a bit, and have a "[un]map_pcix_config()" function?

Why not just work within the existing API? pci_{enable,disable}_device() seems fairly appropriate, as that's a quite clear signal of the bounds within which the driver must work.

pci_enable_device() is already defined as "the PCI device's resources may not be available before <this> point."

Jeff

#17 lore raw

I considered it in the beginning. For device driver itself, it is not a problem. For example, I can define struct pcie_dev_config {
void* base;
int (*read)(pcie_dev_config* conf, unsigned reg, int len, u32* value);
int (*write)(pcie_dev_config* conf, unsigned reg, int len, u32 value);
}
and have constructor/destructor functions that use device coordinates to calculate physical base and ioremap() it.

What you will miss, is uniform access for all devices, including those you are not managing as PCI-E. Notable example is bridges. I can't provide more info (see prev. mail about brain dead, I don't want it to be my last day at work), but you may found appropriate to tweak some stuff for bridges in extended space. One may use /proc/bus/pci/ or 'setpci' for this. Obviously, in this case you have no driver, and generic access method would help you.
Also, 'lspci' don't know about device drivers, it need generic way to access config.

And, strictly say, if it is method that replaces old one, wouldn't it more appropriate to use it rather then rely on backward compatibility hooks? I know, "work - don't touch", but...

Vladimir.

Quoted 22 lines · Jeff Garzik
Quoted 3 lines · Linus Torvalds

So if this will only matter for PCI-X drivers and not for discovery etc, I
wonder if it wouldn't make sense to have this as a totally separate
function? Instead of trying to make the existing "pci_config_xxxx()" stuff work with PCI-X, wouldn't it be nicer to have the driver just map its config space on probe?

Not a bad idea... After posting yesterday on this thread, I had the thought: Just like PCI has readl() and sbus has sbus_readl(), why not pciex_cfg_readl() ?

Any PCI-Ex drivers would obviously _know_ they are PCI Ex, and they could communicate that by virtue of simply using new functions. Older drivers for older hardware would use the old API and not care... Further, PCI-Ex operations are already basically readl/writel anyway, so going through the forest of pci_cfg_ops pointers and such would just add needless layering.

Quoted 1 line

You could do it with just ioremap(), but you'd really want to abstract it out a bit, and have a "[un]map_pcix_config()" function?

Why not just work within the existing API? pci_{enable,disable}_device() seems fairly appropriate, as that's a quite clear signal of the bounds within which the driver must work.

pci_enable_device() is already defined as "the PCI device's resources may not be available before <this> point."

Jeff

#18 lore raw
Quoted 9 lines · Vladimir Kondratiev

What you will miss, is uniform access for all devices, including those you are not managing as PCI-E. Notable example is bridges. I can't provide more info (see prev. mail about brain dead, I don't want it to be my last day at work), but you may found appropriate to tweak some stuff for bridges in extended space. One may use /proc/bus/pci/ or 'setpci' for this. Obviously, in this case you have no driver, and generic access method would help you. Also, 'lspci' don't know about device drivers, it need generic way to access config.

Ok, the /proc/bus/pci argument is a pretty good one, so having a uniform way to access the config space sounds fair.

And it doesn't look that ugly any more, so I guess if just the detection can be fixed (and real devices enter the market) I don't see any reason not to do it that way.

Linus

#19 lore raw
In reply to Jeff Garzik
Quoted 11 lines · Jeff Garzik
Quoted 5 lines · Linus Torvalds

So if this will only matter for PCI-X drivers and not for discovery etc, I wonder if it wouldn't make sense to have this as a totally separate function? Instead of trying to make the existing "pci_config_xxxx()" stuff work with PCI-X, wouldn't it be nicer to have the driver just map its config space on probe?

Not a bad idea... After posting yesterday on this thread, I had the
thought: Just like PCI has readl() and sbus has sbus_readl(), why not pciex_cfg_readl() ?

Quoted 6 lines

Any PCI-Ex drivers would obviously _know_ they are PCI Ex, and they could communicate that by virtue of simply using new functions. Older drivers for older hardware would use the old API and not care... Further, PCI-Ex operations are already basically readl/writel anyway, so going through the forest of pci_cfg_ops pointers and such would just add needless layering.

BUT powermanagement and co will need to potentially do stuff too with the config space...

#20 lore raw

Hello!

Quoted 2 lines

BUT powermanagement and co will need to potentially do stuff too with the config space...

Yes, but again they either access the registers already existing in normal PCI where they can use the old access mechanism, or they need to access the extended registers, but in this case it's a code specific for PCI-X which can use different functions for that, isnt't?

Have a nice fortnight

Signature · 3 lines
Martin `MJ' Mares   <[email protected]>   http://atrey.karlin.mff.cuni.cz/~mj/
Faculty of Math and Physics, Charles University, Prague, Czech Rep., Earth
If a train station is where the train stops, what is a work station?
#21 lore raw
In reply to Arjan van de Ven
Quoted 9 lines · Arjan van de Ven
Quoted 6 lines

Any PCI-Ex drivers would obviously _know_ they are PCI Ex, and they could communicate that by virtue of simply using new functions. Older drivers for older hardware would use the old API and not care... Further, PCI-Ex operations are already basically readl/writel anyway, so going through the forest of pci_cfg_ops pointers and such would just add needless layering.

BUT powermanagement and co will need to potentially do stuff too with the config space...

And X11 will want to access it via /proc interfaces. And someone will eventually go and design a different way to access PCI-EX for their hardware 8)

The PCI layer is a nice abstraction and config space is slow anyway

#22 lore raw
In reply to Linus Torvalds
Quoted 8 lines · Linus Torvalds
Quoted 2 lines · Vladimir Kondratiev

Hopefully, they (or we, should I count myself?) are not completely brain dead. Old method still works.

Good. I just wanted to check from a timing perspective - it means that this won't be an issue for most people for a while (ie until we start seeing actual PCI-X-specific hardware and drivers rather than just the support chipsets - and I obviously have no idea how long that will take)

    [...]

For the record: PCI Express is _not_ PCI-X.

PCI Express is a completely new bus system with new physical connectors. From a software point of view, it's (more or less) backwards-compatible with PCI, while PCI-X is completely backwards compatible with PCI.

Gr{oetje,eeting}s,

Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]

In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds

#23 lore raw
Quoted 2 lines · Geert Uytterhoeven

For the record: PCI Express is _not_ PCI-X.

Ok, but "PCI Express" is too damn long to write, so we'll have to have _some_ sane name for it without typing for half an hour.

Linus "retard on a keyboard" Torvalds

#24 lore raw
Quoted 5 lines · Linus Torvalds
Quoted 1 line · Geert Uytterhoeven

For the record: PCI Express is _not_ PCI-X.

Ok, but "PCI Express" is too damn long to write, so we'll have to have _some_ sane name for it without typing for half an hour.

3GIO? That was the original name :-)

Gr{oetje,eeting}s,

Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]

In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds

#25 lore raw
In reply to Linus Torvalds

Linus Torvalds <[email protected]> remarked:

Quoted 7 lines · Geert Uytterhoeven
Quoted 2 lines

For the record: PCI Express is _not_ PCI-X.

Ok, but "PCI Express" is too damn long to write, so we'll have to have _some_ sane name for it without typing for half an hour.

FWIW, "PCI-E" is in common use in these parts.

Also, wrt the config space backward compatibility issue, it is my understanding that the PCI-E root complex and PCI-E devices can be enumerated and used successfully with no software changes, within the constraints of PCI. The BIOS or OS should be able to enumerate devices, probe BARs and assign resources, perform basic error handling, etc. without any PCI-E-specific changes.

The hardware is required to be able to initialize PCI-E links, set things up to sane states, and so forth without software assistance in order to make this magic happen. It would be interesting to hear from Vladimir as to whether or not this is happening with his PCI-E test system.

Having said all that, it's obviously preferable to end up with native OS and BIOS support for the PCI-E extended configuration space, extra error reporting mechanisms, etc. Native PCI-E devices hanging off the bus somewhere might not work (or, at least, work well) with an OS that doesn't grok the PCI-E extensions.

Dan

#26 lore raw

Dan,

all your statements are correct. Indeed, all PCI devices works without extra knowledge about PCI-E. I have nothing to add.
Vladimir.

Quoted 37 lines · Dan Hopper

Linus Torvalds <[email protected]> remarked:

Quoted 9 lines · Geert Uytterhoeven
Quoted 5 lines

For the record: PCI Express is _not_ PCI-X.

Ok, but "PCI Express" is too damn long to write, so we'll have to have _some_ sane name for it without typing for half an hour.

FWIW, "PCI-E" is in common use in these parts.

Also, wrt the config space backward compatibility issue, it is my understanding that the PCI-E root complex and PCI-E devices can be enumerated and used successfully with no software changes, within the constraints of PCI. The BIOS or OS should be able to enumerate devices, probe BARs and assign resources, perform basic error handling, etc. without any PCI-E-specific changes.

The hardware is required to be able to initialize PCI-E links, set things up to sane states, and so forth without software assistance in order to make this magic happen. It would be interesting to hear from Vladimir as to whether or not this is happening with his PCI-E test system.

Having said all that, it's obviously preferable to end up with native OS and BIOS support for the PCI-E extended configuration space, extra error reporting mechanisms, etc. Native PCI-E devices hanging off the bus somewhere might not work (or, at least, work well) with an OS that doesn't grok the PCI-E extensions.

Dan

#27 lore raw
In reply to Geert Uytterhoeven
Quoted 1 line

For the record: PCI Express is _not_ PCI-X.

Yes, that was just a typo, sorry for the confusion.

Have a nice fortnight

Signature · 3 lines
Martin `MJ' Mares   <[email protected]>   http://atrey.karlin.mff.cuni.cz/~mj/
Faculty of Math and Physics, Charles University, Prague, Czech Rep., Earth
A. G. Bell is alive and well in New York and still waiting for the dial tone.
#28 lore raw
In reply to Vladimir Kondratiev

Vladimir Kondratiev wrote:

Definitely looks a lot better, thanks.

Still a few problems to consider...

Quoted 9 lines

diff -dur linux-2.4.23/arch/i386/config.in linux-2.4.23-pciexp/arch/i386/config.in

arch/i386/config.in +2 -0
@@ -292,6 +292,15 @@
       fi
       if [ "$CONFIG_PCI_GODIRECT" = "y" -o "$CONFIG_PCI_GOANY" = "y" ]; then
          define_bool CONFIG_PCI_DIRECT y
+         bool 'PCI-Express support' CONFIG_PCI_EXPRESS
+         if [ "$CONFIG_PCI_EXPRESS" = "y" ]; then

arch/i386/config.in line 306

Quoted 10 lines
arch/i386/config.in +7 -0
+            bool 'Enable PCI-E custom base address' CONFIG_PCIEXP_USE_CUSTOM_BASE
+            if [ "$CONFIG_PCIEXP_USE_CUSTOM_BASE" = "y" ]; then
+               hex 'PCI-Express base address' CONFIG_PCI_EXPRESS_BASE 0xe
+            else
+               define_hex CONFIG_PCI_EXPRESS_BASE 0xe
+            fi
+         fi
       fi
    fi
    bool 'ISA bus support' CONFIG_ISA

This is OK for development... But until (if ever?) there is a standard way to detect PCI Express, I think it is better to maintain a list of chipsets that support PCI Express.

Users are really going to hate this, once the first chipset comes out that uses a non-standard address.

Quoted 15 lines

diff -dur linux-2.4.23/arch/i386/kernel/pci-pc.c linux-2.4.23-pciexp/arch/i386/kernel/pci-pc.c

arch/i386/kernel/pci-pc.c +11 -0
 
+/**
+ * I don't know how to detect it properly.
+ * assume it is PCI-E, sanity_check will
+ * stop me if it is not.
+ * 
+ * Also, this function supposed to set rrbar_phys
+ */
+static int is_pcie_platform(void)
+{ return 1; }
+
+/**

arch/i386/kernel/pci-pc.c line 490

Quoted 10 lines
arch/i386/kernel/pci-pc.c +10 -0
+ * Initializes PCI Express method for config space access.
+ * 
+ * There is no standard method to recognize presence of PCI Express,
+ * thus we will assume it is PCI-E, and rely on sanity check to
+ * deassert PCI-E presense. If PCI-E not present,
+ * there is no physical RAM on RRBAR address, and we should read
+ * something like 0xff.
+ * 
+ * @return 1 if OK, 0 if error
+ */

Well, I agree with the comment, but that's not what the code does.

Where is your check for 0xff?

Further, is_pcie_platform() unconditionally returns 1... and is only used once, in PCI-Ex-specific code.

arch/i386/kernel/pci-pc.c line 502

Quoted 12 lines
arch/i386/kernel/pci-pc.c +12 -0
+static int pci_express_init(void)
+{
+	rrbar_window_virt = (void*)fix_to_virt(FIX_PCIE_CONFIG);
+	if (!is_pcie_platform())
+		return 0;
+	/**
+	 * adjust mapping for 1-st device (00:00.0)
+	 * to match pcie_last_accessed_device
+	 */
+	set_fixmap_nocache(FIX_PCIE_CONFIG, rrbar_phys);
+	return 1;
+}

Longer term, we want to provide some way to have the read/write routines generic, but support arch-specific mapping methods, I would think...

64-bit arches probably wouldn't need a spinlock at all for each access, I bet, since it's just a single MMIO read or write.

Quoted 60 lines
arch/i386/kernel/pci-pc.c +60 -0
+/**
+ * Shuts down PCI-E resources.
+ */
+static inline void pci_express_fini(void)
+{}
+
+/**
+ * Adjust window for the device.
+ * pci_config_lock should be held
+ */
+static inline void pci_exp_set_dev_base(int bus, int dev, int fn)
+{
+	u32 dev_base = (bus << 20) | (dev << 15) | (fn << 12);
+	if (dev_base != pcie_last_accessed_device) {
+		pcie_last_accessed_device = dev_base;
+		set_fixmap_nocache(FIX_PCIE_CONFIG, rrbar_phys | dev_base);
+	}
+}
+
+static int pci_exp_read (int seg, int bus, int dev, int fn, int reg, int len, u32 *value)
+{
+	unsigned long flags;
+	void *addr = rrbar_window_virt + reg;
+	if (((unsigned)bus > 255 || (unsigned)dev > 31
+	  || (unsigned)fn > 7 || (unsigned)reg > 4095)) 
+		return -EINVAL;
+	switch (len) {
+	case 2:
+		if (reg & 1)
+			return -EINVAL;
+		break;
+	case 4:
+		if (reg & 3)
+			return -EINVAL;
+		break;
+	}
+	spin_lock_irqsave(&pci_config_lock, flags);
+	pci_exp_set_dev_base(bus, dev, fn);
+	switch (len) {
+	case 1:
+		*value = readb(addr);
+		break;
+	case 2:
+		*value = readw(addr);
+		break;
+	case 4:
+		*value = readl(addr);
+		break;
+	}
+	spin_unlock_irqrestore(&pci_config_lock, flags);
+	return 0;
+}
+
+static int pci_exp_write (int seg, int bus, int dev, int fn, int reg, int len, u32 value)
+{
+	unsigned long flags;
+	void *addr = rrbar_window_virt + reg;
+	if (((unsigned)bus > 255 || (unsigned)dev > 31
+	  || (unsigned)fn > 7 || (unsigned)reg > 4095)) 
+		return -EINVAL;

arch/i386/kernel/pci-pc.c line 573

Quoted 10 lines
arch/i386/kernel/pci-pc.c +10 -0
+	switch (len) {
+	case 2:
+		if (reg & 1)
+			return -EINVAL;
+		break;
+	case 4:
+		if (reg & 3)
+			return -EINVAL;
+		break;
+	}

I don't see that read and write should ever return an error.

The above EINVAL conditions are a BUG().

Quoted 5 lines
arch/i386/kernel/pci-pc.c +5 -0
+	spin_lock_irqsave(&pci_config_lock, flags);
+	pci_exp_set_dev_base(bus, dev, fn);
+	switch (len) {
+	case 1:
+		writeb(value, addr);

arch/i386/kernel/pci-pc.c line 588

Quoted 10 lines
arch/i386/kernel/pci-pc.c +10 -0
+		break;
+	case 2:
+		writew(value, addr);
+		break;
+	case 4:
+		writel(value, addr);
+		break;
+	}
+	/* dummy read to flush PCI write */
+	readb(addr);

This is going to choke some hardware, I guarantee.

You always want to make sure your flush is of the same size at the write. Reading a byte from an address that the hardware defines as "32-bit writes only" can get ugly real quick ;-)

Quoted 58 lines

diff -dur linux-2.4.23/drivers/pci/proc.c linux-2.4.23-pciexp/drivers/pci/proc.c

drivers/pci/proc.c +12 -9
@@ -16,7 +16,10 @@
 #include <asm/uaccess.h>
 #include <asm/byteorder.h>
 
-#define PCI_CFG_SPACE_SIZE 256
+/**
+ * For PCI Express, it will be set to 4096 during PCI init
+ */
+int pci_cfg_space_size=256;
 
 static loff_t
 proc_bus_pci_lseek(struct file *file, loff_t off, int whence)
@@ -31,12 +34,12 @@
 		new = file->f_pos + off;
 		break;
 	case 2:
-		new = PCI_CFG_SPACE_SIZE + off;
+		new = pci_cfg_space_size + off;
 		break;
 	default:
 		return -EINVAL;
 	}
-	if (new < 0 || new > PCI_CFG_SPACE_SIZE)
+	if (new < 0 || new > pci_cfg_space_size)
 		return -EINVAL;
 	return (file->f_pos = new);
 }
@@ -57,7 +60,7 @@
 	 */
 
 	if (capable(CAP_SYS_ADMIN))
-		size = PCI_CFG_SPACE_SIZE;
+		size = pci_cfg_space_size;
 	else if (dev->hdr_type == PCI_HEADER_TYPE_CARDBUS)
 		size = 128;
 	else
@@ -132,12 +135,12 @@
 	int pos = *ppos;
 	int cnt;
 
-	if (pos >= PCI_CFG_SPACE_SIZE)
+	if (pos >= pci_cfg_space_size)
 		return 0;
-	if (nbytes >= PCI_CFG_SPACE_SIZE)
-		nbytes = PCI_CFG_SPACE_SIZE;
-	if (pos + nbytes > PCI_CFG_SPACE_SIZE)
-		nbytes = PCI_CFG_SPACE_SIZE - pos;
+	if (nbytes >= pci_cfg_space_size)
+		nbytes = pci_cfg_space_size;
+	if (pos + nbytes > pci_cfg_space_size)
+		nbytes = pci_cfg_space_size - pos;
 	cnt = nbytes;
 
 	if (!access_ok(VERIFY_READ, buf, cnt))

drivers/pci/proc.c line 398

Quoted 8 lines
drivers/pci/proc.c +1 -1
@@ -389,7 +392,7 @@
 		return -ENOMEM;
 	e->proc_fops = &proc_bus_pci_operations;
 	e->data = dev;
-	e->size = PCI_CFG_SPACE_SIZE;
+	e->size = pci_cfg_space_size;
 	return 0;
 }
 

Something I missed in the previous emails comments:

The above seems wrong, to blindly assume PCI-Ex means PCI config space will always be 4k. What about downstream PCI bridges, and ancient devices with only 256 bytes of config registers?

It really seems like the config space size should be per-device or per-bus.

Jeff

#29 lore raw
Quoted 18 lines · Jeff Garzik
Quoted 5 lines
arch/i386/kernel/pci-pc.c +5 -0
+	spin_lock_irqsave(&pci_config_lock, flags);
+	pci_exp_set_dev_base(bus, dev, fn);
+	switch (len) {
+	case 1:
+		writeb(value, addr);

arch/i386/kernel/pci-pc.c line 588

Quoted 10 lines
arch/i386/kernel/pci-pc.c +10 -0
+		break;
+	case 2:
+		writew(value, addr);
+		break;
+	case 4:
+		writel(value, addr);
+		break;
+	}
+	/* dummy read to flush PCI write */
+	readb(addr);

This is going to choke some hardware, I guarantee.

Quoted 3 lines

You always want to make sure your flush is of the same size at the write. Reading a byte from an address that the hardware defines as "32-bit writes only" can get ugly real quick ;-)

also reading back addr might not be the best choice in case some registers have side effects on reading, it's probably better to read back an address that is known to be ok to read (like the vendor ID field)

#30 lore raw
Quoted 9 lines · Arjan van de Ven
Quoted 2 lines · Jeff Garzik

This is going to choke some hardware, I guarantee.
You always want to make sure your flush is of the same size at the write. Reading a byte from an address that the hardware defines as "32-bit writes only" can get ugly real quick ;-)

also reading back addr might not be the best choice in case some registers have side effects on reading, it's probably better to read back an address that is known to be ok to read (like the vendor ID field)

Hum. Never seen this in a PCI config register, but it's certainly possible... good point.

Jeff

#31 lore raw
In reply to Arjan van de Ven
Quoted 6 lines · Arjan van de Ven
Quoted 6 lines

arch/i386/kernel/pci-pc.c line 588

Quoted 4 lines
arch/i386/kernel/pci-pc.c +2 -0
+	/* dummy read to flush PCI write */
+	readb(addr);

This is going to choke some hardware, I guarantee.

Quoted 11 lines
Quoted 3 lines

You always want to make sure your flush is of the same size at the write. Reading a byte from an address that the hardware defines as "32-bit writes only" can get ugly real quick ;-)

also reading back addr might not be the best choice in case some registers have side effects on reading, it's probably better to read back an address that is known to be ok to read (like the vendor ID field)

Good idea!

#32 lore raw
Quoted 14 lines · Vladimir Kondratiev
Quoted 12 lines · Arjan van de Ven
Quoted 11 lines

arch/i386/kernel/pci-pc.c line 588

Quoted 4 lines
arch/i386/kernel/pci-pc.c +2 -0
+	/* dummy read to flush PCI write */
+	readb(addr);

This is going to choke some hardware, I guarantee.

You always want to make sure your flush is of the same size at the write. Reading a byte from an address that the hardware defines as "32-bit writes only" can get ugly real quick ;-)

Quoted 8 lines
Quoted 7 lines

also reading back addr might not be the best choice in case some registers have side effects on reading, it's probably better to read back an address that is known to be ok to read (like the vendor ID field)

Good idea!

You should not abitrarily flush all the writes with a read after each write. The PCI/Bus is a FIFO, stuff will get to the hardware in the order written. If you "flush" every write, you will seriously hurt the performance. I don't like that MACRO/Function, whatever it is with the switch shown previously. All that code just for a "@(*^&$(*^(" write will screw up performance. I've looked at the result of some the the MACRO expansions, in particular the get_user, put_user, and copy_to/from macros where there is a switch. It is not pre-processed out.

Cheers,
Dick Johnson
Penguin : Linux version 2.4.22 on an i686 machine (797.90 BogoMips).
Note 96.31% of all statistics are fiction.

#33 lore raw
In reply to Jeff Garzik
Quoted 35 lines · Jeff Garzik

Vladimir Kondratiev wrote:

Definitely looks a lot better, thanks.

Still a few problems to consider...

Quoted 17 lines

diff -dur linux-2.4.23/arch/i386/config.in linux-2.4.23-pciexp/arch/i386/config.in
--- linux-2.4.23/arch/i386/config.in 2003-11-28 20:26:19.000000000
+0200
+++ linux-2.4.23-pciexp/arch/i386/config.in 2003-12-16
11:18:46.000000000 +0200

(quoted diff) +0 -0
@@ -292,6 +292,15 @@
       fi
       if [ "$CONFIG_PCI_GODIRECT" = "y" -o "$CONFIG_PCI_GOANY" = "y" 

]; then
define_bool CONFIG_PCI_DIRECT y

arch/i386/config.in +3 -0
+         bool 'PCI-Express support' CONFIG_PCI_EXPRESS
+         if [ "$CONFIG_PCI_EXPRESS" = "y" ]; then
+            bool 'Enable PCI-E custom base address' 

CONFIG_PCIEXP_USE_CUSTOM_BASE

(quoted diff) +2 -0
+            if [ "$CONFIG_PCIEXP_USE_CUSTOM_BASE" = "y" ]; then
+               hex 'PCI-Express base address' 

CONFIG_PCI_EXPRESS_BASE 0xe

arch/i386/config.in line 306

Quoted 7 lines
arch/i386/config.in +4 -0
+            else
+               define_hex CONFIG_PCI_EXPRESS_BASE 0xe
+            fi
+         fi
       fi
    fi
    bool 'ISA bus support' CONFIG_ISA

This is OK for development... But until (if ever?) there is a standard way to detect PCI Express, I think it is better to maintain a list of chipsets that support PCI Express.

Quoted 2 lines

Users are really going to hate this, once the first chipset comes out that uses a non-standard address.

Sure, it need to be replaced, or at least complemented, with real auto detection. If I only could provide this list...

Quoted 23 lines
Quoted 12 lines
arch/i386/kernel/pci-pc.c +12 -0
+/**
+ * I don't know how to detect it properly.
+ * assume it is PCI-E, sanity_check will
+ * stop me if it is not.
+ * + * Also, this function supposed to set rrbar_phys
+ */
+static int is_pcie_platform(void)
+{ return 1; }
+
+/**
+ * Initializes PCI Express method for config space access.
+ * + * There is no standard method to recognize presence of PCI 
Quoted 7 lines

Express,
+ * thus we will assume it is PCI-E, and rely on sanity check to
+ * deassert PCI-E presense. If PCI-E not present,
+ * there is no physical RAM on RRBAR address, and we should read
+ * something like 0xff.
+ * + * @return 1 if OK, 0 if error
+ */

Well, I agree with the comment, but that's not what the code does.

Quoted 2 lines

Where is your check for 0xff?

sanity_check do it for me.

Quoted 2 lines

Further, is_pcie_platform() unconditionally returns 1... and is only used once, in PCI-Ex-specific code.

See above, it is placeholder for real auto detection routine.

Quoted 5 lines

Longer term, we want to provide some way to have the read/write routines generic, but support arch-specific mapping methods, I would think...

64-bit arches probably wouldn't need a spinlock at all for each access, I bet, since it's just a single MMIO read or write.

Yes, separation into generic and platform specific part seems nice, but besides memory mapping and locking, all you have is very simple arithmetic. Does it worth the work for separation?

For 64-bit world, agree, it could and should be done uniform.

Quoted 12 lines

arch/i386/kernel/pci-pc.c line 539

Quoted 10 lines
arch/i386/kernel/pci-pc.c +10 -0
+    switch (len) {
+    case 2:
+        if (reg & 1)
+            return -EINVAL;
+        break;
+    case 4:
+        if (reg & 3)
+            return -EINVAL;
+        break;
+    }
Quoted 4 lines

I don't see that read and write should ever return an error.

The above EINVAL conditions are a BUG().

Indeed, it may be better to BUG() here. I did not paid attention. By the way, for other access methods alignment is simply not verified. Probable, it is ensured by some other way and I can just remove this test?

Quoted 9 lines

arch/i386/kernel/pci-pc.c line 588

Quoted 3 lines
arch/i386/kernel/pci-pc.c +2 -0
+    /* dummy read to flush PCI write */
+    readb(addr);

This is going to choke some hardware, I guarantee.

You always want to make sure your flush is of the same size at the write. Reading a byte from an address that the hardware defines as "32-bit writes only" can get ugly real quick ;-)

Good point. s/readb(addr)/readl(addr & ~3)/

Quoted 6 lines

Something I missed in the previous emails comments:

The above seems wrong, to blindly assume PCI-Ex means PCI config space will always be 4k. What about downstream PCI bridges, and ancient devices with only 256 bytes of config registers?

It really seems like the config space size should be per-device or per-bus.

Not this bad. For PCI devices after bridges, 4k also provided. Everything after 256 bytes is simply useless, but present. It reads as 0.

#34 lore raw
In reply to Vladimir Kondratiev

Minor code comments below:

arch/i386/kernel/pci-i386.h line 22

Quoted 5 lines · Vladimir Kondratiev
arch/i386/kernel/pci-i386.h +5 -0
+
+#ifdef CONFIG_PCI_EXPRESS
+#define PCI_PROBE_EXP		0x0008
+#endif
+

If you change this to:
#ifdef CONFIG_PCI_EXPRESS
#define PCI_PROBE_EXP 0x0008
#else
#define PCI_PROBE_EXP 0x0000
#endif

You can get rid of a number of "#ifdef CONFIG_PCI_EXPRESS" statements later on in the .c files by just always using the PCI_PROBE_EXP value. That cleans up the patch a bit.

arch/i386/kernel/pci-pc.c line 34

Quoted 6 lines
arch/i386/kernel/pci-pc.c +5 -1
-unsigned int pci_probe = PCI_PROBE_BIOS | PCI_PROBE_CONF1 | PCI_PROBE_CONF2;
+unsigned int pci_probe = PCI_PROBE_BIOS | PCI_PROBE_CONF1 | PCI_PROBE_CONF2
+#ifdef CONFIG_PCI_EXPRESS
+ | PCI_PROBE_EXP
+#endif
+;

Like right there :)

Quoted 7 lines
arch/i386/kernel/pci-pc.c +7 -0
+/**
+ * RRBAR (memory base for PCI-E config space) resides here.
+ * Initialized to default address. Actually, it is platform specific, and
+ * value may vary.
+ * I don't know how to detect it properly, it is chipset specific.
+ */
+static u32 rrbar_phys = CONFIG_PCI_EXPRESS_BASE * 0x10000000UL;

<snip>

arch/i386/kernel/pci-pc.c line 478

Quoted 9 lines
arch/i386/kernel/pci-pc.c +9 -0
+/**
+ * I don't know how to detect it properly.
+ * assume it is PCI-E, sanity_check will
+ * stop me if it is not.
+ * 
+ * Also, this function supposed to set rrbar_phys
+ */
+static int is_pcie_platform(void)
+{ return 1; }

Shouldn't your comments at least match your code for the rrbar_phys statement for your first release? :)

Quoted 6 lines
arch/i386/kernel/pci-pc.c +6 -0
+/**
+ * Initializes PCI Express method for config space access.
+ * 
+ * There is no standard method to recognize presence of PCI Express,
+ * thus we will assume it is PCI-E, and rely on sanity check to
+ * deassert PCI-E presense. If PCI-E not present,

arch/i386/kernel/pci-pc.c line 495

Quoted 10 lines
arch/i386/kernel/pci-pc.c +10 -0
+ * there is no physical RAM on RRBAR address, and we should read
+ * something like 0xff.
+ * 
+ * @return 1 if OK, 0 if error
+ */
+static int pci_express_init(void)
+{
+	rrbar_window_virt = (void*)fix_to_virt(FIX_PCIE_CONFIG);
+	if (!is_pcie_platform())
+		return 0;

Call fix_to_virt() after you do the check and not before?

arch/i386/kernel/pci-pc.c line 508

Quoted 5 lines
arch/i386/kernel/pci-pc.c +5 -0
+/**
+ * Shuts down PCI-E resources.
+ */
+static inline void pci_express_fini(void)
+{}

If this isn't needed, why have it anymore?

arch/i386/kernel/pci-pc.c line 644

Quoted 8 lines
arch/i386/kernel/pci-pc.c +8 -0
+static struct pci_ops pci_express_conf = {
+	pci_exp_read_config_byte,
+	pci_exp_read_config_word,
+	pci_exp_read_config_dword,
+	pci_exp_write_config_byte,
+	pci_exp_write_config_word,
+	pci_exp_write_config_dword
+};

C99 initializers here?

arch/i386/kernel/pci-pc.c line 694

Quoted 1 line
arch/i386/kernel/pci-pc.c +1 -0
+			printk(KERN_INFO "PCI-Express config at 0x%08x\n", rrbar_phys);

"%p" to show the address might be nicer.

thanks,

greg k-h

#35 lore raw
Quoted 20 lines · Greg KH

Minor code comments below:

arch/i386/kernel/pci-i386.h line 22

Quoted 7 lines · Vladimir Kondratiev
arch/i386/kernel/pci-i386.h +5 -0
+
+#ifdef CONFIG_PCI_EXPRESS
+#define PCI_PROBE_EXP		0x0008
+#endif
+

If you change this to:
#ifdef CONFIG_PCI_EXPRESS
#define PCI_PROBE_EXP 0x0008
#else
#define PCI_PROBE_EXP 0x0000
#endif

Quoted 5 lines

You can get rid of a number of "#ifdef CONFIG_PCI_EXPRESS" statements later on in the .c files by just always using the PCI_PROBE_EXP value. That cleans up the patch a bit.

Good point. Agree.

Quoted 14 lines
Quoted 9 lines
arch/i386/kernel/pci-pc.c +7 -0
+/**
+ * RRBAR (memory base for PCI-E config space) resides here.
+ * Initialized to default address. Actually, it is platform specific, and
+ * value may vary.
+ * I don't know how to detect it properly, it is chipset specific.
+ */
+static u32 rrbar_phys = CONFIG_PCI_EXPRESS_BASE * 0x10000000UL;

<snip>

Quoted 16 lines

arch/i386/kernel/pci-pc.c line 478

Quoted 11 lines
arch/i386/kernel/pci-pc.c +9 -0
+/**
+ * I don't know how to detect it properly.
+ * assume it is PCI-E, sanity_check will
+ * stop me if it is not.
+ * 
+ * Also, this function supposed to set rrbar_phys
+ */
+static int is_pcie_platform(void)
+{ return 1; }

Shouldn't your comments at least match your code for the rrbar_phys statement for your first release? :)

Default moved to CONFIG_PCI_EXPRESS_BASE.
Idea is some time later we fill replace is_pcie_platform with real auto detection...

Quoted 12 lines

arch/i386/kernel/pci-pc.c line 495

Quoted 7 lines
arch/i386/kernel/pci-pc.c +5 -0
+static int pci_express_init(void)
+{
+	rrbar_window_virt = (void*)fix_to_virt(FIX_PCIE_CONFIG);
+	if (!is_pcie_platform())
+		return 0;

Call fix_to_virt() after you do the check and not before?

... and I assume virtual address may be used by auto detection code.

Quoted 13 lines

arch/i386/kernel/pci-pc.c line 508

Quoted 9 lines
arch/i386/kernel/pci-pc.c +5 -0
+/**
+ * Shuts down PCI-E resources.
+ */
+static inline void pci_express_fini(void)
+{}

If this isn't needed, why have it anymore?

Placeholder, to not forget. It is called in case auto detection says it is not PCI-E.

Quoted 16 lines

arch/i386/kernel/pci-pc.c line 644

Quoted 12 lines
arch/i386/kernel/pci-pc.c +8 -0
+static struct pci_ops pci_express_conf = {
+	pci_exp_read_config_byte,
+	pci_exp_read_config_word,
+	pci_exp_read_config_dword,
+	pci_exp_write_config_byte,
+	pci_exp_write_config_word,
+	pci_exp_write_config_dword
+};

C99 initializers here?

cut'n'paste from other methods, to keep style. If change, let's change all.

Quoted 10 lines

arch/i386/kernel/pci-pc.c line 694

Quoted 5 lines
arch/i386/kernel/pci-pc.c +1 -0
+			printk(KERN_INFO "PCI-Express config at 0x%08x\n", rrbar_phys);

"%p" to show the address might be nicer.

I print phys. address, it is u32. Do you mean (void*)rrbar_phys? Don't see why not to change, I have no strong opinion for which format is better.

#36 lore raw
Quoted 9 lines · Vladimir Kondratiev
Quoted 8 lines · Greg KH

Minor code comments below:

arch/i386/kernel/pci-pc.c line 694

Quoted 3 lines · Vladimir Kondratiev
arch/i386/kernel/pci-pc.c +1 -0
+			printk(KERN_INFO "PCI-Express config at 0x%08x\n", rrbar_phys);

"%p" to show the address might be nicer.

Quoted 3 lines

I print phys. address, it is u32. Do you mean (void*)rrbar_phys? Don't see why not to change,
I have no strong opinion for which format is better.

Are you 100% sure the physical address value will be limited to 32-bit on 64-bit platforms, too?

If not (and IMHO even if yes, for clarity), you should use long or void *, which is 64-bit on 64-bit platforms.

Gr{oetje,eeting}s,

Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]

In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds

#37 lore raw
In reply to Gabriel Paubert

Gabriel Paubert wrote:

Gabriel,
I verified with PCI-E designers,
uncacheable memory relates to snoop/non-snoop, not to buffering in bridge. Bridge will still buffer writes. The only way to be sure data has arrived, is to perform read.

Vladimir.

Quoted 17 lines
Quoted 7 lines
Quoted 4 lines

Further, PCI posting: a writeb() / writew() / writel() will not be flushed immediately to the processor. The CPU and/or PCI bridge may post (delay/combine) such writes. I do not think this is a desireable effect, for PCI config register accesses.

Good point. Fixed.

Here I'm somehwat lost. Writes to uncacheable RAM will be in program order and never combined. The bridge itself should not post writes to config space. So it's a matter of pushing the write to the processor bus, a PCI read looks very heavy for this. Isn't there a more
lightweight solution ?

Regards,
Gabriel

#38 lore raw
In reply to Vladimir Kondratiev

Hi!

Quoted 1 line

I did not found this feature in standard.

I did :-) C99, section 6.7.8 "Initialization", constraint 10:

"... If an object that has static storage duration is not initialized explicitly, then:

- if it has pointer type, it is initialized to a null pointer;
- if it has arithmetic type, it is initialized to (positive or unsigned) zero;

..."

Quoted 2 lines

More, future versions of gcc will give at least warning, if not error, like "use of uninitialized variable".

Yes, such warning exists, but only for automatic variables.

Have a nice fortnight

Signature · 3 lines
Martin `MJ' Mares   <[email protected]>   http://atrey.karlin.mff.cuni.cz/~mj/
Faculty of Math and Physics, Charles University, Prague, Czech Rep., Earth
Quote of the day: '
#39 lore raw
In reply to Vladimir Kondratiev
Quoted 6 lines · Vladimir Kondratiev

Please, ignore previous submission with the same subject. Patch file attached was wrong one. Now correct patch attached.

Hi,
PCI-Express platforms will soon appear on the market. It is worth to support it.

Yes it is worth it, any chance to get access to hardware to test this out on?

Quoted 4 lines

Following is patch for 2.4.23 kernel. I tested it on my host, it works properly.
I did it for i386 only, I have no other architecture to test.

Quoted 8 lines

It was patch on the same subject from* Seshadri, Harinarayanan* (/[email protected]/ <mailto:[email protected]>)
http://www.cs.helsinki.fi/linux/linux-kernel/2003-17/0247.html
My version differ in several aspects: it is for 2.4 (vs. 2.6); it do not ioremap/unmap page for each transaction.

How about inclusion in 2.4.24?

No, we need to get this into 2.6 first. Can you please forward port this to 2.6, clean up the formatting and address the issues everyone else has made so far and post it?

Quoted 1 line

* command line argument "pci=exp" to force PCI Express, similar to "conf1" and "conf2"

We should be able to do this automatically, and not force this on the boot command line, correct?

arch/i386/kernel/pci-pc.c line 440

Quoted 7 lines
arch/i386/kernel/pci-pc.c +7 -0
+/**
+ * RRBAR (memory base for PCI-E config space) resides here.
+ * Initialized to default address. Actually, it is platform specific, and
+ * value may vary.
+ * I don't know how to detect it properly, it is chipset specific.
+ */
+static u32 rrbar_phys=0xe0000000UL;

How about information on how to detect it as per chipset type? We need to do this automatically some how.

Quoted 4 lines

+/**
+ * Initializes PCI Express method for config space access.
+ *
+ * There is no standard method to recognize presence of PCI Express,

Are you sure? I thought there was (don't have my spec in front of me right now...)

thanks,

greg k-h

#40 lore raw
Quoted 10 lines · Greg KH
Quoted 5 lines

Hi,
PCI-Express platforms will soon appear on the market. It is worth to support it.

Yes it is worth it, any chance to get access to hardware to test this out on?

If it were up to me, I will give away full specs and test platforms. But it is not...

Quoted 5 lines

No, we need to get this into 2.6 first. Can you please forward port this to 2.6, clean up the formatting and address the issues everyone else has made so far and post it?

I will. It take some time, I did not installed 2.6 yet.

Quoted 8 lines
Quoted 3 lines

* command line argument "pci=exp" to force PCI Express, similar to "conf1" and "conf2"

We should be able to do this automatically, and not force this on the boot command line, correct?

Yes. Default is autodetect. Command line is to suppress autodetection.

Quoted 11 lines

How about information on how to detect it as per chipset type? We need to do this automatically some how.
+ *
+ * There is no standard method to recognize presence of PCI Express,

Are you sure? I thought there was (don't have my spec in front of me right now...)

I thought this way also. But I found that it is not. You may know several chipsets, and do per-chipset stuff, but there is no generic procedure. At least authors of PCI-E don't know (it is nice to have access to the authors ;-) ).

Vladimir.