Skip to contentSkip to conversation

: IO translation with designware PCIe controller

Query/Discussion Pratyush Anand 7 participants 17 messages Dec 5, 2013 – Dec 13, 2013 PCI

lore
Share & tools
lore
#0 lore raw

Hi Arnd / Jingoo,

Sorry for not able to follow earlier discussion you had on this topic while initial reviews of patches from Jingoo.

I have few doubts that how will it actually work.

As per my understanding under current implementation:

1. Physical IO addresses are in the range of 0x1000 to 0xfffff.
2. They are mapped to fixed virtual address 0xFEE00000 using pci_ioremap_io.
3. IO resource addresses to any PCIe device will be allocated in above range.
4. Driver for that PCIe device will write in the above range only for IO transaction to the device.
5. As per current IO translation programming, its one to one translation. It means PCIe translation unit will have both input and output address in the range of 0x1000 to 0xfffff.

Did I miss something, or if above statements are correct, then what I do not understand is how can designware PCIe translation unit accept input address in the range of 0x1000 to 0xfffff?

For example in SPEAr1340, physically RAM is mapped on above addresses. PCIe address translation unit can accept address only in the range of core addresses which are assigned to PCIe RC ie 0x80000000-0x8FFFFFFF.

Regards
Pratyush

#1 lore raw
Quoted 20 lines · Pratyush Anand

Hi Arnd / Jingoo,

Sorry for not able to follow earlier discussion you had on this topic while initial reviews of patches from Jingoo.

I have few doubts that how will it actually work.

As per my understanding under current implementation:

1. Physical IO addresses are in the range of 0x1000 to 0xfffff.
2. They are mapped to fixed virtual address 0xFEE00000 using pci_ioremap_io.
3. IO resource addresses to any PCIe device will be allocated in above range.
4. Driver for that PCIe device will write in the above range only for IO transaction to the device.
5. As per current IO translation programming, its one to one translation. It means PCIe translation unit will have both input and output address in the range of 0x1000 to 0xfffff.

Quoted 3 lines

Did I miss something, or if above statements are correct, then what I do not understand is how can designware PCIe translation unit accept input address in the range of 0x1000 to 0xfffff?

Hi Pratyush,

I think the part that you are missing is that there are four, not two address spaces involved:

1. The "bus" I/O port address, in the range between 0x1000 and 0xffff (65535). This is the address used in data frames in the actual bus transaction going over the PCIe wires. If you have more than one PCIe host bridge, each of them can normally have a range of up to 65536 addresses. The bus addresses get written into BAR registers of the device.

2. The Linux I/O port numbers, between 0x00000 and 0xfffff (1048575). This is a logical number space that aggregates all bus I/O port numbers. The first 4096 ports are normally reserved for ISA devices and are not available for PCIe resources. Each PCI or PCIe host bridge can have a 0x10000 byte naturally aligned range in here. The common case is that you have only one host bridge using the range 0x1000-0xffff. If you have more than one, there may be an offset between them, e.g. the second PCIe host may have io_offset set to 0x10000, which means that its bus range 0x1000-0xffff gets translated to Linux port number 0x11000-0x1ffff. Linux port numbers are visible e.g. in /proc/ioport and /dev/port

3. The CPU physical address space: Each PCIe host normally has its bus I/O port space mapped into PCU visible addresses, either hardwired or (in case of dw_pcie) set up using an outbound translation window. These addresses are what get passed to pci_ioremap_io.

4. The CPU virtual address space: This is an implementation detail of the kernel, which results in the Linux I/O port numbers 0x0-0xfffff to be mapped at 0xfee00000-0xfeefffff.

Quoted 3 lines

For example in SPEAr1340, physically RAM is mapped on above addresses. PCIe address translation unit can accept address only in the range of core addresses which are assigned to PCIe RC ie 0x80000000-0x8FFFFFFF.

Since this is a physical address, it corresponds to address space 3 in my list above, and the address you pick here is what you pass to pci_ioremap_io.

Arnd

#2 lore raw

Hi Arnd,

Thanks a lot for detailed explanation.

Quoted 30 lines · Arnd Bergmann
Quoted 20 lines · Pratyush Anand

Hi Arnd / Jingoo,

Sorry for not able to follow earlier discussion you had on this topic while initial reviews of patches from Jingoo.

I have few doubts that how will it actually work.

As per my understanding under current implementation:

1. Physical IO addresses are in the range of 0x1000 to 0xfffff.
2. They are mapped to fixed virtual address 0xFEE00000 using pci_ioremap_io.
3. IO resource addresses to any PCIe device will be allocated in above range.
4. Driver for that PCIe device will write in the above range only for IO transaction to the device.
5. As per current IO translation programming, its one to one translation. It means PCIe translation unit will have both input and output address in the range of 0x1000 to 0xfffff.

Quoted 3 lines

Did I miss something, or if above statements are correct, then what I do not understand is how can designware PCIe translation unit accept input address in the range of 0x1000 to 0xfffff?

Hi Pratyush,

I think the part that you are missing is that there are four, not two address spaces involved:

Quoted 6 lines

1. The "bus" I/O port address, in the range between 0x1000 and 0xffff (65535). This is the address used in data frames in the actual bus transaction going over the PCIe wires. If you have more than one PCIe host bridge, each of them can normally have a range of up to 65536 addresses. The bus addresses get written into BAR registers of the device.

Correct. Got this point. So this address should be output of designware outbound address translation unit.

Quoted 11 lines

2. The Linux I/O port numbers, between 0x00000 and 0xfffff (1048575). This is a logical number space that aggregates all bus I/O port numbers. The first 4096 ports are normally reserved for ISA devices and are not available for PCIe resources. Each PCI or PCIe host bridge can have a 0x10000 byte naturally aligned range in here. The common case is that you have only one host bridge using the range 0x1000-0xffff. If you have more than one, there may be an offset between them, e.g. the second PCIe host may have io_offset set to 0x10000, which means that its bus range 0x1000-0xffff gets translated to Linux port number 0x11000-0x1ffff. Linux port numbers are visible e.g. in /proc/ioport and /dev/port

Ok, got it. To know more, which binding function maps this logical number space to address space in 4?

Quoted 5 lines

3. The CPU physical address space: Each PCIe host normally has its bus I/O port space mapped into PCU visible addresses, either hardwired or (in case of dw_pcie) set up using an outbound translation window. These addresses are what get passed to pci_ioremap_io.

OK.

Quoted 5 lines

4. The CPU virtual address space: This is an implementation detail of the kernel, which results in the Linux I/O port numbers 0x0-0xfffff to be mapped at 0xfee00000-0xfeefffff.

Quoted 7 lines
Quoted 3 lines

For example in SPEAr1340, physically RAM is mapped on above addresses. PCIe address translation unit can accept address only in the range of core addresses which are assigned to PCIe RC ie 0x80000000-0x8FFFFFFF.

Since this is a physical address, it corresponds to address space 3 in my list above, and the address you pick here is what you pass to pci_ioremap_io.

This is what I was expecting. But currently designware driver does not pass this address to pci_ioremap_io.
OK.. We will fix it and will send patch.

Regards
Pratyush

Quoted 2 lines

Arnd

#3 lore raw
Quoted 10 lines · Pratyush Anand
Quoted 7 lines

1. The "bus" I/O port address, in the range between 0x1000 and 0xffff (65535). This is the address used in data frames in the actual bus transaction going over the PCIe wires. If you have more than one PCIe host bridge, each of them can normally have a range of up to 65536 addresses. The bus addresses get written into BAR registers of the device.

Correct. Got this point. So this address should be output of designware outbound address translation unit.

Right.

Quoted 11 lines
Quoted 10 lines

2. The Linux I/O port numbers, between 0x00000 and 0xfffff (1048575). This is a logical number space that aggregates all bus I/O port numbers. The first 4096 ports are normally reserved for ISA devices and are not available for PCIe resources. Each PCI or PCIe host bridge can have a 0x10000 byte naturally aligned range in here. The common case is that you have only one host bridge using the range 0x1000-0xffff. If you have more than one, there may be an offset between them, e.g. the second PCIe host may have io_offset set to 0x10000, which means that its bus range 0x1000-0xffff gets translated to Linux port number 0x11000-0x1ffff. Linux port numbers are visible e.g. in /proc/ioport and /dev/port

Quoted 2 lines

Ok, got it. To know more, which binding function maps this logical number space to address space in 4?

There is just a linear offset between the two: PCI_IO_VIRT_BASE. This offset is applied in the __io() macro when converting from a port number to a pointer in various places in the kernel (inb/outb as well as pci_iomap/ioport_map), and the offset is added in pci_ioremap_io() when creating the page table entries.

Quoted 8 lines
Quoted 8 lines

4. The CPU virtual address space: This is an implementation detail of the kernel, which results in the Linux I/O port numbers 0x0-0xfffff to be mapped at 0xfee00000-0xfeefffff.

Quoted 3 lines

For example in SPEAr1340, physically RAM is mapped on above addresses. PCIe address translation unit can accept address only in the range of core addresses which are assigned to PCIe RC ie 0x80000000-0x8FFFFFFF.

Quoted 7 lines
Quoted 3 lines

Since this is a physical address, it corresponds to address space 3 in my list above, and the address you pick here is what you pass to pci_ioremap_io.

This is what I was expecting. But currently designware driver does not pass this address to pci_ioremap_io.
OK.. We will fix it and will send patch.

I think it does handle this correctly, look at

static int dw_pcie_setup(int nr, struct pci_sys_data *sys)

{
	...
        if (global_io_offset < SZ_1M && pp->config.io_size > 0) {
                sys->io_offset = global_io_offset - pp->config.io_bus_addr;
                pci_ioremap_io(sys->io_offset, pp->io.start);
                global_io_offset += SZ_64K;
                pci_add_resource_offset(&sys->resources, &pp->io,
                                        sys->io_offset);
        }
	...
}

I believe this does the right thing, but you have to put the correct translation into the 'ranges' property of the host bridge node in DT.

Arnd

#4 lore raw

Hi Arnd,

Quoted 3 lines · Arnd Bergmann
Quoted 1 line · Pratyush Anand
Quoted 1 line

[...]

Quoted 27 lines
Quoted 11 lines
Quoted 7 lines
Quoted 3 lines

For example in SPEAr1340, physically RAM is mapped on above addresses. PCIe address translation unit can accept address only in the range of core addresses which are assigned to PCIe RC ie 0x80000000-0x8FFFFFFF.

Since this is a physical address, it corresponds to address space 3 in my list above, and the address you pick here is what you pass to pci_ioremap_io.

This is what I was expecting. But currently designware driver does not pass this address to pci_ioremap_io.
OK.. We will fix it and will send patch.

I think it does handle this correctly, look at

static int dw_pcie_setup(int nr, struct pci_sys_data *sys)

{
	...
        if (global_io_offset < SZ_1M && pp->config.io_size > 0) {
                sys->io_offset = global_io_offset - pp->config.io_bus_addr;
                pci_ioremap_io(sys->io_offset, pp->io.start);
                global_io_offset += SZ_64K;
                pci_add_resource_offset(&sys->resources, &pp->io,
                                        sys->io_offset);
        }
	...
}
Quoted 2 lines

I believe this does the right thing, but you have to put the correct translation into the 'ranges' property of the host bridge node in DT.

May be not exactly. pp->io is the realio, and it is passed correctly to pci_add_resource_offset. But, as you had also
said that pci_ioremap_io will receive cpu physical address space as input, therefore I think following modification will be needed to work io transaction properly.

drivers/pci/host/pcie-designware.c +2 -2
index be6ce30..cf68632 100644
@@ -378,6 +378,7 @@ int __init dw_pcie_host_init(struct pcie_port *pp)
 					   + global_io_offset);
 			pp->config.io_size = resource_size(&pp->io);
 			pp->config.io_bus_addr = range.pci_addr;
+			pp->io_base = range.cpu_addr;
 		}
 		if (restype == IORESOURCE_MEM) {
 			of_pci_range_to_resource(&range, np, &pp->mem);
@@ -403,7 +404,6 @@ int __init dw_pcie_host_init(struct pcie_port *pp)
 
 	pp->cfg0_base = pp->cfg.start;
 	pp->cfg1_base = pp->cfg.start + pp->config.cfg0_size;
-	pp->io_base = pp->io.start;
 	pp->mem_base = pp->mem.start;
 
 	pp->va_cfg0_base = devm_ioremap(pp->dev, pp->cfg0_base,
@@ -667,7 +667,7 @@ static int dw_pcie_setup(int nr, struct pci_sys_data *sys)
 
 	if (global_io_offset < SZ_1M && pp->config.io_size > 0) {
 		sys->io_offset = global_io_offset - pp->config.io_bus_addr;
-		pci_ioremap_io(sys->io_offset, pp->io.start);
+		pci_ioremap_io(sys->io_offset, pp->io_base);
 		global_io_offset += SZ_64K;
 		pci_add_resource_offset(&sys->resources, &pp->io,
 					sys->io_offset);

Regards
Pratyush

Quoted 2 lines

Arnd

#5 lore raw
Quoted 15 lines · Pratyush Anand
Quoted 15 lines

I think it does handle this correctly, look at

static int dw_pcie_setup(int nr, struct pci_sys_data *sys)

{
	...
        if (global_io_offset < SZ_1M && pp->config.io_size > 0) {
                sys->io_offset = global_io_offset - pp->config.io_bus_addr;
                pci_ioremap_io(sys->io_offset, pp->io.start);
                global_io_offset += SZ_64K;
                pci_add_resource_offset(&sys->resources, &pp->io,
                                        sys->io_offset);
        }
	...
}
Quoted 8 lines
Quoted 2 lines

I believe this does the right thing, but you have to put the correct translation into the 'ranges' property of the host bridge node in DT.

May be not exactly. pp->io is the realio, and it is passed correctly to pci_add_resource_offset. But, as you had also
said that pci_ioremap_io will receive cpu physical address space as input, therefore I think following modification will be needed to work io transaction properly.

I see. I think you are right.

Quoted 13 lines
drivers/pci/host/pcie-designware.c +1 -0
index be6ce30..cf68632 100644
@@ -378,6 +378,7 @@ int __init dw_pcie_host_init(struct pcie_port *pp)
 					   + global_io_offset);
 			pp->config.io_size = resource_size(&pp->io);
 			pp->config.io_bus_addr = range.pci_addr;
+			pp->io_base = range.cpu_addr;
 		}
 		if (restype == IORESOURCE_MEM) {
 			of_pci_range_to_resource(&range, np, &pp->mem);

drivers/pci/host/pcie-designware.c int __init dw_pcie_host_init(struct pcie_port *pp) · line 409

Quoted 7 lines
drivers/pci/host/pcie-designware.c +0 -1
@@ -403,7 +404,6 @@ int __init dw_pcie_host_init(struct pcie_port *pp)
 
 	pp->cfg0_base = pp->cfg.start;
 	pp->cfg1_base = pp->cfg.start + pp->config.cfg0_size;
-	pp->io_base = pp->io.start;
 	pp->mem_base = pp->mem.start;
 
 	pp->va_cfg0_base = devm_ioremap(pp->dev, pp->cfg0_base,

This looks correct to me and it seems to also fix a bug in dw_pcie_prog_viewport_io_outbound if I read this correctly.

drivers/pci/host/pcie-designware.c static int dw_pcie_setup(int nr, struct pci_sys_data *sys) · line 673

Quoted 9 lines
drivers/pci/host/pcie-designware.c +1 -1
@@ -667,7 +667,7 @@ static int dw_pcie_setup(int nr, struct pci_sys_data *sys)
 
 	if (global_io_offset < SZ_1M && pp->config.io_size > 0) {
 		sys->io_offset = global_io_offset - pp->config.io_bus_addr;
-		pci_ioremap_io(sys->io_offset, pp->io.start);
+		pci_ioremap_io(sys->io_offset, pp->io_base);
 		global_io_offset += SZ_64K;
 		pci_add_resource_offset(&sys->resources, &pp->io,
 					sys->io_offset);

I think there is still a related bug in here: we should pass global_io_offset rather than sys->io_offset to pci_ioremap_io, so we map the new window into the first available spot in the Linux view of the I/O space, rather than passing a number that might be zero for any bus, if the 'ranges' are set up to have an identity mapping between Linux I/O spaces and PCI I/O spaces. You should be able to verify this by setting the I/O range for the bus to a random 4KB multiple in DT and observe that Linux start allocating ports from 0x1000 but the raw BAR values would contain the value you have chosen.

Arnd

#6 lore raw
Quoted 46 lines · Arnd Bergmann
Quoted 15 lines · Pratyush Anand
Quoted 15 lines

I think it does handle this correctly, look at

static int dw_pcie_setup(int nr, struct pci_sys_data *sys)

{
	...
        if (global_io_offset < SZ_1M && pp->config.io_size > 0) {
                sys->io_offset = global_io_offset - pp->config.io_bus_addr;
                pci_ioremap_io(sys->io_offset, pp->io.start);
                global_io_offset += SZ_64K;
                pci_add_resource_offset(&sys->resources, &pp->io,
                                        sys->io_offset);
        }
	...
}
Quoted 8 lines
Quoted 2 lines

I believe this does the right thing, but you have to put the correct translation into the 'ranges' property of the host bridge node in DT.

May be not exactly. pp->io is the realio, and it is passed correctly to pci_add_resource_offset. But, as you had also
said that pci_ioremap_io will receive cpu physical address space as input, therefore I think following modification will be needed to work io transaction properly.

I see. I think you are right.

Quoted 19 lines
drivers/pci/host/pcie-designware.c +1 -1
index be6ce30..cf68632 100644
@@ -378,6 +378,7 @@ int __init dw_pcie_host_init(struct pcie_port *pp)
 					   + global_io_offset);
 			pp->config.io_size = resource_size(&pp->io);
 			pp->config.io_bus_addr = range.pci_addr;
+			pp->io_base = range.cpu_addr;
 		}
 		if (restype == IORESOURCE_MEM) {
 			of_pci_range_to_resource(&range, np, &pp->mem);
@@ -403,7 +404,6 @@ int __init dw_pcie_host_init(struct pcie_port *pp)
 
 	pp->cfg0_base = pp->cfg.start;
 	pp->cfg1_base = pp->cfg.start + pp->config.cfg0_size;
-	pp->io_base = pp->io.start;
 	pp->mem_base = pp->mem.start;
 
Quoted 4 lines

drivers/pci/host/pcie-designware.c int __init dw_pcie_host_init(struct pcie_port *pp) · line 409

Quoted 1 line
drivers/pci/host/pcie-designware.c +0 -0
 	pp->va_cfg0_base = devm_ioremap(pp->dev, pp->cfg0_base,

This looks correct to me and it seems to also fix a bug in dw_pcie_prog_viewport_io_outbound if I read this correctly.

Yes, now outbound viewport for IO translation will get correct input address.

Quoted 19 lines

drivers/pci/host/pcie-designware.c static int dw_pcie_setup(int nr, struct pci_sys_data *sys) · line 673

Quoted 10 lines
drivers/pci/host/pcie-designware.c +1 -1
@@ -667,7 +667,7 @@ static int dw_pcie_setup(int nr, struct pci_sys_data *sys)
 
 	if (global_io_offset < SZ_1M && pp->config.io_size > 0) {
 		sys->io_offset = global_io_offset - pp->config.io_bus_addr;
-		pci_ioremap_io(sys->io_offset, pp->io.start);
+		pci_ioremap_io(sys->io_offset, pp->io_base);
 		global_io_offset += SZ_64K;
 		pci_add_resource_offset(&sys->resources, &pp->io,
 					sys->io_offset);

I think there is still a related bug in here: we should pass global_io_offset rather than sys->io_offset to pci_ioremap_io, so we map the new window into the first available spot in the Linux view of the I/O space, rather than passing a number that might be zero for any bus, if the 'ranges' are set up to have an identity mapping between Linux I/O spaces and PCI I/O spaces. You should be able to verify this by setting the I/O range for the bus to a random 4KB multiple in DT and observe that Linux start allocating ports from 0x1000 but the raw BAR values would contain the value you have chosen.

OK.

@ Jingoo, Mohit

Is it possible for you to test following patch.

drivers/pci/host/pcie-designware.c +2 -2
index be6ce30..b83f5e8 100644
@@ -378,6 +378,7 @@ int __init dw_pcie_host_init(struct pcie_port *pp)
 					   + global_io_offset);
 			pp->config.io_size = resource_size(&pp->io);
 			pp->config.io_bus_addr = range.pci_addr;
+			pp->io_base = range.cpu_addr;
 		}
 		if (restype == IORESOURCE_MEM) {
 			of_pci_range_to_resource(&range, np, &pp->mem);
@@ -403,7 +404,6 @@ int __init dw_pcie_host_init(struct pcie_port *pp)
 
 	pp->cfg0_base = pp->cfg.start;
 	pp->cfg1_base = pp->cfg.start + pp->config.cfg0_size;
-	pp->io_base = pp->io.start;
 	pp->mem_base = pp->mem.start;
 
 	pp->va_cfg0_base = devm_ioremap(pp->dev, pp->cfg0_base,
@@ -667,7 +667,7 @@ static int dw_pcie_setup(int nr, struct pci_sys_data *sys)
 
 	if (global_io_offset < SZ_1M && pp->config.io_size > 0) {
 		sys->io_offset = global_io_offset - pp->config.io_bus_addr;
-		pci_ioremap_io(sys->io_offset, pp->io.start);
+		pci_ioremap_io(global_io_offset, pp->io_base);
 		global_io_offset += SZ_64K;
 		pci_add_resource_offset(&sys->resources, &pp->io,
 					sys->io_offset);

Regards
Pratyush

Quoted 2 lines

Arnd

#7 lore raw
Quoted 75 lines · Pratyush Anand
Quoted 46 lines · Arnd Bergmann
Quoted 15 lines · Pratyush Anand
Quoted 15 lines

I think it does handle this correctly, look at

static int dw_pcie_setup(int nr, struct pci_sys_data *sys)

{
	...
        if (global_io_offset < SZ_1M && pp->config.io_size > 0) {
                sys->io_offset = global_io_offset - pp->config.io_bus_addr;
                pci_ioremap_io(sys->io_offset, pp->io.start);
                global_io_offset += SZ_64K;
                pci_add_resource_offset(&sys->resources, &pp->io,
                                        sys->io_offset);
        }
	...
}
Quoted 8 lines
Quoted 2 lines

I believe this does the right thing, but you have to put the correct translation into the 'ranges' property of the host bridge node in DT.

May be not exactly. pp->io is the realio, and it is passed correctly to pci_add_resource_offset. But, as you had also
said that pci_ioremap_io will receive cpu physical address space as input, therefore I think following modification will be needed to work io transaction properly.

I see. I think you are right.

Quoted 19 lines
drivers/pci/host/pcie-designware.c +1 -1
index be6ce30..cf68632 100644
@@ -378,6 +378,7 @@ int __init dw_pcie_host_init(struct pcie_port *pp)
 					   + global_io_offset);
 			pp->config.io_size = resource_size(&pp->io);
 			pp->config.io_bus_addr = range.pci_addr;
+			pp->io_base = range.cpu_addr;
 		}
 		if (restype == IORESOURCE_MEM) {
 			of_pci_range_to_resource(&range, np, &pp->mem);
@@ -403,7 +404,6 @@ int __init dw_pcie_host_init(struct pcie_port *pp)
 
 	pp->cfg0_base = pp->cfg.start;
 	pp->cfg1_base = pp->cfg.start + pp->config.cfg0_size;
-	pp->io_base = pp->io.start;
 	pp->mem_base = pp->mem.start;
 
Quoted 4 lines

drivers/pci/host/pcie-designware.c int __init dw_pcie_host_init(struct pcie_port *pp) · line 409

Quoted 1 line
drivers/pci/host/pcie-designware.c +0 -0
 	pp->va_cfg0_base = devm_ioremap(pp->dev, pp->cfg0_base,

This looks correct to me and it seems to also fix a bug in dw_pcie_prog_viewport_io_outbound if I read this correctly.

Yes, now outbound viewport for IO translation will get correct input address.

Quoted 19 lines

drivers/pci/host/pcie-designware.c static int dw_pcie_setup(int nr, struct pci_sys_data *sys) · line 673

Quoted 10 lines
drivers/pci/host/pcie-designware.c +1 -1
@@ -667,7 +667,7 @@ static int dw_pcie_setup(int nr, struct pci_sys_data *sys)
 
 	if (global_io_offset < SZ_1M && pp->config.io_size > 0) {
 		sys->io_offset = global_io_offset - pp->config.io_bus_addr;
-		pci_ioremap_io(sys->io_offset, pp->io.start);
+		pci_ioremap_io(sys->io_offset, pp->io_base);
 		global_io_offset += SZ_64K;
 		pci_add_resource_offset(&sys->resources, &pp->io,
 					sys->io_offset);

I think there is still a related bug in here: we should pass global_io_offset rather than sys->io_offset to pci_ioremap_io, so we map the new window into the first available spot in the Linux view of the I/O space, rather than passing a number that might be zero for any bus, if the 'ranges' are set up to have an identity mapping between Linux I/O spaces and PCI I/O spaces. You should be able to verify this by setting the I/O range for the bus to a random 4KB multiple in DT and observe that Linux start allocating ports from 0x1000 but the raw BAR values would contain the value you have chosen.

Quoted 5 lines

OK.

@ Jingoo, Mohit

Is it possible for you to test following patch.

Hi Pratyush Anand,

This patch works properly on Exynos platform with two different NICs. Also, I am tracing this discussion.
Thank you for sending your patch.

Best regards,
Jingoo Han

Quoted 30 lines
drivers/pci/host/pcie-designware.c +2 -2
index be6ce30..b83f5e8 100644
@@ -378,6 +378,7 @@ int __init dw_pcie_host_init(struct pcie_port *pp)
 					   + global_io_offset);
 			pp->config.io_size = resource_size(&pp->io);
 			pp->config.io_bus_addr = range.pci_addr;
+			pp->io_base = range.cpu_addr;
 		}
 		if (restype == IORESOURCE_MEM) {
 			of_pci_range_to_resource(&range, np, &pp->mem);
@@ -403,7 +404,6 @@ int __init dw_pcie_host_init(struct pcie_port *pp)
 
 	pp->cfg0_base = pp->cfg.start;
 	pp->cfg1_base = pp->cfg.start + pp->config.cfg0_size;
-	pp->io_base = pp->io.start;
 	pp->mem_base = pp->mem.start;
 
 	pp->va_cfg0_base = devm_ioremap(pp->dev, pp->cfg0_base,
@@ -667,7 +667,7 @@ static int dw_pcie_setup(int nr, struct pci_sys_data *sys)
 
 	if (global_io_offset < SZ_1M && pp->config.io_size > 0) {
 		sys->io_offset = global_io_offset - pp->config.io_bus_addr;
-		pci_ioremap_io(sys->io_offset, pp->io.start);
+		pci_ioremap_io(global_io_offset, pp->io_base);
 		global_io_offset += SZ_64K;
 		pci_add_resource_offset(&sys->resources, &pp->io,
 					sys->io_offset);
#8 lore raw

Hello Pratyush,

Quoted 88 lines

-----Original Message-----
From: Jingoo Han [mailto:[email protected]]
Sent: Tuesday, December 10, 2013 10:55 AM
To: Pratyush ANAND; 'Arnd Bergmann'; Mohit KUMAR DCG
Cc: 'Marek Vasut'; 'Richard Zhu'; 'Kishon Vijay Abraham I'; linux- [email protected]; 'Tim Harvey'; 'Jingoo Han'
Subject: Re: [Query/Discussion]: IO translation with designware PCIe controller

Quoted 16 lines · Pratyush Anand
Quoted 15 lines · Arnd Bergmann
Quoted 14 lines · Pratyush Anand
Quoted 14 lines

I think it does handle this correctly, look at

static int dw_pcie_setup(int nr, struct pci_sys_data *sys) {
	...
        if (global_io_offset < SZ_1M && pp->config.io_size > 0) {
                sys->io_offset = global_io_offset - pp->config.io_bus_addr;
                pci_ioremap_io(sys->io_offset, pp->io.start);
                global_io_offset += SZ_64K;
                pci_add_resource_offset(&sys->resources, &pp->io,
                                        sys->io_offset);
        }
	...
}
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

I believe this does the right thing, but you have to put the correct translation into the 'ranges' property of the host bridge node

in DT.

Quoted 7 lines
Quoted 7 lines
Quoted 6 lines

May be not exactly. pp->io is the realio, and it is passed correctly to pci_add_resource_offset. But, as you had also said that pci_ioremap_io will receive cpu physical address space as input, therefore I think following modification will be needed to work io transaction properly.

Quoted 8 lines
Quoted 8 lines

I see. I think you are right.

drivers/pci/host/pcie-designware.c int __init dw_pcie_host_init(struct pcie_port

Quoted 6 lines
drivers/pci/host/pcie-designware.c +0 -0
index be6ce30..cf68632 100644
@@ -378,6 +378,7 @@ int __init dw_pcie_host_init(struct pcie_port

*pp)

Quoted 7 lines
Quoted 7 lines

drivers/pci/host/pcie-designware.c int __init dw_pcie_host_init(struct pcie_port *pp) · line 384

Quoted 7 lines
drivers/pci/host/pcie-designware.c +1 -0
 					   + global_io_offset);
 			pp->config.io_size = resource_size(&pp->io);
 			pp->config.io_bus_addr = range.pci_addr;
+			pp->io_base = range.cpu_addr;
 		}
 		if (restype == IORESOURCE_MEM) {
 			of_pci_range_to_resource(&range, np, &pp->mem);

@@ -403,7

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

+404,6 @@ int __init dw_pcie_host_init(struct pcie_port *pp)

drivers/pci/host/pcie-designware.c +0 -1
 	pp->cfg0_base = pp->cfg.start;
 	pp->cfg1_base = pp->cfg.start + pp->config.cfg0_size;
-	pp->io_base = pp->io.start;
 	pp->mem_base = pp->mem.start;
Quoted 28 lines
Quoted 4 lines
Quoted 1 line

pp->va_cfg0_base = devm_ioremap(pp->dev, pp->cfg0_base,

This looks correct to me and it seems to also fix a bug in dw_pcie_prog_viewport_io_outbound if I read this correctly.

Yes, now outbound viewport for IO translation will get correct input address.

Quoted 20 lines
Quoted 11 lines
drivers/pci/host/pcie-designware.c +1 -1
@@ -667,7 +667,7 @@ static int dw_pcie_setup(int nr, struct
 pci_sys_data *sys)
 
 	if (global_io_offset < SZ_1M && pp->config.io_size > 0) {
 		sys->io_offset = global_io_offset - pp->config.io_bus_addr;
-		pci_ioremap_io(sys->io_offset, pp->io.start);
+		pci_ioremap_io(sys->io_offset, pp->io_base);
 		global_io_offset += SZ_64K;
 		pci_add_resource_offset(&sys->resources, &pp->io,

sys->io_offset);

I think there is still a related bug in here: we should pass global_io_offset rather than sys->io_offset to pci_ioremap_io, so we map the new window into the first available spot in the Linux view of the I/O space, rather than passing a number that might be zero for any bus, if the 'ranges' are set up to have an identity mapping between Linux I/O spaces and PCI I/O spaces. You should be able to verify this by setting the I/O range for the bus to a random 4KB multiple in DT and observe that Linux start allocating ports from 0x1000

Quoted 5 lines

but the raw BAR values would contain the value you have chosen.

Quoted 4 lines

OK.

@ Jingoo, Mohit

- Now its working properly for SPEAr1310 tested with directly connected EP(xhc cards) as well as EP Connected through switch(Lecroy PTC in AIC mode). Without this patch my EP devices were not working when connected through Switch.

Thanks
Mohit

Quoted 46 lines
Quoted 2 lines

Is it possible for you to test following patch.

Hi Pratyush Anand,

This patch works properly on Exynos platform with two different NICs. Also, I am tracing this discussion.
Thank you for sending your patch.

Best regards,
Jingoo Han

Quoted 7 lines
drivers/pci/host/pcie-designware.c +0 -0
index be6ce30..b83f5e8 100644

drivers/pci/host/pcie-designware.c int __init dw_pcie_host_init(struct pcie_port *pp) · line 384

Quoted 7 lines
drivers/pci/host/pcie-designware.c +1 -0
@@ -378,6 +378,7 @@ int __init dw_pcie_host_init(struct pcie_port *pp)
 					   + global_io_offset);
 			pp->config.io_size = resource_size(&pp->io);
 			pp->config.io_bus_addr = range.pci_addr;
+			pp->io_base = range.cpu_addr;
 		}
 		if (restype == IORESOURCE_MEM) {
 			of_pci_range_to_resource(&range, np, &pp->mem);

@@ -403,7 +404,6

drivers/pci/host/pcie-designware.c int __init dw_pcie_host_init(struct pcie_port *pp)

Quoted 8 lines

@@ int __init dw_pcie_host_init(struct pcie_port *pp)

drivers/pci/host/pcie-designware.c +0 -1
 	pp->cfg0_base = pp->cfg.start;
 	pp->cfg1_base = pp->cfg.start + pp->config.cfg0_size;
-	pp->io_base = pp->io.start;
 	pp->mem_base = pp->mem.start;
 
 	pp->va_cfg0_base = devm_ioremap(pp->dev, pp->cfg0_base, @@ -

667,7

Quoted 9 lines

+667,7 @@ static int dw_pcie_setup(int nr, struct pci_sys_data *sys)

drivers/pci/host/pcie-designware.c +1 -1
 	if (global_io_offset < SZ_1M && pp->config.io_size > 0) {
 		sys->io_offset = global_io_offset - pp->config.io_bus_addr;
-		pci_ioremap_io(sys->io_offset, pp->io.start);
+		pci_ioremap_io(global_io_offset, pp->io_base);
 		global_io_offset += SZ_64K;
 		pci_add_resource_offset(&sys->resources, &pp->io,
 					sys->io_offset);
#9 lore raw

Hi Tim,

Quoted 92 lines · Mohit KUMAR DCG

Hello Pratyush,

Quoted 90 lines

-----Original Message-----
From: Jingoo Han [mailto:[email protected]]
Sent: Tuesday, December 10, 2013 10:55 AM
To: Pratyush ANAND; 'Arnd Bergmann'; Mohit KUMAR DCG
Cc: 'Marek Vasut'; 'Richard Zhu'; 'Kishon Vijay Abraham I'; linux- [email protected]; 'Tim Harvey'; 'Jingoo Han'
Subject: Re: [Query/Discussion]: IO translation with designware PCIe controller

Quoted 16 lines · Pratyush Anand
Quoted 15 lines · Arnd Bergmann
Quoted 14 lines · Pratyush Anand
Quoted 14 lines

I think it does handle this correctly, look at

static int dw_pcie_setup(int nr, struct pci_sys_data *sys) {
	...
        if (global_io_offset < SZ_1M && pp->config.io_size > 0) {
                sys->io_offset = global_io_offset - pp->config.io_bus_addr;
                pci_ioremap_io(sys->io_offset, pp->io.start);
                global_io_offset += SZ_64K;
                pci_add_resource_offset(&sys->resources, &pp->io,
                                        sys->io_offset);
        }
	...
}
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

I believe this does the right thing, but you have to put the correct translation into the 'ranges' property of the host bridge node

in DT.

Quoted 7 lines
Quoted 7 lines
Quoted 6 lines

May be not exactly. pp->io is the realio, and it is passed correctly to pci_add_resource_offset. But, as you had also said that pci_ioremap_io will receive cpu physical address space as input, therefore I think following modification will be needed to work io transaction properly.

Quoted 8 lines
Quoted 8 lines

I see. I think you are right.

drivers/pci/host/pcie-designware.c int __init dw_pcie_host_init(struct pcie_port

Quoted 6 lines
drivers/pci/host/pcie-designware.c +0 -0
index be6ce30..cf68632 100644
@@ -378,6 +378,7 @@ int __init dw_pcie_host_init(struct pcie_port

*pp)

Quoted 7 lines
Quoted 7 lines

drivers/pci/host/pcie-designware.c int __init dw_pcie_host_init(struct pcie_port *pp) · line 384

Quoted 7 lines
drivers/pci/host/pcie-designware.c +1 -0
 					   + global_io_offset);
 			pp->config.io_size = resource_size(&pp->io);
 			pp->config.io_bus_addr = range.pci_addr;
+			pp->io_base = range.cpu_addr;
 		}
 		if (restype == IORESOURCE_MEM) {
 			of_pci_range_to_resource(&range, np, &pp->mem);

@@ -403,7

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

+404,6 @@ int __init dw_pcie_host_init(struct pcie_port *pp)

drivers/pci/host/pcie-designware.c +0 -1
 	pp->cfg0_base = pp->cfg.start;
 	pp->cfg1_base = pp->cfg.start + pp->config.cfg0_size;
-	pp->io_base = pp->io.start;
 	pp->mem_base = pp->mem.start;
Quoted 28 lines
Quoted 4 lines
Quoted 1 line

pp->va_cfg0_base = devm_ioremap(pp->dev, pp->cfg0_base,

This looks correct to me and it seems to also fix a bug in dw_pcie_prog_viewport_io_outbound if I read this correctly.

Yes, now outbound viewport for IO translation will get correct input address.

Quoted 20 lines
Quoted 11 lines
drivers/pci/host/pcie-designware.c +1 -1
@@ -667,7 +667,7 @@ static int dw_pcie_setup(int nr, struct
 pci_sys_data *sys)
 
 	if (global_io_offset < SZ_1M && pp->config.io_size > 0) {
 		sys->io_offset = global_io_offset - pp->config.io_bus_addr;
-		pci_ioremap_io(sys->io_offset, pp->io.start);
+		pci_ioremap_io(sys->io_offset, pp->io_base);
 		global_io_offset += SZ_64K;
 		pci_add_resource_offset(&sys->resources, &pp->io,

sys->io_offset);

I think there is still a related bug in here: we should pass global_io_offset rather than sys->io_offset to pci_ioremap_io, so we map the new window into the first available spot in the Linux view of the I/O space, rather than passing a number that might be zero for any bus, if the 'ranges' are set up to have an identity mapping between Linux I/O spaces and PCI I/O spaces. You should be able to verify this by setting the I/O range for the bus to a random 4KB multiple in DT and observe that Linux start allocating ports from 0x1000

but the raw BAR values would contain the value you have chosen.

Quoted 1 line
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines

OK.

@ Jingoo, Mohit

- Now its working properly for SPEAr1310 tested with directly connected EP(xhc cards) as well as EP Connected through switch(Lecroy PTC in AIC mode). Without this patch my EP devices were not working when connected through Switch.

Can you also check if this patch resolves your issue, and you are able to work with switch without commenting dw_pcie_prog_viewport_io_outbound.

Regards
Pratyush

Quoted 50 lines

Thanks
Mohit

Quoted 46 lines
Quoted 2 lines

Is it possible for you to test following patch.

Hi Pratyush Anand,

This patch works properly on Exynos platform with two different NICs. Also, I am tracing this discussion.
Thank you for sending your patch.

Best regards,
Jingoo Han

Quoted 7 lines
drivers/pci/host/pcie-designware.c +0 -0
index be6ce30..b83f5e8 100644

drivers/pci/host/pcie-designware.c int __init dw_pcie_host_init(struct pcie_port *pp) · line 384

Quoted 7 lines
drivers/pci/host/pcie-designware.c +1 -0
@@ -378,6 +378,7 @@ int __init dw_pcie_host_init(struct pcie_port *pp)
 					   + global_io_offset);
 			pp->config.io_size = resource_size(&pp->io);
 			pp->config.io_bus_addr = range.pci_addr;
+			pp->io_base = range.cpu_addr;
 		}
 		if (restype == IORESOURCE_MEM) {
 			of_pci_range_to_resource(&range, np, &pp->mem);

@@ -403,7 +404,6

drivers/pci/host/pcie-designware.c int __init dw_pcie_host_init(struct pcie_port *pp)

Quoted 8 lines

@@ int __init dw_pcie_host_init(struct pcie_port *pp)

drivers/pci/host/pcie-designware.c +0 -1
 	pp->cfg0_base = pp->cfg.start;
 	pp->cfg1_base = pp->cfg.start + pp->config.cfg0_size;
-	pp->io_base = pp->io.start;
 	pp->mem_base = pp->mem.start;
 
 	pp->va_cfg0_base = devm_ioremap(pp->dev, pp->cfg0_base, @@ -

667,7

Quoted 9 lines

+667,7 @@ static int dw_pcie_setup(int nr, struct pci_sys_data *sys)

drivers/pci/host/pcie-designware.c +1 -1
 	if (global_io_offset < SZ_1M && pp->config.io_size > 0) {
 		sys->io_offset = global_io_offset - pp->config.io_bus_addr;
-		pci_ioremap_io(sys->io_offset, pp->io.start);
+		pci_ioremap_io(global_io_offset, pp->io_base);
 		global_io_offset += SZ_64K;
 		pci_add_resource_offset(&sys->resources, &pp->io,
 					sys->io_offset);
#10 lore raw
In reply to Mohit KUMAR DCG
Quoted 82 lines · Mohit KUMAR DCG
Quoted 81 lines · Jingoo Han
Quoted 16 lines · Pratyush Anand
Quoted 15 lines · Arnd Bergmann
Quoted 14 lines · Pratyush Anand
Quoted 14 lines

I think it does handle this correctly, look at

static int dw_pcie_setup(int nr, struct pci_sys_data *sys) {
	...
        if (global_io_offset < SZ_1M && pp->config.io_size > 0) {
                sys->io_offset = global_io_offset - pp->config.io_bus_addr;
                pci_ioremap_io(sys->io_offset, pp->io.start);
                global_io_offset += SZ_64K;
                pci_add_resource_offset(&sys->resources, &pp->io,
                                        sys->io_offset);
        }
	...
}
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

I believe this does the right thing, but you have to put the correct translation into the 'ranges' property of the host bridge node

in DT.

Quoted 7 lines
Quoted 7 lines
Quoted 6 lines

May be not exactly. pp->io is the realio, and it is passed correctly to pci_add_resource_offset. But, as you had also said that pci_ioremap_io will receive cpu physical address space as input, therefore I think following modification will be needed to work io transaction properly.

Quoted 8 lines
Quoted 8 lines

I see. I think you are right.

drivers/pci/host/pcie-designware.c int __init dw_pcie_host_init(struct pcie_port

Quoted 6 lines
drivers/pci/host/pcie-designware.c +0 -0
index be6ce30..cf68632 100644
@@ -378,6 +378,7 @@ int __init dw_pcie_host_init(struct pcie_port

*pp)

Quoted 7 lines
Quoted 7 lines

drivers/pci/host/pcie-designware.c int __init dw_pcie_host_init(struct pcie_port · line 384

Quoted 7 lines
drivers/pci/host/pcie-designware.c +1 -0
 					   + global_io_offset);
 			pp->config.io_size = resource_size(&pp->io);
 			pp->config.io_bus_addr = range.pci_addr;
+			pp->io_base = range.cpu_addr;
 		}
 		if (restype == IORESOURCE_MEM) {
 			of_pci_range_to_resource(&range, np, &pp->mem);

@@ -403,7

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

+404,6 @@ int __init dw_pcie_host_init(struct pcie_port *pp)

drivers/pci/host/pcie-designware.c +0 -1
 	pp->cfg0_base = pp->cfg.start;
 	pp->cfg1_base = pp->cfg.start + pp->config.cfg0_size;
-	pp->io_base = pp->io.start;
 	pp->mem_base = pp->mem.start;
Quoted 28 lines
Quoted 4 lines
Quoted 1 line

pp->va_cfg0_base = devm_ioremap(pp->dev, pp->cfg0_base,

This looks correct to me and it seems to also fix a bug in dw_pcie_prog_viewport_io_outbound if I read this correctly.

Yes, now outbound viewport for IO translation will get correct input address.

Quoted 20 lines
Quoted 11 lines
drivers/pci/host/pcie-designware.c +1 -1
@@ -667,7 +667,7 @@ static int dw_pcie_setup(int nr, struct
 pci_sys_data *sys)
 
 	if (global_io_offset < SZ_1M && pp->config.io_size > 0) {
 		sys->io_offset = global_io_offset - pp->config.io_bus_addr;
-		pci_ioremap_io(sys->io_offset, pp->io.start);
+		pci_ioremap_io(sys->io_offset, pp->io_base);
 		global_io_offset += SZ_64K;
 		pci_add_resource_offset(&sys->resources, &pp->io,

sys->io_offset);

I think there is still a related bug in here: we should pass global_io_offset rather than sys->io_offset to pci_ioremap_io, so we map the new window into the first available spot in the Linux view of the I/O space, rather than passing a number that might be zero for any bus, if the 'ranges' are set up to have an identity mapping between Linux I/O spaces and PCI I/O spaces. You should be able to verify this by setting the I/O range for the bus to a random 4KB multiple in DT and observe that Linux start allocating ports from 0x1000

but the raw BAR values would contain the value you have chosen.

Quoted 1 line
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines

OK.

@ Jingoo, Mohit

- Now its working properly for SPEAr1310 tested with directly connected EP(xhc cards) as well as EP Connected through switch(Lecroy PTC in AIC mode). Without this patch my EP devices were not working
when
connected through Switch.

Hi Marek, Richard,

How about testing the following Pratyush's patch on imx6 platform with the Pericom PI7C9X2G303EL PCIe switch?

As Mohit tested, it looks to resolve the i.MX6 problem when the PCIe switch is connected to the i.MX6 platform.

Best regards,
Jingoo Han

[.....]

drivers/pci/host/pcie-designware.c +2 -2
index be6ce30..b83f5e8 100644
@@ -378,6 +378,7 @@ int __init dw_pcie_host_init(struct pcie_port *pp)
 					   + global_io_offset);
 			pp->config.io_size = resource_size(&pp->io);
 			pp->config.io_bus_addr = range.pci_addr;
+			pp->io_base = range.cpu_addr;
 		}
 		if (restype == IORESOURCE_MEM) {
 			of_pci_range_to_resource(&range, np, &pp->mem);
@@ -403,7 +404,6 @@ int __init dw_pcie_host_init(struct pcie_port *pp)
 
 	pp->cfg0_base = pp->cfg.start;
 	pp->cfg1_base = pp->cfg.start + pp->config.cfg0_size;
-	pp->io_base = pp->io.start;
 	pp->mem_base = pp->mem.start;
 
 	pp->va_cfg0_base = devm_ioremap(pp->dev, pp->cfg0_base,
@@ -667,7 +667,7 @@ static int dw_pcie_setup(int nr, struct pci_sys_data *sys)
 
 	if (global_io_offset < SZ_1M && pp->config.io_size > 0) {
 		sys->io_offset = global_io_offset - pp->config.io_bus_addr;
-		pci_ioremap_io(sys->io_offset, pp->io.start);
+		pci_ioremap_io(global_io_offset, pp->io_base);
 		global_io_offset += SZ_64K;
 		pci_add_resource_offset(&sys->resources, &pp->io,
 					sys->io_offset);
#11 lore raw

Hi Jinhoo:

Quoted 108 lines

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Jingoo Han
Sent: Tuesday, December 10, 2013 3:03 PM
To: 'Marek Vasut'; Zhu Richard-R65037
Cc: 'Mohit KUMAR DCG'; 'Pratyush ANAND'; 'Arnd Bergmann'; 'Kishon Vijay Abraham I'; [email protected]; 'Tim Harvey'; 'Jingoo Han'
Subject: Re: [Query/Discussion]: IO translation with designware PCIe controller

Quoted 86 lines · Mohit KUMAR DCG
Quoted 9 lines · Jingoo Han
Quoted 8 lines · Pratyush Anand
Quoted 7 lines · Arnd Bergmann
Quoted 6 lines · Pratyush Anand
Quoted 6 lines

I think it does handle this correctly, look at

static int dw_pcie_setup(int nr, struct pci_sys_data *sys) {
	...
        if (global_io_offset < SZ_1M && pp->config.io_size > 0) {
                sys->io_offset = global_io_offset - pp-

config.io_bus_addr;

Quoted 75 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
                pci_ioremap_io(sys->io_offset, pp->io.start);
                global_io_offset += SZ_64K;
                pci_add_resource_offset(&sys->resources, &pp->io,
                                        sys->io_offset);
        }
	...
}

I believe this does the right thing, but you have to put the correct translation into the 'ranges' property of the host bridge node

in DT.

Quoted 9 lines
Quoted 9 lines
Quoted 6 lines

May be not exactly. pp->io is the realio, and it is passed correctly to pci_add_resource_offset. But, as you had also said that pci_ioremap_io will receive cpu physical address space as input, therefore I think following modification will be needed to work io transaction properly.

I see. I think you are right.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
drivers/pci/host/pcie-designware.c +0 -0
index be6ce30..cf68632 100644
@@ -378,6 +378,7 @@ int __init dw_pcie_host_init(struct

pcie_port

*pp)

Quoted 7 lines
Quoted 7 lines

drivers/pci/host/pcie-designware.c int __init dw_pcie_host_init(struct pcie_port *pp) · line 384

Quoted 7 lines
drivers/pci/host/pcie-designware.c +1 -0
 					   + global_io_offset);
 			pp->config.io_size = resource_size(&pp->io);
 			pp->config.io_bus_addr = range.pci_addr;
+			pp->io_base = range.cpu_addr;
 		}
 		if (restype == IORESOURCE_MEM) {
 			of_pci_range_to_resource(&range, np, &pp->mem);

@@ -403,7

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

+404,6 @@ int __init dw_pcie_host_init(struct pcie_port *pp)

drivers/pci/host/pcie-designware.c +0 -1
 	pp->cfg0_base = pp->cfg.start;
 	pp->cfg1_base = pp->cfg.start + pp->config.cfg0_size;
-	pp->io_base = pp->io.start;
 	pp->mem_base = pp->mem.start;
Quoted 29 lines
Quoted 4 lines
Quoted 1 line

pp->va_cfg0_base = devm_ioremap(pp->dev, pp->cfg0_base,

This looks correct to me and it seems to also fix a bug in dw_pcie_prog_viewport_io_outbound if I read this correctly.

Yes, now outbound viewport for IO translation will get correct input address.

Quoted 21 lines
Quoted 11 lines
drivers/pci/host/pcie-designware.c +1 -1
@@ -667,7 +667,7 @@ static int dw_pcie_setup(int nr, struct
 pci_sys_data *sys)
 
 	if (global_io_offset < SZ_1M && pp->config.io_size > 0) {
 		sys->io_offset = global_io_offset - pp->config.io_bus_addr;
-		pci_ioremap_io(sys->io_offset, pp->io.start);
+		pci_ioremap_io(sys->io_offset, pp->io_base);
 		global_io_offset += SZ_64K;
 		pci_add_resource_offset(&sys->resources, &pp->io,

sys->io_offset);

I think there is still a related bug in here: we should pass global_io_offset rather than sys->io_offset to pci_ioremap_io, so we map the new window into the first available spot in the Linux view of the I/O space, rather than passing a number that might be zero for any bus, if the 'ranges' are set up to have an identity mapping between Linux I/O spaces and PCI I/O spaces. You should be able to verify this by setting the I/O range for the bus to a random 4KB multiple in DT and observe that Linux start allocating ports from 0x1000

but the raw BAR values would contain the value you have chosen.

Quoted 1 line
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines

OK.

@ Jingoo, Mohit

- Now its working properly for SPEAr1310 tested with directly connected EP(xhc cards) as well as EP Connected through switch(Lecroy PTC in AIC mode). Without this patch my EP devices were not working when connected through Switch.

Hi Marek, Richard,

Quoted 9 lines

How about testing the following Pratyush's patch on imx6 platform with the Pericom PI7C9X2G303EL PCIe switch?

As Mohit tested, it looks to resolve the i.MX6 problem when the PCIe switch is connected to the i.MX6 platform.

Best regards,
Jingoo Han

[Richard] Sorry to reply late, I went to hospital in the past two days. This patch is great, I tested at my side, it works, thanks a lot.

Quoted 34 lines

[.....]

drivers/pci/host/pcie- renamed from drivers/pci/host/pcie-designware.c +0 -0

designware.c
index be6ce30..b83f5e8 100644

drivers/pci/host/pcie-designware.c +2 -1
@@ -378,6 +378,7 @@ int __init dw_pcie_host_init(struct pcie_port *pp)
 					   + global_io_offset);
 			pp->config.io_size = resource_size(&pp->io);
 			pp->config.io_bus_addr = range.pci_addr;
+			pp->io_base = range.cpu_addr;
 		}
 		if (restype == IORESOURCE_MEM) {
 			of_pci_range_to_resource(&range, np, &pp->mem); @@ -403,7
+404,6 @@ int __init dw_pcie_host_init(struct pcie_port *pp)
 
 	pp->cfg0_base = pp->cfg.start;
 	pp->cfg1_base = pp->cfg.start + pp->config.cfg0_size;
-	pp->io_base = pp->io.start;
 	pp->mem_base = pp->mem.start;
 
 	pp->va_cfg0_base = devm_ioremap(pp->dev, pp->cfg0_base, @@ -667,7 +667,7

@@ static int dw_pcie_setup(int nr, struct pci_sys_data *sys)

drivers/pci/host/pcie-designware.c +1 -2
 	if (global_io_offset < SZ_1M && pp->config.io_size > 0) {
 		sys->io_offset = global_io_offset - pp->config.io_bus_addr;
-		pci_ioremap_io(sys->io_offset, pp->io.start);
+		pci_ioremap_io(global_io_offset, pp->io_base);
 		global_io_offset += SZ_64K;
 		pci_add_resource_offset(&sys->resources, &pp->io,
 					sys->io_offset);
 
--
Quoted 4 lines

To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to [email protected] More majordomo info at
http://vger.kernel.org/majordomo-info.html

Best Regards
Richard Zhu

#12 lore raw
In reply to Pratyush Anand
Quoted 81 lines · Pratyush Anand

Hi Arnd,

Quoted 1 line · Arnd Bergmann

On Friday 06 December 2013, Pratyush Anand wrote:

[...]

Quoted 34 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

For example in SPEAr1340, physically RAM is mapped on above addresses. PCIe address translation unit can accept address only in the range of core addresses which are assigned to PCIe RC ie 0x80000000-0x8FFFFFFF.

Quoted 7 lines
Quoted 3 lines

Since this is a physical address, it corresponds to address space 3 in my list above, and the address you pick here is what you pass to pci_ioremap_io.

This is what I was expecting. But currently designware driver does not pass this address to pci_ioremap_io.
OK.. We will fix it and will send patch.

I think it does handle this correctly, look at

static int dw_pcie_setup(int nr, struct pci_sys_data *sys) {

	...
	
        if (global_io_offset < SZ_1M && pp->config.io_size > 0) {
        
                sys->io_offset = global_io_offset -
                pp->config.io_bus_addr; pci_ioremap_io(sys->io_offset,
                pp->io.start);
                global_io_offset += SZ_64K;
                pci_add_resource_offset(&sys->resources, &pp->io,
                
                                        sys->io_offset);
        
        }
	
	...
Quoted 4 lines

}

I believe this does the right thing, but you have to put the correct translation into the 'ranges' property of the host bridge node in DT.

May be not exactly. pp->io is the realio, and it is passed correctly to pci_add_resource_offset. But, as you had also
said that pci_ioremap_io will receive cpu physical address space as input, therefore I think following modification will be needed to work io transaction properly.

(quoted diff) +0 -0

b/drivers/pci/host/pcie-designware.c index be6ce30..cf68632 100644

drivers/pci/host/pcie-designware.c +2 -2
@@ -378,6 +378,7 @@ int __init dw_pcie_host_init(struct pcie_port *pp)
 					   + global_io_offset);
 			pp->config.io_size = resource_size(&pp->io);
 			pp->config.io_bus_addr = range.pci_addr;
+			pp->io_base = range.cpu_addr;
 		}
 		if (restype == IORESOURCE_MEM) {
 			of_pci_range_to_resource(&range, np, &pp->mem);
@@ -403,7 +404,6 @@ int __init dw_pcie_host_init(struct pcie_port *pp)
 
 	pp->cfg0_base = pp->cfg.start;
 	pp->cfg1_base = pp->cfg.start + pp->config.cfg0_size;
-	pp->io_base = pp->io.start;
 	pp->mem_base = pp->mem.start;
 
 	pp->va_cfg0_base = devm_ioremap(pp->dev, pp->cfg0_base,
@@ -667,7 +667,7 @@ static int dw_pcie_setup(int nr, struct pci_sys_data
 *sys)
 
 	if (global_io_offset < SZ_1M && pp->config.io_size > 0) {
 		sys->io_offset = global_io_offset - pp->config.io_bus_addr;
-		pci_ioremap_io(sys->io_offset, pp->io.start);
+		pci_ioremap_io(sys->io_offset, pp->io_base);
 		global_io_offset += SZ_64K;
 		pci_add_resource_offset(&sys->resources, &pp->io,
Quoted 1 line

sys->io_offset);

Tim, can you test if this patch fixes your SKY2 IOspace problem please ?

#13 lore raw
Quoted 1 line · Marek Vasut

On Monday, December 09, 2013 at 08:12:42 AM, Pratyush Anand wrote:

[.....]

Quoted 24 lines
Quoted 24 lines
(quoted diff) +0 -0

b/drivers/pci/host/pcie-designware.c index be6ce30..cf68632 100644

drivers/pci/host/pcie-designware.c +1 -1
@@ -378,6 +378,7 @@ int __init dw_pcie_host_init(struct pcie_port *pp)
 					   + global_io_offset);
 			pp->config.io_size = resource_size(&pp->io);
 			pp->config.io_bus_addr = range.pci_addr;
+			pp->io_base = range.cpu_addr;
 		}
 		if (restype == IORESOURCE_MEM) {
 			of_pci_range_to_resource(&range, np, &pp->mem);
@@ -403,7 +404,6 @@ int __init dw_pcie_host_init(struct pcie_port *pp)
 
 	pp->cfg0_base = pp->cfg.start;
 	pp->cfg1_base = pp->cfg.start + pp->config.cfg0_size;
-	pp->io_base = pp->io.start;
 	pp->mem_base = pp->mem.start;
 
 	pp->va_cfg0_base = devm_ioremap(pp->dev, pp->cfg0_base,
@@ -667,7 +667,7 @@ static int dw_pcie_setup(int nr, struct pci_sys_data
 *sys)
 
Quoted 9 lines
Quoted 7 lines
drivers/pci/host/pcie-designware.c +1 -1
 	if (global_io_offset < SZ_1M && pp->config.io_size > 0) {
 		sys->io_offset = global_io_offset - pp->config.io_bus_addr;
-		pci_ioremap_io(sys->io_offset, pp->io.start);
+		pci_ioremap_io(sys->io_offset, pp->io_base);
 		global_io_offset += SZ_64K;
 		pci_add_resource_offset(&sys->resources, &pp->io,

sys->io_offset);

Tim, can you test if this patch fixes your SKY2 IOspace problem please ?

Above mentioned patch is NOT the latest patch. Pratyush Anand already submitted the next patch as below.

Tim,
Would you test the following patch on i.MX6 platform?

drivers/pci/host/pcie-designware.c +2 -2
index be6ce30..b83f5e8 100644
@@ -378,6 +378,7 @@ int __init dw_pcie_host_init(struct pcie_port *pp)
 					   + global_io_offset);
 			pp->config.io_size = resource_size(&pp->io);
 			pp->config.io_bus_addr = range.pci_addr;
+			pp->io_base = range.cpu_addr;
 		}
 		if (restype == IORESOURCE_MEM) {
 			of_pci_range_to_resource(&range, np, &pp->mem);
@@ -403,7 +404,6 @@ int __init dw_pcie_host_init(struct pcie_port *pp)
 
 	pp->cfg0_base = pp->cfg.start;
 	pp->cfg1_base = pp->cfg.start + pp->config.cfg0_size;
-	pp->io_base = pp->io.start;
 	pp->mem_base = pp->mem.start;
 
 	pp->va_cfg0_base = devm_ioremap(pp->dev, pp->cfg0_base,
@@ -667,7 +667,7 @@ static int dw_pcie_setup(int nr, struct pci_sys_data *sys)
 
 	if (global_io_offset < SZ_1M && pp->config.io_size > 0) {
 		sys->io_offset = global_io_offset - pp->config.io_bus_addr;
-		pci_ioremap_io(sys->io_offset, pp->io.start);
+		pci_ioremap_io(global_io_offset, pp->io_base);
 		global_io_offset += SZ_64K;
 		pci_add_resource_offset(&sys->resources, &pp->io,
 					sys->io_offset);

Best regards,
Jingoo Han

#14 lore raw
Quoted 76 lines · Jingoo Han
Quoted 1 line · Marek Vasut

On Monday, December 09, 2013 at 08:12:42 AM, Pratyush Anand wrote:

[.....]

Quoted 24 lines
Quoted 24 lines
(quoted diff) +0 -0

b/drivers/pci/host/pcie-designware.c index be6ce30..cf68632 100644

drivers/pci/host/pcie-designware.c +1 -1
@@ -378,6 +378,7 @@ int __init dw_pcie_host_init(struct pcie_port *pp)
                                       + global_io_offset);
                    pp->config.io_size = resource_size(&pp->io);
                    pp->config.io_bus_addr = range.pci_addr;
+                   pp->io_base = range.cpu_addr;
            }
            if (restype == IORESOURCE_MEM) {
                    of_pci_range_to_resource(&range, np, &pp->mem);
@@ -403,7 +404,6 @@ int __init dw_pcie_host_init(struct pcie_port *pp)
 
    pp->cfg0_base = pp->cfg.start;
    pp->cfg1_base = pp->cfg.start + pp->config.cfg0_size;
-   pp->io_base = pp->io.start;
    pp->mem_base = pp->mem.start;
 
    pp->va_cfg0_base = devm_ioremap(pp->dev, pp->cfg0_base,
@@ -667,7 +667,7 @@ static int dw_pcie_setup(int nr, struct pci_sys_data
 *sys)
 
Quoted 9 lines
Quoted 7 lines
drivers/pci/host/pcie-designware.c +1 -1
    if (global_io_offset < SZ_1M && pp->config.io_size > 0) {
            sys->io_offset = global_io_offset - pp->config.io_bus_addr;
-           pci_ioremap_io(sys->io_offset, pp->io.start);
+           pci_ioremap_io(sys->io_offset, pp->io_base);
            global_io_offset += SZ_64K;
            pci_add_resource_offset(&sys->resources, &pp->io,
                                    sys->io_offset);

Tim, can you test if this patch fixes your SKY2 IOspace problem please ?

Above mentioned patch is NOT the latest patch. Pratyush Anand already submitted the next patch as below.

Tim,
Would you test the following patch on i.MX6 platform?

drivers/pci/host/pcie-designware.c +2 -2
index be6ce30..b83f5e8 100644
@@ -378,6 +378,7 @@ int __init dw_pcie_host_init(struct pcie_port *pp)
                                           + global_io_offset);
                        pp->config.io_size = resource_size(&pp->io);
                        pp->config.io_bus_addr = range.pci_addr;
+                       pp->io_base = range.cpu_addr;
                }
                if (restype == IORESOURCE_MEM) {
                        of_pci_range_to_resource(&range, np, &pp->mem);
@@ -403,7 +404,6 @@ int __init dw_pcie_host_init(struct pcie_port *pp)
 
        pp->cfg0_base = pp->cfg.start;
        pp->cfg1_base = pp->cfg.start + pp->config.cfg0_size;
-       pp->io_base = pp->io.start;
        pp->mem_base = pp->mem.start;
 
        pp->va_cfg0_base = devm_ioremap(pp->dev, pp->cfg0_base,
@@ -667,7 +667,7 @@ static int dw_pcie_setup(int nr, struct pci_sys_data *sys)
 
        if (global_io_offset < SZ_1M && pp->config.io_size > 0) {
                sys->io_offset = global_io_offset - pp->config.io_bus_addr;
-               pci_ioremap_io(sys->io_offset, pp->io.start);
+               pci_ioremap_io(global_io_offset, pp->io_base);
                global_io_offset += SZ_64K;
                pci_add_resource_offset(&sys->resources, &pp->io,
                                        sys->io_offset);
Quoted 3 lines

Best regards,
Jingoo Han

Yes, this patch resolves the issue and now I can use devices with io resources behind a bridge attached to the IMX6.

Thanks everyone for your efforts!

Tim

#15 lore raw
Quoted 103 lines · Tim Harvey
Quoted 102 lines · Jingoo Han
Quoted 1 line · Marek Vasut

On Monday, December 09, 2013 at 08:12:42 AM, Pratyush Anand wrote:

[.....]

Quoted 40 lines
Quoted 40 lines
(quoted diff) +0 -0

b/drivers/pci/host/pcie-designware.c index be6ce30..cf68632 100644

drivers/pci/host/pcie-designware.c +1 -0
@@ -378,6 +378,7 @@ int __init dw_pcie_host_init(struct pcie_port *pp)
 
                                       + global_io_offset);
 
                    pp->config.io_size = resource_size(&pp->io);
                    pp->config.io_bus_addr = range.pci_addr;
 
+                   pp->io_base = range.cpu_addr;
            }
            if (restype == IORESOURCE_MEM) {
            
                    of_pci_range_to_resource(&range, np, &pp->mem);
drivers/pci/host/pcie-designware.c +0 -1
@@ -403,7 +404,6 @@ int __init dw_pcie_host_init(struct pcie_port *pp)
 
    pp->cfg0_base = pp->cfg.start;
    pp->cfg1_base = pp->cfg.start + pp->config.cfg0_size;
 
-   pp->io_base = pp->io.start;
 
    pp->mem_base = pp->mem.start;
    
    pp->va_cfg0_base = devm_ioremap(pp->dev, pp->cfg0_base,
drivers/pci/host/pcie-designware.c +1 -1
@@ -667,7 +667,7 @@ static int dw_pcie_setup(int nr, struct
 pci_sys_data *sys)
 
    if (global_io_offset < SZ_1M && pp->config.io_size > 0) {
 
            sys->io_offset = global_io_offset -
            pp->config.io_bus_addr;
 
-           pci_ioremap_io(sys->io_offset, pp->io.start);
+           pci_ioremap_io(sys->io_offset, pp->io_base);
Quoted 6 lines
Quoted 4 lines
            global_io_offset += SZ_64K;
            pci_add_resource_offset(&sys->resources, &pp->io,
            
                                    sys->io_offset);

Tim, can you test if this patch fixes your SKY2 IOspace problem please ?

Above mentioned patch is NOT the latest patch. Pratyush Anand already submitted the next patch as below.

Tim,
Would you test the following patch on i.MX6 platform?

(quoted diff) +0 -0

b/drivers/pci/host/pcie-designware.c index be6ce30..b83f5e8 100644

drivers/pci/host/pcie-designware.c +1 -0
@@ -378,6 +378,7 @@ int __init dw_pcie_host_init(struct pcie_port *pp)
 
                                           + global_io_offset);
 
                        pp->config.io_size = resource_size(&pp->io);
                        pp->config.io_bus_addr = range.pci_addr;
 
+                       pp->io_base = range.cpu_addr;
                }
                if (restype == IORESOURCE_MEM) {
                
                        of_pci_range_to_resource(&range, np, &pp->mem);
drivers/pci/host/pcie-designware.c +0 -1
@@ -403,7 +404,6 @@ int __init dw_pcie_host_init(struct pcie_port *pp)
 
        pp->cfg0_base = pp->cfg.start;
        pp->cfg1_base = pp->cfg.start + pp->config.cfg0_size;
 
-       pp->io_base = pp->io.start;
 
        pp->mem_base = pp->mem.start;
        
        pp->va_cfg0_base = devm_ioremap(pp->dev, pp->cfg0_base,
drivers/pci/host/pcie-designware.c +1 -1
@@ -667,7 +667,7 @@ static int dw_pcie_setup(int nr, struct pci_sys_data
 *sys)
 
        if (global_io_offset < SZ_1M && pp->config.io_size > 0) {
 
                sys->io_offset = global_io_offset -
                pp->config.io_bus_addr;
 
-               pci_ioremap_io(sys->io_offset, pp->io.start);
+               pci_ioremap_io(global_io_offset, pp->io_base);
 
                global_io_offset += SZ_64K;
                pci_add_resource_offset(&sys->resources, &pp->io,
 
                                        sys->io_offset);
Quoted 7 lines
Quoted 2 lines

Best regards,
Jingoo Han

Yes, this patch resolves the issue and now I can use devices with io resources behind a bridge attached to the IMX6.

Thanks everyone for your efforts!

This is awesome, thanks guys!

#16 lore raw
In reply to Tim Harvey
Quoted 77 lines · Tim Harvey
Quoted 76 lines · Jingoo Han
Quoted 1 line · Marek Vasut

On Monday, December 09, 2013 at 08:12:42 AM, Pratyush Anand wrote:

[.....]

Quoted 24 lines
Quoted 24 lines
(quoted diff) +0 -0

b/drivers/pci/host/pcie-designware.c index be6ce30..cf68632 100644

drivers/pci/host/pcie-designware.c +1 -1
@@ -378,6 +378,7 @@ int __init dw_pcie_host_init(struct pcie_port *pp)
                                       + global_io_offset);
                    pp->config.io_size = resource_size(&pp->io);
                    pp->config.io_bus_addr = range.pci_addr;
+                   pp->io_base = range.cpu_addr;
            }
            if (restype == IORESOURCE_MEM) {
                    of_pci_range_to_resource(&range, np, &pp->mem);
@@ -403,7 +404,6 @@ int __init dw_pcie_host_init(struct pcie_port *pp)
 
    pp->cfg0_base = pp->cfg.start;
    pp->cfg1_base = pp->cfg.start + pp->config.cfg0_size;
-   pp->io_base = pp->io.start;
    pp->mem_base = pp->mem.start;
 
    pp->va_cfg0_base = devm_ioremap(pp->dev, pp->cfg0_base,
@@ -667,7 +667,7 @@ static int dw_pcie_setup(int nr, struct pci_sys_data
 *sys)
 
Quoted 9 lines
Quoted 7 lines
drivers/pci/host/pcie-designware.c +1 -1
    if (global_io_offset < SZ_1M && pp->config.io_size > 0) {
            sys->io_offset = global_io_offset - pp->config.io_bus_addr;
-           pci_ioremap_io(sys->io_offset, pp->io.start);
+           pci_ioremap_io(sys->io_offset, pp->io_base);
            global_io_offset += SZ_64K;
            pci_add_resource_offset(&sys->resources, &pp->io,
                                    sys->io_offset);

Tim, can you test if this patch fixes your SKY2 IOspace problem please ?

Above mentioned patch is NOT the latest patch. Pratyush Anand already submitted the next patch as below.

Tim,
Would you test the following patch on i.MX6 platform?

drivers/pci/host/pcie-designware.c +2 -2
index be6ce30..b83f5e8 100644
@@ -378,6 +378,7 @@ int __init dw_pcie_host_init(struct pcie_port *pp)
                                           + global_io_offset);
                        pp->config.io_size = resource_size(&pp->io);
                        pp->config.io_bus_addr = range.pci_addr;
+                       pp->io_base = range.cpu_addr;
                }
                if (restype == IORESOURCE_MEM) {
                        of_pci_range_to_resource(&range, np, &pp->mem);
@@ -403,7 +404,6 @@ int __init dw_pcie_host_init(struct pcie_port *pp)
 
        pp->cfg0_base = pp->cfg.start;
        pp->cfg1_base = pp->cfg.start + pp->config.cfg0_size;
-       pp->io_base = pp->io.start;
        pp->mem_base = pp->mem.start;
 
        pp->va_cfg0_base = devm_ioremap(pp->dev, pp->cfg0_base,
@@ -667,7 +667,7 @@ static int dw_pcie_setup(int nr, struct pci_sys_data *sys)
 
        if (global_io_offset < SZ_1M && pp->config.io_size > 0) {
                sys->io_offset = global_io_offset - pp->config.io_bus_addr;
-               pci_ioremap_io(sys->io_offset, pp->io.start);
+               pci_ioremap_io(global_io_offset, pp->io_base);
                global_io_offset += SZ_64K;
                pci_add_resource_offset(&sys->resources, &pp->io,
                                        sys->io_offset);
Quoted 8 lines
Quoted 3 lines

Best regards,
Jingoo Han

Yes, this patch resolves the issue and now I can use devices with io resources behind a bridge attached to the IMX6.

Thanks everyone for your efforts!

Wow, happy ending! :-)
I was deeply moved.

Pratyush Anand,
Would you re-send this patch with commit message? You did a good job! I really appreciate your patch.

Best regards,
Jingoo Han