Skip to contentSkip to conversation

Number of address translation regions in designware

QUERY Kishon Vijay Abraham I 6 participants 17 messages Oct 21, 2013 – Nov 18, 2013

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

Hi,

Currently I see in pcie-designware.c we use only 2 ATU regions. We re-use INDEX0 for mem outbound and cfg0, and INDEX1 for cfg1 and io. So I'd like to know if in your platform, do you have only 2 address translation regions? In DRA7xx we have 16 outbound regions and 4 inbound regions.

Also the same designware IP can be used as a EP also no? Shouldn't we move it out of drivers/pci/host and allow it to be configured as EP also?

Thanks
Kishon

#1 lore raw

Hi,

Quoted 6 lines · Kishon Vijay Abraham I

Hi,

Currently I see in pcie-designware.c we use only 2 ATU regions. We re-use INDEX0 for mem outbound and cfg0, and INDEX1 for cfg1 and io. So I'd like to know if in your platform, do you have only 2 address translation regions? In DRA7xx we have 16 outbound regions and 4 inbound regions.

I do not know about samsung, Jingoo can confirm. In SPEAr1340 there are only 2 inbound and outbound viewports, but in SPEAr1310 there are
6. It would be good to modify the driver, to avoid re-use of viewport whereever possible.

Quoted 2 lines

Also the same designware IP can be used as a EP also no? Shouldn't we move it

Yes same IP can be configured as EP. But..

Quoted 1 line

out of drivers/pci/host and allow it to be configured as EP also?

Same driver may not be used. (Few of these functions might be reused, but not whole). Moreover, in my experience running Linux over an EP can have only importance of validation/demo. Practical use case will find a simple firmware more suitable.

Regards
Pratyush

Quoted 7 lines

Thanks
Kishon
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#2 lore raw
In reply to Kishon Vijay Abraham I
Quoted 5 lines · Kishon Vijay Abraham I

Currently I see in pcie-designware.c we use only 2 ATU regions. We re-use INDEX0 for mem outbound and cfg0, and INDEX1 for cfg1 and io. So I'd like to know if in your platform, do you have only 2 address translation regions? In DRA7xx we have 16 outbound regions and 4 inbound regions.

In Exynos, there are only 2 inbound and 2 outbound viewpoints.

Quoted 2 lines

Also the same designware IP can be used as a EP also no? Shouldn't we move it out of drivers/pci/host and allow it to be configured as EP also?

Currently, Exynos PCIe IP does not support EP mode.

Best regards,
Jingoo Han

#3 lore raw

Hi Pratyush, Jingoo,

Quoted 7 lines · Jingoo Han
Quoted 5 lines · Kishon Vijay Abraham I

Currently I see in pcie-designware.c we use only 2 ATU regions. We re-use INDEX0 for mem outbound and cfg0, and INDEX1 for cfg1 and io. So I'd like to know if in your platform, do you have only 2 address translation regions? In DRA7xx we have 16 outbound regions and 4 inbound regions.

Quoted 6 lines

In Exynos, there are only 2 inbound and 2 outbound viewpoints.

Quoted 2 lines

Also the same designware IP can be used as a EP also no? Shouldn't we move it out of drivers/pci/host and allow it to be configured as EP also?

Currently, Exynos PCIe IP does not support EP mode.

Thanks for the information. I think we can do some optimization w.r.t address translation regions. Will post a RFC soon.

One more query.
In dw_pcie_prog_viewport_cfg0/dw_pcie_prog_viewport_cfg1 functions, the *lower target* is programmed to busdev. Is it for any specific reason?
I mean doing that will leave lot of holes in the PCIe address space. IIUC, if we don't set that to busdev, consecutive pcie address space will be used for each function no?

Thanks
Kishon

#4 lore raw

Hi Kishon,

Quoted 20 lines · Kishon Vijay Abraham I

Hi Pratyush, Jingoo,

Quoted 7 lines · Jingoo Han
Quoted 5 lines · Kishon Vijay Abraham I

Currently I see in pcie-designware.c we use only 2 ATU regions. We re-use INDEX0 for mem outbound and cfg0, and INDEX1 for cfg1 and io. So I'd like to know if in your platform, do you have only 2 address translation regions? In DRA7xx we have 16 outbound regions and 4 inbound regions.

Quoted 6 lines

In Exynos, there are only 2 inbound and 2 outbound viewpoints.

Quoted 2 lines

Also the same designware IP can be used as a EP also no? Shouldn't we move it out of drivers/pci/host and allow it to be configured as EP also?

Currently, Exynos PCIe IP does not support EP mode.

Thanks for the information. I think we can do some optimization w.r.t address translation regions. Will post a RFC soon.

Quoted 6 lines

One more query.
In dw_pcie_prog_viewport_cfg0/dw_pcie_prog_viewport_cfg1 functions, the *lower target* is programmed to busdev. Is it for any specific reason?
I mean doing that will leave lot of holes in the PCIe address space. IIUC, if we don't set that to busdev, consecutive pcie address space will be used for each function no?

Yes, I think even if CX_ATU_MIN_REGION_SIZE is 64KB in any controller, bus, dev and function number can be used to define target address.

So you are trying to allocate statically a separate viewport for each function's cfg0 transfer (whereever sufficient number of viewport is avilable)?

If viewport is programmed dynamically at each cfg transfer, then whether you use bus and dev only or function also, will it really make any difference in saving of address space?

Regards
Pratyush

Quoted 3 lines

Thanks
Kishon

#5 lore raw

Hi Pratyush,

Quoted 30 lines · Pratyush Anand

Hi Kishon,

Quoted 20 lines · Kishon Vijay Abraham I

Hi Pratyush, Jingoo,

Quoted 7 lines · Jingoo Han
Quoted 5 lines · Kishon Vijay Abraham I

Currently I see in pcie-designware.c we use only 2 ATU regions. We re-use INDEX0 for mem outbound and cfg0, and INDEX1 for cfg1 and io. So I'd like to know if in your platform, do you have only 2 address translation regions? In DRA7xx we have 16 outbound regions and 4 inbound regions.

Quoted 6 lines

In Exynos, there are only 2 inbound and 2 outbound viewpoints.

Quoted 2 lines

Also the same designware IP can be used as a EP also no? Shouldn't we move it out of drivers/pci/host and allow it to be configured as EP also?

Currently, Exynos PCIe IP does not support EP mode.

Thanks for the information. I think we can do some optimization w.r.t address translation regions. Will post a RFC soon.

Quoted 6 lines

One more query.
In dw_pcie_prog_viewport_cfg0/dw_pcie_prog_viewport_cfg1 functions, the *lower target* is programmed to busdev. Is it for any specific reason?
I mean doing that will leave lot of holes in the PCIe address space. IIUC, if we don't set that to busdev, consecutive pcie address space will be used for each function no?

Quoted 2 lines

Yes, I think even if CX_ATU_MIN_REGION_SIZE is 64KB in any controller, bus, dev and function number can be used to define target address.

I meant for function 0 the configuration space will be from (0x0 t0 0xfff - 4KB). If we use *busdev* for programming target, for function 1 the configuration space will start at 0x10000 PCIe address which will leave a small hole from (0x1000 to 0xffff). I'm not sure if it will have a big impact though. Also till we haven't exhausted the 64KB space (minimum), we wouldn't be needing to program a new viewport. That's good enough to accommodate 18 functions for the number of devices.

Quoted 4 lines

So you are trying to allocate statically a separate viewport for each function's cfg0 transfer (whereever sufficient number of viewport is avilable)?

Actually in DRA7xx, we face a different problem in that, we can't program the translation region directly from the ranges. Because while programming the address translation window, the most significant 4 bits are not used.

For example if you see, the cpu address of memory window starts @ 0x20012000 but while programming the address translation window we have to give 0x00012000.

My dt data looked like this.
ranges = <0x00000800 0 0x20001000 0x20001000 0 0x00001000

	  0x81000000 0 0	  0x20002000 0 0x00010000
	  0x82000000 0 0x20012000 0x20012000 0 0xffee000>;

translation = <(OUTBOUND | INDEX0) CFG0 0x0 0x00001000 0x0000fff 0x0 0x00001000

	       (OUTBOUND | INDEX1) IO   0x0 0x00002000 0x000ffff 0x0 0x00002000
	       (OUTBOUND | INDEX2) MEM  0x0 0x00012000 0xffee000 0x0 0x20012000
Quoted 1 line

;

(btw that translation is a wip I tried to get pcie working in dra7x).

Since I wanted to do the translation configuration from dt data, I was thinking whether it is necessary to program the viewport dynamically.

But now I think some sort of dynamic programming would be needed, considering the limitation on few SoCs.

Thanks
Kishon

Quoted 11 lines

If viewport is programmed dynamically at each cfg transfer, then whether you use bus and dev only or function also, will it really make any difference in saving of address space?

Regards
Pratyush

Quoted 3 lines

Thanks
Kishon

#6 lore raw

+ Marek, Tim, arnd

Hi Kishon,

Quoted 36 lines · Kishon Vijay Abraham I

Hi Pratyush,

Quoted 30 lines · Pratyush Anand

Hi Kishon,

Quoted 20 lines · Kishon Vijay Abraham I

Hi Pratyush, Jingoo,

Quoted 7 lines · Jingoo Han
Quoted 5 lines · Kishon Vijay Abraham I

Currently I see in pcie-designware.c we use only 2 ATU regions. We re-use INDEX0 for mem outbound and cfg0, and INDEX1 for cfg1 and io. So I'd like to know if in your platform, do you have only 2 address translation regions? In DRA7xx we have 16 outbound regions and 4 inbound regions.

Quoted 6 lines

In Exynos, there are only 2 inbound and 2 outbound viewpoints.

Quoted 2 lines

Also the same designware IP can be used as a EP also no? Shouldn't we move it out of drivers/pci/host and allow it to be configured as EP also?

Currently, Exynos PCIe IP does not support EP mode.

Thanks for the information. I think we can do some optimization w.r.t address translation regions. Will post a RFC soon.

Quoted 6 lines

One more query.
In dw_pcie_prog_viewport_cfg0/dw_pcie_prog_viewport_cfg1 functions, the *lower target* is programmed to busdev. Is it for any specific reason?
I mean doing that will leave lot of holes in the PCIe address space. IIUC, if we don't set that to busdev, consecutive pcie address space will be used for each function no?

Quoted 2 lines

Yes, I think even if CX_ATU_MIN_REGION_SIZE is 64KB in any controller, bus, dev and function number can be used to define target address.

Quoted 7 lines

I meant for function 0 the configuration space will be from (0x0 t0 0xfff - 4KB). If we use *busdev* for programming target, for function 1 the configuration space will start at 0x10000 PCIe address which will leave a small hole from (0x1000 to 0xffff). I'm not sure if it will have a big impact though. Also till we haven't exhausted the 64KB space (minimum), we wouldn't be needing to program a new viewport. That's good enough to accommodate 18 functions for the number of devices.

Yes, I agree with you that using *busdev* will create hole and we should avoid it. More on this at the end.

Quoted 12 lines
Quoted 4 lines

So you are trying to allocate statically a separate viewport for each function's cfg0 transfer (whereever sufficient number of viewport is avilable)?

Actually in DRA7xx, we face a different problem in that, we can't program the translation region directly from the ranges. Because while programming the address translation window, the most significant 4 bits are not used.

For example if you see, the cpu address of memory window starts @ 0x20012000 but while programming the address translation window we have to give 0x00012000.

Quoted 9 lines

My dt data looked like this.
ranges = <0x00000800 0 0x20001000 0x20001000 0 0x00001000

	  0x81000000 0 0	  0x20002000 0 0x00010000
	  0x82000000 0 0x20012000 0x20012000 0 0xffee000>;

translation = <(OUTBOUND | INDEX0) CFG0 0x0 0x00001000 0x0000fff 0x0 0x00001000

	       (OUTBOUND | INDEX1) IO   0x0 0x00002000 0x000ffff 0x0 0x00002000
	       (OUTBOUND | INDEX2) MEM  0x0 0x00012000 0xffee000 0x0 0x20012000
Quoted 1 line

;

I too have some doubt here..not very clear.. may be someone more expert can correct (I keep arnd in cc).... How should be the translation?
In my opinion, input address of translation should be in "bus address domain" ie address what PCIe sees and translated output for memory transaction should be in "parent bus address domain" ie what CPUs see. If my understanding is correct, then pcie designware driver needs fixes at some places.

I am not much used to DT, but if I understood correctly, in your case parent bus address and bus address are different.

pp->mem_base = bus address = range.pci_addr = 0x0000000000012000 pp->config.mem_bus_addr = parent bus address = range.cpu_addr = 0x20012000

So, shouldn't your ranges look like this:

ranges = <0x00000800 0 0x00001000 0x20001000 0 0x00001000
0x81000000 0 0 0x20002000 0 0x00010000
0x82000000 0 0x00012000 0x20012000 0 0xffee000>;

And then I think with the proper designware driver fixes you should be able to work.

Quoted 7 lines

(btw that translation is a wip I tried to get pcie working in dra7x).

Since I wanted to do the translation configuration from dt data, I was thinking whether it is necessary to program the viewport dynamically.

But now I think some sort of dynamic programming would be needed, considering the limitation on few SoCs.

Yes, some dynamic programming is needed and not sure how to manage in case of only one outbound viewport. What about following idea:
1. We keep dynamic for cfg and static for others.
2. We use single viewport for cfg0 and cfg1 both. We do not use separate cfg0/1 base rather use only cfg_base and cfg_size=64K (we could have done with even 4K, but some controller might have min viewport size as 64K). As per viewport CR programmed, translated output will be cfg0 or cfg1.
3. Now if a controller has only one viewport, a warning message is generated that IO transaction may not work correctly. (I think IO will always need translation. Just for curiosity anyone knows a ARM based system which uses PCIe IO transaction?).
4. If a controller has atleast 2 viewport, we program second viewport statically for IO transaction.
5. If it has only two viewport, and cpu_addr != pci_addr, a warning message is generated that MEM transaction may not work correctly.
6. Else, 3rd viewport is programmed for mem transaction.

Regards
Pratyush

Quoted 15 lines

Thanks
Kishon

Quoted 11 lines

If viewport is programmed dynamically at each cfg transfer, then whether you use bus and dev only or function also, will it really make any difference in saving of address space?

Regards
Pratyush

Quoted 3 lines

Thanks
Kishon

#7 lore raw

Hi Pratyush,

Quoted 72 lines · Pratyush Anand

+ Marek, Tim, arnd

Hi Kishon,

Quoted 36 lines · Kishon Vijay Abraham I

Hi Pratyush,

Quoted 30 lines · Pratyush Anand

Hi Kishon,

Quoted 20 lines · Kishon Vijay Abraham I

Hi Pratyush, Jingoo,

Quoted 7 lines · Jingoo Han
Quoted 5 lines · Kishon Vijay Abraham I

Currently I see in pcie-designware.c we use only 2 ATU regions. We re-use INDEX0 for mem outbound and cfg0, and INDEX1 for cfg1 and io. So I'd like to know if in your platform, do you have only 2 address translation regions? In DRA7xx we have 16 outbound regions and 4 inbound regions.

Quoted 6 lines

In Exynos, there are only 2 inbound and 2 outbound viewpoints.

Quoted 2 lines

Also the same designware IP can be used as a EP also no? Shouldn't we move it out of drivers/pci/host and allow it to be configured as EP also?

Currently, Exynos PCIe IP does not support EP mode.

Thanks for the information. I think we can do some optimization w.r.t address translation regions. Will post a RFC soon.

Quoted 6 lines

One more query.
In dw_pcie_prog_viewport_cfg0/dw_pcie_prog_viewport_cfg1 functions, the *lower target* is programmed to busdev. Is it for any specific reason?
I mean doing that will leave lot of holes in the PCIe address space. IIUC, if we don't set that to busdev, consecutive pcie address space will be used for each function no?

Quoted 2 lines

Yes, I think even if CX_ATU_MIN_REGION_SIZE is 64KB in any controller, bus, dev and function number can be used to define target address.

Quoted 7 lines

I meant for function 0 the configuration space will be from (0x0 t0 0xfff - 4KB). If we use *busdev* for programming target, for function 1 the configuration space will start at 0x10000 PCIe address which will leave a small hole from (0x1000 to 0xffff). I'm not sure if it will have a big impact though. Also till we haven't exhausted the 64KB space (minimum), we wouldn't be needing to program a new viewport. That's good enough to accommodate 18 functions for the number of devices.

Yes, I agree with you that using *busdev* will create hole and we should avoid it. More on this at the end.

Quoted 20 lines
Quoted 4 lines

So you are trying to allocate statically a separate viewport for each function's cfg0 transfer (whereever sufficient number of viewport is avilable)?

Actually in DRA7xx, we face a different problem in that, we can't program the translation region directly from the ranges. Because while programming the address translation window, the most significant 4 bits are not used.

For example if you see, the cpu address of memory window starts @ 0x20012000 but while programming the address translation window we have to give 0x00012000.

My dt data looked like this.
ranges = <0x00000800 0 0x20001000 0x20001000 0 0x00001000

	  0x81000000 0 0	  0x20002000 0 0x00010000
	  0x82000000 0 0x20012000 0x20012000 0 0xffee000>;

translation = <(OUTBOUND | INDEX0) CFG0 0x0 0x00001000 0x0000fff 0x0 0x00001000

	       (OUTBOUND | INDEX1) IO   0x0 0x00002000 0x000ffff 0x0 0x00002000
	       (OUTBOUND | INDEX2) MEM  0x0 0x00012000 0xffee000 0x0 0x20012000
Quoted 7 lines
Quoted 1 line
Quoted 1 line

;

I too have some doubt here..not very clear.. may be someone more expert can correct (I keep arnd in cc).... How should be the translation?
In my opinion, input address of translation should be in "bus address domain" ie address what PCIe sees and translated output for memory

It actually depends on whether we are configuring inbound or outbound. IIUC, what you just mentioned will be for configuring inbound translation window.

For the outbound translation window, it'll be the other way round. *base* will be the CPU address and target will be the PCIe address.

Quoted 9 lines

transaction should be in "parent bus address domain" ie what CPUs see. If my understanding is correct, then pcie designware driver needs fixes at some places.

I am not much used to DT, but if I understood correctly, in your case parent bus address and bus address are different.

pp->mem_base = bus address = range.pci_addr = 0x0000000000012000 pp->config.mem_bus_addr = parent bus address = range.cpu_addr = 0x20012000

Not like that. It's the same.
In DRA7, the cpu sees 32bit address, but the pcie controller can see only 28bit address. So whenever the cpu issues a read/write request, the 4 most significant bits are used by L3 to determine the target controller. For example, the cpu reserves 0x2000_0000 - 0x2FFF_FFFF for PCIe controller but the PCIe controller will see only (0x000_0000 - 0xFFF_FFF). So for programming the outbound translation window the *base* should be programmed as 0x000_0000. Whenever we try to write to say 0x2000_0000, it will be translated to whatever we have programmed in the translation window with base as 0x000_0000.

Thanks
Kishon

#8 lore raw
Quoted 83 lines · Kishon Vijay Abraham I

Hi Pratyush,

Quoted 72 lines · Pratyush Anand

+ Marek, Tim, arnd

Hi Kishon,

Quoted 36 lines · Kishon Vijay Abraham I

Hi Pratyush,

Quoted 30 lines · Pratyush Anand

Hi Kishon,

Quoted 20 lines · Kishon Vijay Abraham I

Hi Pratyush, Jingoo,

Quoted 7 lines · Jingoo Han
Quoted 5 lines · Kishon Vijay Abraham I

Currently I see in pcie-designware.c we use only 2 ATU regions. We re-use INDEX0 for mem outbound and cfg0, and INDEX1 for cfg1 and io. So I'd like to know if in your platform, do you have only 2 address translation regions? In DRA7xx we have 16 outbound regions and 4 inbound regions.

Quoted 6 lines

In Exynos, there are only 2 inbound and 2 outbound viewpoints.

Quoted 2 lines

Also the same designware IP can be used as a EP also no? Shouldn't we move it out of drivers/pci/host and allow it to be configured as EP also?

Currently, Exynos PCIe IP does not support EP mode.

Thanks for the information. I think we can do some optimization w.r.t address translation regions. Will post a RFC soon.

Quoted 6 lines

One more query.
In dw_pcie_prog_viewport_cfg0/dw_pcie_prog_viewport_cfg1 functions, the *lower target* is programmed to busdev. Is it for any specific reason?
I mean doing that will leave lot of holes in the PCIe address space. IIUC, if we don't set that to busdev, consecutive pcie address space will be used for each function no?

Quoted 2 lines

Yes, I think even if CX_ATU_MIN_REGION_SIZE is 64KB in any controller, bus, dev and function number can be used to define target address.

Quoted 7 lines

I meant for function 0 the configuration space will be from (0x0 t0 0xfff - 4KB). If we use *busdev* for programming target, for function 1 the configuration space will start at 0x10000 PCIe address which will leave a small hole from (0x1000 to 0xffff). I'm not sure if it will have a big impact though. Also till we haven't exhausted the 64KB space (minimum), we wouldn't be needing to program a new viewport. That's good enough to accommodate 18 functions for the number of devices.

Yes, I agree with you that using *busdev* will create hole and we should avoid it. More on this at the end.

Quoted 20 lines
Quoted 4 lines

So you are trying to allocate statically a separate viewport for each function's cfg0 transfer (whereever sufficient number of viewport is avilable)?

Actually in DRA7xx, we face a different problem in that, we can't program the translation region directly from the ranges. Because while programming the address translation window, the most significant 4 bits are not used.

For example if you see, the cpu address of memory window starts @ 0x20012000 but while programming the address translation window we have to give 0x00012000.

My dt data looked like this.
ranges = <0x00000800 0 0x20001000 0x20001000 0 0x00001000

       0x81000000 0 0          0x20002000 0 0x00010000
       0x82000000 0 0x20012000 0x20012000 0 0xffee000>;

translation = <(OUTBOUND | INDEX0) CFG0 0x0 0x00001000 0x0000fff 0x0 0x00001000

            (OUTBOUND | INDEX1) IO   0x0 0x00002000 0x000ffff 0x0 0x00002000
            (OUTBOUND | INDEX2) MEM  0x0 0x00012000 0xffee000 0x0 0x20012000
Quoted 7 lines
Quoted 1 line
Quoted 1 line

;

I too have some doubt here..not very clear.. may be someone more expert can correct (I keep arnd in cc).... How should be the translation?
In my opinion, input address of translation should be in "bus address domain" ie address what PCIe sees and translated output for memory

Quoted 5 lines

It actually depends on whether we are configuring inbound or outbound. IIUC, what you just mentioned will be for configuring inbound translation window.

For the outbound translation window, it'll be the other way round. *base* will be the CPU address and target will be the PCIe address.

Sure? At the the end you say base address is 0x000_0000 (bus address) and not 0x2000_0000 (parent bus address). It contradicts what you say above, no?

Quoted 10 lines
Quoted 9 lines

transaction should be in "parent bus address domain" ie what CPUs see. If my understanding is correct, then pcie designware driver needs fixes at some places.

I am not much used to DT, but if I understood correctly, in your case parent bus address and bus address are different.

pp->mem_base = bus address = range.pci_addr = 0x0000000000012000 pp->config.mem_bus_addr = parent bus address = range.cpu_addr = 0x20012000

Quoted 9 lines

Not like that. It's the same.
In DRA7, the cpu sees 32bit address, but the pcie controller can see only 28bit address. So whenever the cpu issues a read/write request, the 4 most significant bits are used by L3 to determine the target controller. For example, the cpu reserves 0x2000_0000 - 0x2FFF_FFFF for PCIe controller but the PCIe controller will see only (0x000_0000 - 0xFFF_FFF). So for programming the outbound translation window the *base* should be programmed as 0x000_0000. Whenever we try to write to say 0x2000_0000, it will be translated to whatever we have programmed in the translation window with base as 0x000_0000.

What you say here is also my understanding.
Only question then remains, what should be translated address? All translated address must lie in the address range programmed in EP's BAR, then only EP will able to receive that mem transaction, no? All EP's BAR address will lie in the range of pp->mem.start and pp->mem.end, am I correct here?
And, pp->mem.start is range.cpu_addr, therefore I said that if target address is programmed as cpu_address, it should work fine.

Regards
Pratyush

Quoted 7 lines

Thanks
Kishon
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#9 lore raw

Hi,

Quoted 84 lines · Pratyush Anand
Quoted 83 lines · Kishon Vijay Abraham I

Hi Pratyush,

Quoted 79 lines · Pratyush Anand

+ Marek, Tim, arnd

Hi Kishon,

Quoted 36 lines · Kishon Vijay Abraham I

Hi Pratyush,

Quoted 30 lines · Pratyush Anand

Hi Kishon,

Quoted 20 lines · Kishon Vijay Abraham I

Hi Pratyush, Jingoo,

Quoted 7 lines · Jingoo Han
Quoted 5 lines · Kishon Vijay Abraham I

Currently I see in pcie-designware.c we use only 2 ATU regions. We re-use INDEX0 for mem outbound and cfg0, and INDEX1 for cfg1 and io. So I'd like to know if in your platform, do you have only 2 address translation regions? In DRA7xx we have 16 outbound regions and 4 inbound regions.

Quoted 6 lines

In Exynos, there are only 2 inbound and 2 outbound viewpoints.

Quoted 2 lines

Also the same designware IP can be used as a EP also no? Shouldn't we move it out of drivers/pci/host and allow it to be configured as EP also?

Currently, Exynos PCIe IP does not support EP mode.

Thanks for the information. I think we can do some optimization w.r.t address translation regions. Will post a RFC soon.

Quoted 6 lines

One more query.
In dw_pcie_prog_viewport_cfg0/dw_pcie_prog_viewport_cfg1 functions, the *lower target* is programmed to busdev. Is it for any specific reason?
I mean doing that will leave lot of holes in the PCIe address space. IIUC, if we don't set that to busdev, consecutive pcie address space will be used for each function no?

Quoted 2 lines

Yes, I think even if CX_ATU_MIN_REGION_SIZE is 64KB in any controller, bus, dev and function number can be used to define target address.

Quoted 7 lines

I meant for function 0 the configuration space will be from (0x0 t0 0xfff - 4KB). If we use *busdev* for programming target, for function 1 the configuration space will start at 0x10000 PCIe address which will leave a small hole from (0x1000 to 0xffff). I'm not sure if it will have a big impact though. Also till we haven't exhausted the 64KB space (minimum), we wouldn't be needing to program a new viewport. That's good enough to accommodate 18 functions for the number of devices.

Yes, I agree with you that using *busdev* will create hole and we should avoid it. More on this at the end.

Quoted 12 lines
Quoted 4 lines

So you are trying to allocate statically a separate viewport for each function's cfg0 transfer (whereever sufficient number of viewport is avilable)?

Actually in DRA7xx, we face a different problem in that, we can't program the translation region directly from the ranges. Because while programming the address translation window, the most significant 4 bits are not used.

For example if you see, the cpu address of memory window starts @ 0x20012000 but while programming the address translation window we have to give 0x00012000.

Quoted 9 lines

My dt data looked like this.
ranges = <0x00000800 0 0x20001000 0x20001000 0 0x00001000

        0x81000000 0 0          0x20002000 0 0x00010000
        0x82000000 0 0x20012000 0x20012000 0 0xffee000>;

translation = <(OUTBOUND | INDEX0) CFG0 0x0 0x00001000 0x0000fff 0x0 0x00001000

             (OUTBOUND | INDEX1) IO   0x0 0x00002000 0x000ffff 0x0 0x00002000
             (OUTBOUND | INDEX2) MEM  0x0 0x00012000 0xffee000 0x0 0x20012000
Quoted 1 line

;

I too have some doubt here..not very clear.. may be someone more expert can correct (I keep arnd in cc).... How should be the translation?
In my opinion, input address of translation should be in "bus address domain" ie address what PCIe sees and translated output for memory

Quoted 7 lines
Quoted 5 lines

It actually depends on whether we are configuring inbound or outbound. IIUC, what you just mentioned will be for configuring inbound translation window.

For the outbound translation window, it'll be the other way round. *base* will be the CPU address and target will be the PCIe address.

Sure? At the the end you say base address is 0x000_0000 (bus address) and not

This address is not actually a bus address (note it has only 28bits). Rather it's used only to program the address translation unit of desigware (for DRA7x). Both the cpu address and pcie address is the same.

Quoted 21 lines

0x2000_0000 (parent bus address). It contradicts what you say above, no?

Quoted 10 lines
Quoted 9 lines

transaction should be in "parent bus address domain" ie what CPUs see. If my understanding is correct, then pcie designware driver needs fixes at some places.

I am not much used to DT, but if I understood correctly, in your case parent bus address and bus address are different.

pp->mem_base = bus address = range.pci_addr = 0x0000000000012000 pp->config.mem_bus_addr = parent bus address = range.cpu_addr = 0x20012000

Quoted 9 lines

Not like that. It's the same.
In DRA7, the cpu sees 32bit address, but the pcie controller can see only 28bit address. So whenever the cpu issues a read/write request, the 4 most significant bits are used by L3 to determine the target controller. For example, the cpu reserves 0x2000_0000 - 0x2FFF_FFFF for PCIe controller but the PCIe controller will see only (0x000_0000 - 0xFFF_FFF). So for programming the outbound translation window the *base* should be programmed as 0x000_0000. Whenever we try to write to say 0x2000_0000, it will be translated to whatever we have programmed in the translation window with base as 0x000_0000.

Quoted 4 lines

What you say here is also my understanding.
Only question then remains, what should be translated address? All translated address must lie in the address range programmed in EP's BAR, then only EP will able to receive that mem transaction, no?

right.

Quoted 2 lines

All EP's BAR address will lie in the range of pp->mem.start and pp->mem.end, am I correct here?

I'm not sure how the PCI core code sets the BAR of the EP. I have to check it.

Quoted 2 lines

And, pp->mem.start is range.cpu_addr, therefore I said that if target address is programmed as cpu_address, it should work fine.

Agreed.

Thanks
Kishon

#10 lore raw

Hi Kishon,

just curious, what's the current status of the DWC PCIe driver? How's the iATU issue going? Can you please update us a little?

Thank you!

Cheers!

#11 lore raw

Hi,

Quoted 4 lines · Marek Vasut

Hi Kishon,

just curious, what's the current status of the DWC PCIe driver? How's the iATU issue going? Can you please update us a little?

I was able to get it working in DRA7. Few of the dependent stuffs are not in mainline yet (some clock related patches, rest framework changes, PHY changes). Will post a RFC soon for pcie only part.

Thanks
Kishon

#12 lore raw
Quoted 9 lines · Kishon Vijay Abraham I
Quoted 4 lines · Marek Vasut

Hi Kishon,

just curious, what's the current status of the DWC PCIe driver? How's the iATU issue going? Can you please update us a little?

I was able to get it working in DRA7. Few of the dependent stuffs are not in mainline yet (some clock related patches, rest framework changes, PHY changes). Will post a RFC soon for pcie only part.

OK, I see.
If you send the RFC patch, I will review ant test your patch as soon as possible. Thank you for your effort. :-)

Best regards,
Jingoo Han

#13 lore raw

Hi Kishon, Jingoo,

Quoted 7 lines · Kishon Vijay Abraham I
Quoted 6 lines · Marek Vasut
Quoted 4 lines

Hi Kishon,

just curious, what's the current status of the DWC PCIe driver? How's the iATU issue going? Can you please update us a little?

Quoted 7 lines
Quoted 3 lines

I was able to get it working in DRA7. Few of the dependent stuffs are not in mainline yet (some clock related patches, rest framework changes, PHY changes). Will post a RFC soon for pcie only part.

OK, I see.
If you send the RFC patch, I will review ant test your patch as soon as possible. Thank you for your effort. :-)

DTTO here, I can test it on MX6 as well. If you can roll out a prototype, I'm sure we can help in some ways too :)

Best regards,
Marek Vasut

#14 lore raw

Hi,

Quoted 13 lines · Marek Vasut

Hi Kishon, Jingoo,

Quoted 11 lines · Kishon Vijay Abraham I
Quoted 9 lines · Marek Vasut
Quoted 4 lines

Hi Kishon,

just curious, what's the current status of the DWC PCIe driver? How's the iATU issue going? Can you please update us a little?

I was able to get it working in DRA7. Few of the dependent stuffs are not in mainline yet (some clock related patches, rest framework changes, PHY changes). Will post a RFC soon for pcie only part.

Quoted 6 lines
Quoted 3 lines

OK, I see.
If you send the RFC patch, I will review ant test your patch as soon as possible. Thank you for your effort. :-)

DTTO here, I can test it on MX6 as well. If you can roll out a prototype, I'm sure we can help in some ways too :)

huh.. I didn't change much in designware code that would affect other platforms. I just added a property base_mask that should be populated from the dt data. (if it's not populated from the dt data, it will take the default value of all f's). Here is the patch anyway.

Thanks
Kishon

commit b91b61d594f293d994d87f10eb8a1a3b2e5f8977

Author: Kishon Vijay Abraham I <[email protected]>
Date:   Wed Oct 30 14:51:58 2013 +0530

pci: host: pcie-designware: Use *base-mask* for configuring the iATU

In DRA7, the cpu sees 32bit address, but the pcie controller can see only 28bit address. So whenever the cpu issues a read/write request, the 4 most significant bits are used by L3 to determine the target controller. For example, the cpu reserves 0x2000_0000 - 0x2FFF_FFFF for PCIe controller but the PCIe controller will see only (0x000_0000 - 0xFFF_FFF). So for programming the outbound translation window the *base* should be programmed as 0x000_0000. Whenever we try to write to say 0x2000_0000, it will be translated to whatever we have programmed in the translation window with base as 0x000_0000.

(quoted diff) +0 -0

b/Documentation/devicetree/bindings/pci/designware-pcie.txt index d5d26d4..fc04ab7 100644

Documentation/devicetree/bindings/pci/designware-pcie.txt +1 -0
@@ -25,6 +25,7 @@ Optional properties for fsl,imx6q-pcie
 - power-on-gpio: gpio pin number of power-enable signal
 - wake-up-gpio: gpio pin number of incoming wakeup signal
 - disable-gpio: gpio pin number of outgoing rfkill/endpoint disable signal
+- base-mask: address mask for the PCIe controller target port
 
 Example:
 
(quoted diff) +0 -0

b/drivers/pci/host/pcie-designware.c
index f879116..fef64cf 100644

drivers/pci/host/pcie-designware.c +9 -3
@@ -440,6 +440,9 @@ int __init dw_pcie_host_init(struct pcie_port *pp)
 		return -EINVAL;
 	}
 
+	if (of_property_read_u64(np, "base-mask", &pp->base_mask))
+		pp->base_mask = ~(0x0ULL);
+
 	if (IS_ENABLED(CONFIG_PCI_MSI)) {
 		pp->irq_domain = irq_domain_add_linear(pp->dev->of_node,
 					MAX_MSI_IRQS, &msi_domain_ops,
@@ -479,12 +482,15 @@ int __init dw_pcie_host_init(struct pcie_port *pp)
 
 static void dw_pcie_prog_viewport_cfg0(struct pcie_port *pp, u32 busdev)
 {
+	u64 cfg0_base;
+
+	cfg0_base = pp->cfg0_base & pp->base_mask;
 	/* Program viewport 0 : OUTBOUND : CFG0 */
 	dw_pcie_writel_rc(pp, PCIE_ATU_REGION_OUTBOUND | PCIE_ATU_REGION_INDEX0,
 			  PCIE_ATU_VIEWPORT);
-	dw_pcie_writel_rc(pp, pp->cfg0_base, PCIE_ATU_LOWER_BASE);
-	dw_pcie_writel_rc(pp, (pp->cfg0_base >> 32), PCIE_ATU_UPPER_BASE);
-	dw_pcie_writel_rc(pp, pp->cfg0_base + pp->config.cfg0_size - 1,
+	dw_pcie_writel_rc(pp, cfg0_base, PCIE_ATU_LOWER_BASE);
+	dw_pcie_writel_rc(pp, (cfg0_base >> 32), PCIE_ATU_UPPER_BASE);
+	dw_pcie_writel_rc(pp, cfg0_base + pp->config.cfg0_size - 1,
 			  PCIE_ATU_LIMIT);
 	dw_pcie_writel_rc(pp, busdev, PCIE_ATU_LOWER_TARGET);
 	dw_pcie_writel_rc(pp, 0, PCIE_ATU_UPPER_TARGET);
@@ -494,14 +500,17 @@ static void dw_pcie_prog_viewport_cfg0(struct pcie_port

*pp, u32 busdev)

static void dw_pcie_prog_viewport_cfg1(struct pcie_port *pp, u32 busdev) {
+ u64 cfg1_base;
+
+ cfg1_base = pp->cfg1_base & pp->base_mask;
/* Program viewport 1 : OUTBOUND : CFG1 */ dw_pcie_writel_rc(pp, PCIE_ATU_REGION_OUTBOUND | PCIE_ATU_REGION_INDEX1,
PCIE_ATU_VIEWPORT);
dw_pcie_writel_rc(pp, PCIE_ATU_TYPE_CFG1, PCIE_ATU_CR1); dw_pcie_writel_rc(pp, PCIE_ATU_ENABLE, PCIE_ATU_CR2);
- dw_pcie_writel_rc(pp, pp->cfg1_base, PCIE_ATU_LOWER_BASE);
- dw_pcie_writel_rc(pp, (pp->cfg1_base >> 32), PCIE_ATU_UPPER_BASE);
- dw_pcie_writel_rc(pp, pp->cfg1_base + pp->config.cfg1_size - 1,
+ dw_pcie_writel_rc(pp, cfg1_base, PCIE_ATU_LOWER_BASE);
+ dw_pcie_writel_rc(pp, (cfg1_base >> 32), PCIE_ATU_UPPER_BASE);
+ dw_pcie_writel_rc(pp, cfg1_base + pp->config.cfg1_size - 1,
PCIE_ATU_LIMIT);
dw_pcie_writel_rc(pp, busdev, PCIE_ATU_LOWER_TARGET); dw_pcie_writel_rc(pp, 0, PCIE_ATU_UPPER_TARGET);
@@ -509,14 +518,17 @@ static void dw_pcie_prog_viewport_cfg1(struct pcie_port *pp, u32 busdev)

static void dw_pcie_prog_viewport_mem_outbound(struct pcie_port *pp) {
+ u64 mem_base;
+
+ mem_base = pp->mem_base & pp->base_mask;
/* Program viewport 0 : OUTBOUND : MEM */ dw_pcie_writel_rc(pp, PCIE_ATU_REGION_OUTBOUND | PCIE_ATU_REGION_INDEX0,
PCIE_ATU_VIEWPORT);
dw_pcie_writel_rc(pp, PCIE_ATU_TYPE_MEM, PCIE_ATU_CR1); dw_pcie_writel_rc(pp, PCIE_ATU_ENABLE, PCIE_ATU_CR2);
- dw_pcie_writel_rc(pp, pp->mem_base, PCIE_ATU_LOWER_BASE);
- dw_pcie_writel_rc(pp, (pp->mem_base >> 32), PCIE_ATU_UPPER_BASE);
- dw_pcie_writel_rc(pp, pp->mem_base + pp->config.mem_size - 1,
+ dw_pcie_writel_rc(pp, mem_base, PCIE_ATU_LOWER_BASE);
+ dw_pcie_writel_rc(pp, (mem_base >> 32), PCIE_ATU_UPPER_BASE);
+ dw_pcie_writel_rc(pp, mem_base + pp->config.mem_size - 1,
PCIE_ATU_LIMIT);
dw_pcie_writel_rc(pp, pp->config.mem_bus_addr, PCIE_ATU_LOWER_TARGET); dw_pcie_writel_rc(pp, upper_32_bits(pp->config.mem_bus_addr),
@@ -525,14 +537,17 @@ static void dw_pcie_prog_viewport_mem_outbound(struct pcie_port *pp)

static void dw_pcie_prog_viewport_io_outbound(struct pcie_port *pp) {
+ u64 io_base;
+
+ io_base = pp->io_base & pp->base_mask;
/* Program viewport 1 : OUTBOUND : IO */ dw_pcie_writel_rc(pp, PCIE_ATU_REGION_OUTBOUND | PCIE_ATU_REGION_INDEX1,
PCIE_ATU_VIEWPORT);
dw_pcie_writel_rc(pp, PCIE_ATU_TYPE_IO, PCIE_ATU_CR1); dw_pcie_writel_rc(pp, PCIE_ATU_ENABLE, PCIE_ATU_CR2);
- dw_pcie_writel_rc(pp, pp->io_base, PCIE_ATU_LOWER_BASE);
- dw_pcie_writel_rc(pp, (pp->io_base >> 32), PCIE_ATU_UPPER_BASE);
- dw_pcie_writel_rc(pp, pp->io_base + pp->config.io_size - 1,
+ dw_pcie_writel_rc(pp, io_base, PCIE_ATU_LOWER_BASE);
+ dw_pcie_writel_rc(pp, (io_base >> 32), PCIE_ATU_UPPER_BASE);
+ dw_pcie_writel_rc(pp, io_base + pp->config.io_size - 1,
PCIE_ATU_LIMIT);
dw_pcie_writel_rc(pp, pp->config.io_bus_addr, PCIE_ATU_LOWER_TARGET); dw_pcie_writel_rc(pp, upper_32_bits(pp->config.io_bus_addr),

(quoted diff) +0 -0

b/drivers/pci/host/pcie-designware.h
index c15379b..2fe1dc5 100644

drivers/pci/host/pcie-designware.h +1 -0
@@ -35,6 +35,7 @@ struct pcie_port {
 	struct device		*dev;
 	u8			root_bus_nr;
 	void __iomem		*dbi_base;
+	u64			base_mask;
 	u64			cfg0_base;
 	void __iomem		*va_cfg0_base;
 	u64			cfg1_base;
Quoted 4 lines

Best regards,
Marek Vasut

#15 lore raw
Quoted 15 lines · Kishon Vijay Abraham I

commit b91b61d594f293d994d87f10eb8a1a3b2e5f8977

Author: Kishon Vijay Abraham I <[email protected]>
Date:   Wed Oct 30 14:51:58 2013 +0530

pci: host: pcie-designware: Use *base-mask* for configuring the iATU

In DRA7, the cpu sees 32bit address, but the pcie controller can see only 28bit address. So whenever the cpu issues a read/write request, the 4 most significant bits are used by L3 to determine the target controller. For example, the cpu reserves 0x2000_0000 - 0x2FFF_FFFF for PCIe controller but the PCIe controller will see only (0x000_0000 - 0xFFF_FFF). So for programming the outbound translation window the base should be programmed as 0x000_0000. Whenever we try to write to say 0x2000_0000, it will be translated to whatever we have programmed in the translation window with base as 0x000_0000.

Quoted 1 line

Sorry I didn't see that patch earlier. Have you had a look at the definition of the "dma-ranges" property? I think that is something we have already defined that can handle this in a more generic way.

Arnd

#16 lore raw
In reply to Pratyush Anand

Hi,

Quoted 22 lines

+ Marek, Tim, arnd

Hi Kishon,

Quoted 17 lines · Kishon Vijay Abraham I

Hi Pratyush,

Quoted 14 lines · Pratyush Anand

Hi Kishon,

Quoted 11 lines · Kishon Vijay Abraham I

Hi Pratyush, Jingoo,

Quoted 8 lines · Jingoo Han
Quoted 5 lines · Kishon Vijay Abraham I

Currently I see in pcie-designware.c we use only 2 ATU regions. We re-use INDEX0 for mem outbound and cfg0, and INDEX1 for cfg1 and io. So I'd like to know if in your platform, do you have only 2 address translation regions? In DRA7xx we have 16 outbound regions and 4 inbound regions.

In Exynos, there are only 2 inbound and 2 outbound viewpoints.

Jumping in a tad late, MX6 has 4 inbound and 4 outbound according to MX6DQRM Rev. 1, 04/2013 sections:
- 48.3.9.1.1 (bullet 8)
- 48.3.9.1.2 (bullet 7)
- 48.10.42 (Region_Index description in the table PCIE_PL_iATUVR)

Best regards,
Marek Vasut