PPC PCI bus registers
Hello,
I have a question regarding reading PCI bus registers from a user space application running on a PPC SBC. Seeing as though the PCI bus is little endian and PPC is big endian is it typical that one must perform a byte swap on all 16 and 32 bit register reads?
I've found this is true on a custom board I am working on(with an FPGA connected via the PCI bus) and as a result I've added a byte swap command in busybox to accommodate this feature...
Signature · 5 lines
Best Regards, ________________________________________________________________ Eddie Dawydiuk, Technologic Systems | voice: (480) 837-5200 16525 East Laser Drive | fax: (480) 837-5300 Fountain Hills, AZ 85268 | web: www.embeddedARM.com
Quoted 6 lines · Eddie Dawydiuk
Hello,
I have a question regarding reading PCI bus registers from a user space application running on a PPC SBC. Seeing as though the PCI bus is little endian and PPC is big endian is it typical that one must perform a byte swap on all 16 and 32 bit register reads?
Yes, this is correct.
g.
Quoted 10 lines · Eddie Dawydiuk
Hello,
I have a question regarding reading PCI bus registers from a user space application running on a PPC SBC. Seeing as though the PCI bus is little endian and PPC is big endian is it typical that one must perform a byte swap on all 16 and 32 bit register reads?
I've found this is true on a custom board I am working on(with an FPGA connected via the PCI bus) and as a result I've added a byte swap command in busybox to accommodate this feature...
Note that powerpc has efficient load/store reverse instructions that perform the byteswap for you. We use them for IOs in the kernel for example.
Also, if you're going to access a PCI device directly, beware of other issues such as ordering. PPC is an out of order architecture, you need to ensure you add the appropriate memory barriers if you want to ensure you accesses are done in the order you write them in your program.
For "standard" stuff that doesn't involve DMA or locks, an eieio after both MMIO loads and stores should do the trick.
If you need to order vs. DMA and/or locks, you may want to look at what the kernel does in io.h
Cheers,
Ben.
Benjamin,
Quoted 7 lines
Also, if you're going to access a PCI device directly, beware of other issues such as ordering. PPC is an out of order architecture, you need to ensure you add the appropriate memory barriers if you want to ensure you accesses are done in the order you write them in your program.
For "standard" stuff that doesn't involve DMA or locks, an eieio after both MMIO loads and stores should do the trick.
I'm not sure I understand. To clarify I have an FPGA connected via the PCI bus which implements several peripherals, I've implemented device drivers for. Currently I am calling ioremap() to get a virtual address corresponding to the PCI devices. Then I use ___raw_writeN / ___raw_readN for reading/writing data via the PCI bus to the FPGA registers. From looking at io.h I believe this method is safe with regard to out of order execution.
"* ioremap is the standard one and provides non-cacheable guarded mappings
* and can be hooked by the platform via ppc_md "
Can you verify if my understanding is correct, or let me know if I need to add memory barriers?
Signature · 5 lines
Best Regards, ________________________________________________________________ Eddie Dawydiuk, Technologic Systems | voice: (480) 837-5200 16525 East Laser Drive | fax: (480) 837-5300 Fountain Hills, AZ 85268 | web: www.embeddedARM.com
Quoted 12 lines · Eddie Dawydiuk
I'm not sure I understand. To clarify I have an FPGA connected via the PCI bus
which implements several peripherals, I've implemented device drivers for.
Currently I am callingioremap()to get a virtual address corresponding to the
PCI devices. Then I use___raw_writeN/___raw_readNfor reading/writing data
via the PCI bus to the FPGA registers. From looking atio.hI believe this
method is safe with regard to out of order execution.
Quoted 8 lines
"* ioremap is the standard one and provides non-cacheable guarded mappings
* and can be hooked by the platform viappc_md"Can you verify if my understanding is correct, or let me know if I need to add
memory barriers?
Out of order execution != out of order storage. ioremap() will give you guarded space which means it cannot be speculatively accessed for example, and you do get -some- guarantees but not that your stores are going to hit the device in order, nor that your loads are going to be performed until the CPU actually use the result of the load, which can be delayed beyond a store.
Cheers,
Ben.
Benjamin,
Quoted 6 lines
Out of order execution != out of order storage.
ioremap()will give you guarded space which means it cannot be speculatively accessed for example, and you do get -some- guarantees but not that your stores are going to hit the device in order, nor that your loads are going to be performed until the CPU actually use the result of the load, which can be delayed beyond a store.
Interesting. So IIUC if I were to use ioremap and then bit bang bus cycles using readN() / writeN() I would need to add memory barrier to ensure the order of the load and store operations does not change. Is my understanding correct? If so what are the recommended macros/functions for adding memory barriers, or are there preferred functions to use in place of ioremap() and readN()/writeN?
PS I've looked in Documentation/ in the 2.6.30 Linux kernel tree for more information, but I was unable to find any. If there are any example drivers or documentation that you know of on this topic I'd be quite interested if you could point me to it.
Thanks for your patience and time you've been quite helpful.
Signature · 5 lines
Best Regards, ________________________________________________________________ Eddie Dawydiuk, Technologic Systems | voice: (480) 837-5200 16525 East Laser Drive | fax: (480) 837-5300 Fountain Hills, AZ 85268 | web: www.embeddedARM.com
Quoted 9 lines · Eddie Dawydiuk
Benjamin,
Quoted 6 lines
Out of order execution != out of order storage.
ioremap()will give you guarded space which means it cannot be speculatively accessed for example, and you do get -some- guarantees but not that your stores are going to hit the device in order, nor that your loads are going to be performed until the CPU actually use the result of the load, which can be delayed beyond a store.
Quoted 3 lines
Interesting. So IIUC if I were to use ioremap and then bit bang bus cycles using
readN()/writeN()I would need to add memory barrier to ensure the order of the load and store operations does not change. Is my understanding correct?
Only if you use the __raw variants. readN/writeN already provide full barriers, including barriers against leakage outside of locks and barriers vs. cacheable accesses (MMIO vs. DMA) and thus are slow.
If you use the raw variants, it then depends whether your code is going to be ppc specific, in which case you can manually use things like eieio() etc... or not in which case ... it's harder :-) You can always use rmb/wmb/mb but those are more heavyweight than just an eieio
There's iobarrier_r/w/rw, dunno if those are generic tho.
Quoted 3 lines
If so
what are the recommended macros/functions for adding memory barriers, or are there preferred functions to use in place ofioremap()and readN()/writeN?
ioremap + readN / writeN will give you fully ordered behaviour. It's only when you try to be sneaky or speedy and use the __raw variants that you have to be more careful.
Quoted 6 lines
PS I've looked in
Documentation/in the 2.6.30 Linux kernel tree for more information, but I was unable to find any. If there are any example drivers or documentation that you know of on this topic I'd be quite interested if you could point me to it.Thanks for your patience and time you've been quite helpful.
Cheers,
Ben.
Benjamin,
Quoted 3 lines
ioremap + readN / writeN will give you fully ordered behaviour. It's only when you try to be sneaky or speedy and use the
__rawvariants that you have to be more careful.
You've been most helpful, thanks.
Signature · 5 lines
Best Regards, ________________________________________________________________ Eddie Dawydiuk, Technologic Systems | voice: (480) 837-5200 16525 East Laser Drive | fax: (480) 837-5300 Fountain Hills, AZ 85268 | web: www.embeddedARM.com