arm64: pci: add support for pci_mmap_page_range
PATCH 1/1 arm64: pci: add support for pci_mmap_page_range
certain X11 servers and user space network drivers frameworks need PCI mmaped /sys/bus/pci/devices/B:D:F/resourceX file to access PCI bar address space from user space.
--- arch/arm64/include/asm/pci.h | 6 ++++++ arch/arm64/kernel/pci.c | 20 ++++++++++++++++++++ 2 files changed, 26 insertions(+)
arch/arm64/include/asm/pci.h +6 -0
@@ -39,5 +39,11 @@ static inline int pci_proc_domain(struct pci_bus *bus) }#endif /* CONFIG_PCI */ +#define HAVE_PCI_MMAP + +extern int pci_mmap_page_range (struct pci_dev *dev,struct vm_area_struct *vma, +enum pci_mmap_state mmap_state,int write_combine); + +#endif /* __KERNEL__ */ #endif /* __ASM_PCI_H */
arch/arm64/kernel/pci.c +20 -0
@@ -71,3 +71,23 @@ struct pci_bus *pci_acpi_scan_root(struct acpi_pci_root *root) return NULL; }#endif + +int pci_mmap_page_range (struct pci_dev *dev,struct vm_area_struct *vma, +enum pci_mmap_state mmap_state,int write_combine) +{ +/* +* I/O space can be accessed via normal processor loads and stores on +* this platform but for now we elect not to do this and portable +* drivers should not do this anyway. +*/ +if (mmap_state == pci_mmap_io) +return -EINVAL ; + +if (write_combine) + vma->vm_page_prot =pgprot_writecombine (vma->vm_page_prot); +else + vma->vm_page_prot =pgprot_noncached (vma->vm_page_prot); + +return remap_pfn_range (vma, vma->vm_start, vma->vm_pgoff, + vma->vm_end - vma->vm_start, vma->vm_page_prot); +}
Signature · 1 line
2.1.0
PATCH 1/1 arm64: pci: add support for pci_mmap_page_range
Quoted 44 lines · Jerin Jacob
certain X11 servers and user space network drivers frameworks need PCI mmaped /sys/bus/pci/devices/B:D:F/resourceX file to access PCI bar address space from user space.
Signed-off-by Jerin Jacob <[email protected]>--- arch/arm64/include/asm/pci.h | 6 ++++++ arch/arm64/kernel/pci.c | 20 ++++++++++++++++++++ 2 files changed, 26 insertions(+)arch/arm64/include/asm/pci.h +6
-0@@ -39,5 +39,11 @@ static inline int pci_proc_domain(struct pci_bus *bus) }#endif /* CONFIG_PCI */ +#define HAVE_PCI_MMAP + +extern int pci_mmap_page_range (struct pci_dev *dev,struct vm_area_struct *vma, +enum pci_mmap_state mmap_state,int write_combine); + +#endif /* __KERNEL__ */ #endif /* __ASM_PCI_H */ arch/arm64/kernel/pci.c +10
-0@@ -71,3 +71,23 @@ struct pci_bus *pci_acpi_scan_root(struct acpi_pci_root *root) return NULL; }#endif + +int pci_mmap_page_range (struct pci_dev *dev,struct vm_area_struct *vma, +enum pci_mmap_state mmap_state,int write_combine) +{ +/* +* I/O space can be accessed via normal processor loads and stores on +* this platform but for now we elect not to do this and portable +* drivers should not do this anyway. +*/ +if (mmap_state == pci_mmap_io)
arch/arm64/kernel/pci.c struct pci_bus *pci_acpi_scan_root(struct acpi_pci_root *root) · line 93
Quoted 10 lines
arch/arm64/kernel/pci.c +10
-0+return -EINVAL ; + +if (write_combine) + vma->vm_page_prot =pgprot_writecombine (vma->vm_page_prot); +else + vma->vm_page_prot =pgprot_noncached (vma->vm_page_prot); + +return remap_pfn_range (vma, vma->vm_start, vma->vm_pgoff, + vma->vm_end - vma->vm_start, vma->vm_page_prot); +}
So pci_iomap_range chooses the memory attributes based on the BAR flags (and even then it looks weird -- CACHEABLE => ioremap, else ioremap_nocache, which is just the same as ioremap on arm64).
It would be good to understand (a) why this is different and (b) what the consistent set of attributes should be.
Will
PATCH 1/1 arm64: pci: add support for pci_mmap_page_range
Quoted 60 lines · Will Deacon
Quoted 44 lines · Jerin Jacob
certain X11 servers and user space network drivers frameworks need PCI mmaped /sys/bus/pci/devices/B:D:F/resourceX file to access PCI bar address space from user space.
Signed-off-by Jerin Jacob <[email protected]>--- arch/arm64/include/asm/pci.h | 6 ++++++ arch/arm64/kernel/pci.c | 20 ++++++++++++++++++++ 2 files changed, 26 insertions(+)arch/arm64/include/asm/pci.h +6
-0@@ -39,5 +39,11 @@ static inline int pci_proc_domain(struct pci_bus *bus) }#endif /* CONFIG_PCI */ +#define HAVE_PCI_MMAP + +extern int pci_mmap_page_range (struct pci_dev *dev,struct vm_area_struct *vma, +enum pci_mmap_state mmap_state,int write_combine); + +#endif /* __KERNEL__ */ #endif /* __ASM_PCI_H */ arch/arm64/kernel/pci.c +10
-0@@ -71,3 +71,23 @@ struct pci_bus *pci_acpi_scan_root(struct acpi_pci_root *root) return NULL; }#endif + +int pci_mmap_page_range (struct pci_dev *dev,struct vm_area_struct *vma, +enum pci_mmap_state mmap_state,int write_combine) +{ +/* +* I/O space can be accessed via normal processor loads and stores on +* this platform but for now we elect not to do this and portable +* drivers should not do this anyway. +*/ +if (mmap_state == pci_mmap_io)arch/arm64/kernel/pci.c struct pci_bus *pci_acpi_scan_root(struct acpi_pci_root *root) · line 93
Quoted 10 lines
arch/arm64/kernel/pci.c +10
-0+return -EINVAL ; + +if (write_combine) + vma->vm_page_prot =pgprot_writecombine (vma->vm_page_prot); +else + vma->vm_page_prot =pgprot_noncached (vma->vm_page_prot); + +return remap_pfn_range (vma, vma->vm_start, vma->vm_pgoff, + vma->vm_end - vma->vm_start, vma->vm_page_prot); +}So
pci_iomap_rangechooses the memory attributes based on the BAR flags (and even then it looks weird -- CACHEABLE => ioremap, elseioremap_nocache, which is just the same as ioremap on arm64).
Quoted 1 line
It would be good to understand (a) why this is different and (b) what
AFAIU, pci_iomap_range is the generic implementation and chooses to use only minimal attributes that works on all the architectures. The primary consumer of pci_iomap_range is virtio_pci driver,Which doesn't care about HW PCI memory attributes like Prefetchable.
Quoted 1 line
the consistent set of attributes should be.
PCI perspective, memory attributes are Prefetchable and non-Prefetchable for a given BAR.
Former one does have read side-effects or supports write merging(typically used
by graphics memory) and latter one has read side effects and does not support write merging(typically used by register files)
IMO, In armv8 nomenclature, MT_NORMAL_NC and MT_DEVICE_nGnRnE map correctly to above PCI memory attribute definitions.
and pci-sysfs.c(consumer of pci_mmap_page_range)
creates the /sys/bus/pci/devices/B:D:F/resource[_wc]X file based on BAR flag(Prefetchable or non-Prefetchable) and expose to userspace.
Jerin
Quoted 2 lines
Will
PATCH 1/1 arm64: pci: add support for pci_mmap_page_range
Quoted 26 lines · Jerin Jacob
Quoted 25 lines · Will Deacon
Quoted 9 lines · Jerin Jacob
arch/arm64/kernel/pci.c +9
-0+int pci_mmap_page_range (struct pci_dev *dev,struct vm_area_struct *vma, +enum pci_mmap_state mmap_state,int write_combine) +{ +/* +* I/O space can be accessed via normal processor loads and stores on +* this platform but for now we elect not to do this and portable +* drivers should not do this anyway. +*/ +if (mmap_state == pci_mmap_io)arch/arm64/kernel/pci.c struct pci_bus *pci_acpi_scan_root(struct acpi_pci_root *root) · line 93
Quoted 10 lines
arch/arm64/kernel/pci.c +10
-0+return -EINVAL ; + +if (write_combine) + vma->vm_page_prot =pgprot_writecombine (vma->vm_page_prot); +else + vma->vm_page_prot =pgprot_noncached (vma->vm_page_prot); + +return remap_pfn_range (vma, vma->vm_start, vma->vm_pgoff, + vma->vm_end - vma->vm_start, vma->vm_page_prot); +}So
pci_iomap_rangechooses the memory attributes based on the BAR flags (and even then it looks weird -- CACHEABLE => ioremap, elseioremap_nocache, which is just the same as ioremap on arm64).
Quoted 6 lines
Quoted 1 line
It would be good to understand (a) why this is different and (b) what
AFAIU,
pci_iomap_rangeis the generic implementation and chooses to use only minimal attributes that works on all the architectures. The primary consumer ofpci_iomap_rangeisvirtio_pcidriver,Which doesn't care about HW PCI memory attributes like Prefetchable.
pci_iomap calls pci_iomap_range.
Quoted 5 lines
Quoted 1 line
the consistent set of attributes should be.
PCI perspective, memory attributes are Prefetchable and non-Prefetchable for a given BAR.
Quoted 7 lines
Former one does have read side-effects or supports write merging(typically used
by graphics memory) and latter one has read side effects and does not support write merging(typically used by register files)IMO, In armv8 nomenclature,
MT_NORMAL_NCandMT_DEVICE_nGnRnEmap correctly to above PCI memory attribute definitions.
I agree with your choice of memory types (well, almost. We probably don't need the nE), I'm just after some consistency within the kernel, because pci_iomap_range looks wrong to me. Another different example is pci_ioremap_bar, which always uses ioremap_nocache.
Will
PATCH 1/1 arm64: pci: add support for pci_mmap_page_range
Quoted 27 lines · Will Deacon
Quoted 26 lines · Jerin Jacob
Quoted 25 lines · Will Deacon
Quoted 9 lines · Jerin Jacob
arch/arm64/kernel/pci.c +9
-0+int pci_mmap_page_range (struct pci_dev *dev,struct vm_area_struct *vma, +enum pci_mmap_state mmap_state,int write_combine) +{ +/* +* I/O space can be accessed via normal processor loads and stores on +* this platform but for now we elect not to do this and portable +* drivers should not do this anyway. +*/ +if (mmap_state == pci_mmap_io)arch/arm64/kernel/pci.c struct pci_bus *pci_acpi_scan_root(struct acpi_pci_root *root) · line 93
Quoted 10 lines
arch/arm64/kernel/pci.c +10
-0+return -EINVAL ; + +if (write_combine) + vma->vm_page_prot =pgprot_writecombine (vma->vm_page_prot); +else + vma->vm_page_prot =pgprot_noncached (vma->vm_page_prot); + +return remap_pfn_range (vma, vma->vm_start, vma->vm_pgoff, + vma->vm_end - vma->vm_start, vma->vm_page_prot); +}So
pci_iomap_rangechooses the memory attributes based on the BAR flags (and even then it looks weird -- CACHEABLE => ioremap, elseioremap_nocache, which is just the same as ioremap on arm64).
Quoted 8 lines
Quoted 6 lines
Quoted 1 line
It would be good to understand (a) why this is different and (b) what
AFAIU,
pci_iomap_rangeis the generic implementation and chooses to use only minimal attributes that works on all the architectures. The primary consumer ofpci_iomap_rangeisvirtio_pcidriver,Which doesn't care about HW PCI memory attributes like Prefetchable.
pci_iomapcallspci_iomap_range.
Yes, I missed that.
Quoted 11 lines
Quoted 11 lines
Quoted 2 lines
the consistent set of attributes should be.
PCI perspective, memory attributes are Prefetchable and non-Prefetchable for a given BAR.
Former one does have read side-effects or supports write merging(typically used
by graphics memory) and latter one has read side effects and does not support write merging(typically used by register files)
Quoted 5 lines
Quoted 2 lines
IMO, In armv8 nomenclature,
MT_NORMAL_NCandMT_DEVICE_nGnRnEmap correctly to above PCI memory attribute definitions.I agree with your choice of memory types (well, almost. We probably don't need the nE), I'm just after some consistency within the kernel, because
I thought so, However, Since PCIe deals with off-chip peripherals, IMO it should be MT_DEVICE_nGnRnE(strongly ordered).
Early write acknowledgment may have side effect based on the topology. (connected under Bridge etc).
IMO, It make sense to mark as nGnRE for on-chip peripherals. But If you still think we need to mark as nGnRE then I can submit v2 with nGnRE else IMO, currect patch is fine for pci_mmap_page_range
Jerin
Quoted 4 lines
pci_iomap_rangelooks wrong to me. Another different example ispci_ioremap_bar, which always usesioremap_nocache.Will
PATCH 1/1 arm64: pci: add support for pci_mmap_page_range
Quoted 35 lines · Jerin Jacob
Quoted 6 lines · Will Deacon
Quoted 5 lines · Jerin Jacob
Quoted 4 lines · Will Deacon
So
pci_iomap_rangechooses the memory attributes based on the BAR flags (and even then it looks weird -- CACHEABLE => ioremap, elseioremap_nocache, which is just the same as ioremap on arm64).Quoted 8 lines
Quoted 6 lines
Quoted 1 line
It would be good to understand (a) why this is different and (b) what
AFAIU,
pci_iomap_rangeis the generic implementation and chooses to use only minimal attributes that works on all the architectures. The primary consumer ofpci_iomap_rangeisvirtio_pcidriver,Which doesn't care about HW PCI memory attributes like Prefetchable.
pci_iomapcallspci_iomap_range.Yes, I missed that.
Quoted 11 lines
Quoted 11 lines
Quoted 2 lines
the consistent set of attributes should be.
PCI perspective, memory attributes are Prefetchable and non-Prefetchable for a given BAR.
Former one does have read side-effects or supports write merging(typically used
by graphics memory) and latter one has read side effects and does not support write merging(typically used by register files)Quoted 5 lines
Quoted 2 lines
IMO, In armv8 nomenclature,
MT_NORMAL_NCandMT_DEVICE_nGnRnEmap correctly to above PCI memory attribute definitions.I agree with your choice of memory types (well, almost. We probably don't need the nE), I'm just after some consistency within the kernel, because
Quoted 7 lines
I thought so, However, Since PCIe deals with off-chip peripherals, IMO it should be
MT_DEVICE_nGnRnE(strongly ordered).
Early write acknowledgment may have side effect based on the topology. (connected under Bridge etc).
IMO, It make sense to mark as nGnRE for on-chip peripherals. But If you still think we need to mark as nGnRE then I can submit v2 with nGnRE else IMO, currect patch is fine forpci_mmap_page_range
It's the inconsistencies that I've pointed out previously that bother me the most. I don't really care what attributes we use, as long as there's a common way to translate a given BAR to a given MT_*.
Will