ARM: Use 64-bit DMA addresses for LPAE+VirtIO-MMIO
Subsystems (3)
PATCH 1/1 ARM: Use 64-bit DMA addresses for LPAE+VirtIO-MMIO
On an LPAE system, the physical addresses used by VirtIO-MMIO may be larger than 32 bits, even if the header and configuration space addresses fit into 32 bits. For example with the Versatile Express memory map using 4G memory, the following error occured when trying to use a VirtIO-MMIO block device.
EXT2-fs (vda): error: ext2_check_page: bad entry in directory #2: : unaligned directory entry - offset=0, inode=3755990991, rec_len=57311, name_len=223
To fix this, select ARCH_DMA_ADDR_T_64BIT when both LPAE and VIRTIO_MMIO are selected.
--- arch/arm/mm/Kconfig | 1 + 1 file changed, 1 insertion(+)
arch/arm/mm/Kconfig +1 -0
@@ -617,6 +617,7 @@ config ARM_LPAE bool "Support for the Large Physical Address Extension" depends on MMU && CPU_32v7 && !CPU_32v6 && !CPU_32v5 && \ !CPU_32v4 && !CPU_32v3 + select ARCH_DMA_ADDR_T_64BIT if VIRTIO_MMIO help Say Y if you have an ARMv7 processor supporting the LPAE page table format and you would like to access memory beyond the
Signature · 2 lines
Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, hosted by the Linux Foundation.
Quoted 23 lines · Christopher Covington
On an LPAE system, the physical addresses used by VirtIO-MMIO may be larger than 32 bits, even if the header and configuration space addresses fit into 32 bits. For example with the Versatile Express memory map using 4G memory, the following error occured when trying to use a VirtIO-MMIO block device.
EXT2-fs (vda): error:
ext2_check_page: bad entry in directory #2: :
unaligned directory entry - offset=0, inode=3755990991, rec_len=57311, name_len=223To fix this, select
ARCH_DMA_ADDR_T_64BITwhen both LPAE andVIRTIO_MMIOare selected.Signed-off-by Christopher Covington <[email protected]>--- arch/arm/mm/Kconfig | 1 + 1 file changed, 1 insertion(+)arch/arm/mm/Kconfig +0
-0
arch/arm/mm/Kconfig config ARM_LPAE · line 620
Quoted 4 lines
arch/arm/mm/Kconfig +1
-0@@ -617,6 +617,7 @@ config ARM_LPAE bool "Support for the Large Physical Address Extension" depends on MMU && CPU_32v7 && !CPU_32v6 && !CPU_32v5 && \ !CPU_32v4 && !CPU_32v3 + select ARCH_DMA_ADDR_T_64BIT if VIRTIO_MMIO
That's the wrong place to enable ARCH_DMA_ADDR_T_64BIT. Do you have a platform with >32-bit physical address space? If yes, it should be selected there.
Signature · 1 line
Catalin
Hi Catalin,
Quoted 19 lines · Catalin Marinas
Quoted 18 lines · Christopher Covington
On an LPAE system, the physical addresses used by VirtIO-MMIO may be larger than 32 bits, even if the header and configuration space addresses fit into 32 bits. For example with the Versatile Express memory map using 4G memory, the following error occured when trying to use a VirtIO-MMIO block device.
EXT2-fs (vda): error:
ext2_check_page: bad entry in directory #2: :
unaligned directory entry - offset=0, inode=3755990991, rec_len=57311, name_len=223To fix this, select
ARCH_DMA_ADDR_T_64BITwhen both LPAE andVIRTIO_MMIOare selected.Signed-off-by Christopher Covington <[email protected]>--- arch/arm/mm/Kconfig | 1 + 1 file changed, 1 insertion(+)
Quoted 13 lines
arch/arm/mm/Kconfig config ARM_LPAE · line 620
Quoted 9 lines
arch/arm/mm/Kconfig +1
-0@@ -617,6 +617,7 @@ config ARM_LPAE bool "Support for the Large Physical Address Extension" depends on MMU && CPU_32v7 && !CPU_32v6 && !CPU_32v5 && \ !CPU_32v4 && !CPU_32v3 + select ARCH_DMA_ADDR_T_64BIT if VIRTIO_MMIOThat's the wrong place to enable
ARCH_DMA_ADDR_T_64BIT. Do you have a platform with >32-bit physical address space? If yes, it should be selected there.
The platforms I'm currently using are models like the Versatile Express RTSM/FVP. I can respin with changes to ARCH_VEXPRESS and ARCH_VIRT instead.
Thanks,
Christopher
Signature · 3 lines
Employee of Qualcomm Innovation Center, Inc. Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by the Linux Foundation.
Quoted 34 lines · Christopher Covington
Quoted 19 lines · Catalin Marinas
Quoted 18 lines · Christopher Covington
On an LPAE system, the physical addresses used by VirtIO-MMIO may be larger than 32 bits, even if the header and configuration space addresses fit into 32 bits. For example with the Versatile Express memory map using 4G memory, the following error occured when trying to use a VirtIO-MMIO block device.
EXT2-fs (vda): error:
ext2_check_page: bad entry in directory #2: :
unaligned directory entry - offset=0, inode=3755990991, rec_len=57311, name_len=223To fix this, select
ARCH_DMA_ADDR_T_64BITwhen both LPAE andVIRTIO_MMIOare selected.Signed-off-by Christopher Covington <[email protected]>---
arch/arm/mm/Kconfig| 1 +1 file changed, 1 insertion(+)Quoted 13 lines
arch/arm/mm/Kconfig config ARM_LPAE · line 620
Quoted 9 lines
arch/arm/mm/Kconfig +1
-0@@ -617,6 +617,7 @@ config ARM_LPAE bool "Support for the Large Physical Address Extension" depends on MMU && CPU_32v7 && !CPU_32v6 && !CPU_32v5 && \ !CPU_32v4 && !CPU_32v3 + select ARCH_DMA_ADDR_T_64BIT if VIRTIO_MMIOThat's the wrong place to enable
ARCH_DMA_ADDR_T_64BIT. Do you have a platform with >32-bit physical address space? If yes, it should be selected there.
Quoted 2 lines
The platforms I'm currently using are models like the Versatile Express RTSM/FVP. I can respin with changes to
ARCH_VEXPRESSandARCH_VIRTinstead.
But do you use RAM beyond 32-bit on such models?
Catalin
Quoted 16 lines · On Friday 21 March 2014 23:27:24 Catalin Marinas wrote:
Quoted 15 lines · Christopher Covington
Quoted 13 lines · Catalin Marinas
arch/arm/mm/Kconfig config ARM_LPAE · line 620
Quoted 9 lines
arch/arm/mm/Kconfig +1
-0@@ -617,6 +617,7 @@ config ARM_LPAE bool "Support for the Large Physical Address Extension" depends on MMU && CPU_32v7 && !CPU_32v6 && !CPU_32v5 && \ !CPU_32v4 && !CPU_32v3 + select ARCH_DMA_ADDR_T_64BIT if VIRTIO_MMIOThat's the wrong place to enable
ARCH_DMA_ADDR_T_64BIT. Do you have a platform with >32-bit physical address space? If yes, it should be selected there.
Quoted 4 lines
Quoted 2 lines
The platforms I'm currently using are models like the Versatile Express RTSM/FVP. I can respin with changes to
ARCH_VEXPRESSandARCH_VIRTinstead.But do you use RAM beyond 32-bit on such models?
I think the more important question here is what the normal behavior is for these platforms. I believe in most cases you don't have RAM above the boundary, so we should not enable the option by default as it can have noticeable overhead (we'd turn it on all the time if it didn't).
How about one of these two
a)
arch/arm/mm/Kconfig +3 -1
@@ -628,7 +628,9 @@ config ARCH_PHYS_ADDR_T_64BIT def_bool ARM_LPAE config ARCH_DMA_ADDR_T_64BIT - bool + def_bool "Allow DMA to high (>4GB) addresses" + depends on ARCH_PHYS_ADDR_T_64BIT + default y help config ARM_THUMB
bool "Support Thumb user binaries" if !CPU_THUMBONLY
b)
arch/arm/mm/Kconfig +16 -4
@@ -619,16 +619,28 @@ config ARM_LPAE help Say Y if you have an ARMv7 processor supporting the LPAE page table format and you would like to access memory beyond the - 4GB limit. The resulting kernel image will not run on - processors without the LPA extension. + 4GB limit, or run virtual machines usign KVM. + The resulting kernel image will not run on processors without + the LPA extension. If unsure, say N. config ARCH_PHYS_ADDR_T_64BIT - def_bool ARM_LPAE + bool "Support more than 4GB of physical address space" if EXPERT + depends on ARM_LPAE + default y + help + Say Y if you use LPAE to access RAM or MMIO registers at + addresses beyond the low 4GB of address space. If this + option is disabled, the kernel will use the LPAE page table + format and it can offer KVM support, but will use a slightly + more efficient representation of physical memory addresses + that restricts access to 32-bit addressable locations. + + If unsure, say Y. config ARCH_DMA_ADDR_T_64BIT - bool + def_bool ARCH_PHYS_ADDR_T_64BIT config ARM_THUMB bool "Support Thumb user binaries" if !CPU_THUMBONLY
In either case, platforms that need this support can always 'select' it, while a kernel built only for platforms that don't need it can offer this as a user-selectable option.
Arnd
Quoted 36 lines · Arnd Bergmann
Quoted 16 lines · On Friday 21 March 2014 23:27:24 Catalin Marinas wrote:
Quoted 15 lines · Christopher Covington
Quoted 13 lines · Catalin Marinas
arch/arm/mm/Kconfig config ARM_LPAE · line 620
Quoted 9 lines
arch/arm/mm/Kconfig +1
-0@@ -617,6 +617,7 @@ config ARM_LPAE bool "Support for the Large Physical Address Extension" depends on MMU && CPU_32v7 && !CPU_32v6 && !CPU_32v5 && \ !CPU_32v4 && !CPU_32v3 + select ARCH_DMA_ADDR_T_64BIT if VIRTIO_MMIOThat's the wrong place to enable
ARCH_DMA_ADDR_T_64BIT. Do you have a platform with >32-bit physical address space? If yes, it should be selected there.Quoted 4 lines
Quoted 2 lines
The platforms I'm currently using are models like the Versatile Express RTSM/FVP. I can respin with changes to
ARCH_VEXPRESSandARCH_VIRTinstead.But do you use RAM beyond 32-bit on such models?
I think the more important question here is what the normal behavior is for these platforms. I believe in most cases you don't have RAM above the boundary, so we should not enable the option by default as it can have noticeable overhead (we'd turn it on all the time if it didn't).
How about one of these two
a)
arch/arm/mm/Kconfig +0
-0
arch/arm/mm/Kconfig config ARCH_PHYS_ADDR_T_64BIT · line 634
Quoted 8 lines
arch/arm/mm/Kconfig +3
-1@@ -628,7 +628,9 @@ config ARCH_PHYS_ADDR_T_64BIT def_bool ARM_LPAE config ARCH_DMA_ADDR_T_64BIT - bool + def_bool "Allow DMA to high (>4GB) addresses" + depends on ARCH_PHYS_ADDR_T_64BIT + default y help
We should probably call this "Allow DMA to high (>4GB) _bus_ addresses" to distinguish it from the actual CPU physical address.
Quoted 6 lines
arch/arm/mm/Kconfig +0
-0b)arch/arm/mm/Kconfig +0
-0
[...]
arch/arm/mm/Kconfig config ARM_LPAE · line 631
Quoted 5 lines
config
ARCH_PHYS_ADDR_T_64BITarch/arm/mm/Kconfig +3
-1- def_bool ARM_LPAE + bool "Support more than 4GB of physical address space" if EXPERT + depends on ARM_LPAE + default y
I don't think I ever tested LPAE with ARCH_PHYS_ADDR_T_64BIT off. Even if you enable EXPERT, it's a risk that some bit shifting of phys_addr_t would break (the pte expects 64-bit entries).
arch/arm/mm/Kconfig config ARM_LPAE · line 643
Quoted 3 lines
config
ARCH_DMA_ADDR_T_64BITarch/arm/mm/Kconfig +1
-1- bool + def_bool ARCH_PHYS_ADDR_T_64BIT
I recall Russell objecting to this when we reviewed the LPAE patches because the CPU address is different from the bus address. I assume those arguments still stand.
I would go for (a) but with individual SoCs still selecting ARCH_DMA_ADDR_T_64BIT explicitly if they expect bus addresses over 32-bit.
Signature · 1 line
Catalin
Hi Arnd,
Quoted 20 lines · Arnd Bergmann
Quoted 19 lines · On Friday 21 March 2014 23:27:24 Catalin Marinas wrote:
Quoted 17 lines · Christopher Covington
Quoted 5 lines · Catalin Marinas
Quoted 5 lines
arch/arm/mm/Kconfig +0
-0@@ -617,6 +617,7 @@ config ARM_LPAEQuoted 8 lines
arch/arm/mm/Kconfig config ARM_LPAE · line 620
Quoted 4 lines
arch/arm/mm/Kconfig +1
-0bool "Support for the Large Physical Address Extension" depends on MMU && CPU_32v7 && !CPU_32v6 && !CPU_32v5 && \ !CPU_32v4 && !CPU_32v3 + select ARCH_DMA_ADDR_T_64BIT if VIRTIO_MMIOThat's the wrong place to enable
ARCH_DMA_ADDR_T_64BIT. Do you have a platform with >32-bit physical address space? If yes, it should be selected there.The platforms I'm currently using are models like the Versatile Express RTSM/FVP. I can respin with changes to
ARCH_VEXPRESSandARCH_VIRTinstead.
Quoted 6 lines
Quoted 1 line
But do you use RAM beyond 32-bit on such models?
I think the more important question here is what the normal behavior is for these platforms. I believe in most cases you don't have RAM above the boundary, so we should not enable the option by default as it can have noticeable overhead (we'd turn it on all the time if it didn't).
I don't understand why people concerned with the overhead of 64-bit pointers to memory are turning on LPAE.
Quoted 3 lines
In either case, platforms that need this support can always 'select' it, while a kernel built only for platforms that don't need it can offer this as a user-selectable option.
I was hoping to fix the issue of things mysteriously breaking at runtime, regardless of what the user or randconfig selects. I'll have to investigate the details, but it seems like one could enforce zone DMA at run-time if the memory size is greater than the DMA pointers are able to address.
Thanks,
Christopher
Signature · 3 lines
Employee of Qualcomm Innovation Center, Inc. Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by the Linux Foundation.
PATCH v2 1/1 ARM: virt, vexpress: Use 64-bit DMA for LPAE
On an LPAE system, the physical addresses used by VirtIO-MMIO may be higher than 32 bits. For example on a Versatile Express system with 4G RAM, the following error surfaces when trying to use a VirtIO-MMIO block device.
EXT2-fs (vda): error: ext2_check_page: bad entry in directory #2: : unaligned directory entry - offset=0, inode=3755990991, rec_len=57311, name_len=223
To fix this, select ARCH_DMA_ADDR_T_64BIT when LPAE is selected and mach-vexpress, mach-virt, or both are being built.
--- arch/arm/Kconfig | 1 + arch/arm/mach-vexpress/Kconfig | 1 + 2 files changed, 2 insertions(+)
arch/arm/Kconfig +1 -0
@@ -901,6 +901,7 @@ endmenu config ARCH_VIRT bool "Dummy Virtual Machine" if ARCH_MULTI_V7 + select ARCH_DMA_ADDR_T_64BIT if ARM_LPAE select ARM_AMBA select ARM_GIC select ARM_PSCI
arch/arm/mach-vexpress/Kconfig +1 -0
@@ -1,5 +1,6 @@ menuconfig ARCH_VEXPRESS bool "ARM Ltd. Versatile Express family" if ARCH_MULTI_V7 + select ARCH_DMA_ADDR_T_64BIT if ARM_LPAE select ARCH_REQUIRE_GPIOLIB select ARCH_SUPPORTS_BIG_ENDIAN select ARM_AMBA
Signature · 2 lines
Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, hosted by the Linux Foundation.
Quoted 35 lines · Catalin Marinas
Quoted 34 lines · Christopher Covington
Quoted 32 lines · Catalin Marinas
Quoted 23 lines · Christopher Covington
On an LPAE system, the physical addresses used by VirtIO-MMIO may be larger than 32 bits, even if the header and configuration space addresses fit into 32 bits. For example with the Versatile Express memory map using 4G memory, the following error occured when trying to use a VirtIO-MMIO block device.
EXT2-fs (vda): error:
ext2_check_page: bad entry in directory #2: :
unaligned directory entry - offset=0, inode=3755990991, rec_len=57311, name_len=223To fix this, select
ARCH_DMA_ADDR_T_64BITwhen both LPAE andVIRTIO_MMIOare selected.Signed-off-by Christopher Covington <[email protected]>---
arch/arm/mm/Kconfig| 1 +1 file changed, 1 insertion(+)arch/arm/mm/Kconfig +0
-0arch/arm/mm/Kconfig config ARM_LPAE · line 620
Quoted 4 lines
arch/arm/mm/Kconfig +1
-0@@ -617,6 +617,7 @@ config ARM_LPAE bool "Support for the Large Physical Address Extension" depends on MMU && CPU_32v7 && !CPU_32v6 && !CPU_32v5 && \ !CPU_32v4 && !CPU_32v3 + select ARCH_DMA_ADDR_T_64BIT if VIRTIO_MMIOThat's the wrong place to enable
ARCH_DMA_ADDR_T_64BIT. Do you have a platform with >32-bit physical address space? If yes, it should be selected there.
Quoted 4 lines
Quoted 2 lines
The platforms I'm currently using are models like the Versatile Express RTSM/FVP. I can respin with changes to
ARCH_VEXPRESSandARCH_VIRTinstead.But do you use RAM beyond 32-bit on such models?
Yes. On the Versatile Express memory map only 2G RAM is 32-bit accessible, and I'm using 4G as I tried to note in the commit message.
http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.dui0575h/Bbajihec.html
(For reference, the device tree I'm passing the A32 kernel is arch/arm64/boot/dts/rtsm_ve-aemv8a.dts.)
Christopher
Signature · 3 lines
Employee of Qualcomm Innovation Center, Inc. Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by the Linux Foundation.